Sunday, November 13, 2005

Change to Image Adjustments

Click on the screenshot and if you can't figure it out the change...stop turking
Update: Btw This appears to be an anti-bot change (trying to keep bots frombeing able to trick mturk)
Update 2: Wow this is annoying....just a tip always click the black box on your way down it's easy to later on down click another one then have to go back up and find the black box. Secondly what is sucking is I keep getting ones with no black box only options.....so the actually place isn't in any of the photos so I have to give them one that is wrong (I usually just choose the best looking photo) Though the server just went down again...so maybe they changed their minds...lol

14 Comments:

Blogger Sandoz said...

I have to say i'm not happy with it.

I've got this thing down to about 6 seconds/HIT and i do it manually. But now if there is no applicable picture i have to scroll back up to find this randomly placed photo. This can cost me seconds at a time! Lol.. and we all know that time is money.

I hope the move to a system where. Randomly they throw a few of these at you to test you. Perhaps a 'Trial Period' when you first signup and then random periods to verify the human element. Sort of to certify you are bot-free.

6:39 PM  
Anonymous Fish said...

I don't like it either. Tried to send them a feedback asking them to put the link back on the bottom but their feedback page is broken.

7:32 PM  
Anonymous Nate said...

Wow this is really annoying, although from what I have been seeing it is obvious someone has set up a bot to just select 'none' every time. I wish they could come up with some HTML trickery to combat this though, because it is slowing me down a lot.

8:12 PM  
Blogger cobaltgrc said...

yeah this really blows.

9:20 PM  
Anonymous drekmonger said...

If this is slowing you down, you're probably going too fast. My speed seems about the same (a little more than 150/hr, discounting the 20 or so an hour I return).

No doubt there's bots farming mturk, this change will be a (minor) obstacle to them. More importantly, it'll be a speedbump for human beings as well.

There's tons of a9 HITs that require a fine attention to detail in order to choose the correct building. You really need to squint to see certain building numbers and business names. I have no doubt that there's a horde of zombies just clicking "None of the above." on all non-obvious HITs.

Anyone who intends to farm mturk in the future beyond the beta should be concerned with the quality of answers given to the system. If the service doesn't produce results, who's going to pay?

10:10 PM  
Blogger Sandoz said...

Drekmonger you claim that more than 150 hits / hour is probably going too fast. Well. that's bull. That equals about $4.00/hr and would be considered slave labour.

I do about 600 hits / hr before this new slowdown.. and closer to 500 / hour now. The quality of my selections is no less than yours i have an accepted rate of nearly 90% now. This new system slows people down significantly and is unnecessary there are better ways for them to verify if a user is a bot or not. Ways that do not include making the work force less efficient.

I do not spend more time being more careful i spend omre time scrolling. Your theory is full of holes.

10:29 PM  
Anonymous drekmonger said...

600 HITs an hour...500 HITs an hour. hrm. No offense, but if you actually picked out the correct image you're approval rate would be closer to 85%-80%--many correct choices are "outvoted" by bots and zombies and rejected.

I'm guessing you're 600 HITs an hour consist of at least 300 junk answers. If mturk is filled with junk, then NO ONE WILL PAY. The experiment will fail.

Agreed that $4 an hour is pathetic for a job...it's not a job, it's a beta test of a system and something to do whilst not otherwise engaded. I suspect that, if the system works out the bugs (like zombies and bots) then there will be eventually class of HITs that a person *could* make a living on. But, again, that won't happen if the answers produced by the system are garbage.

No offense, that's just how it is.

10:42 PM  
Blogger Sandoz said...

Wow you just don't get it. I work more efficiently. It does not mean i have a lower quality of work. I use two monitors. and scripts to hit the accept button for me. So as i work on one screen another page is loading so i have no downtime and no wait time.

You are pretty pathetic to discredit my ability to do quality work just because you cannot match that performance level. I would be more than happy to invite you over to watch the way i do this work and to verify that it is as accurate as any other person that actually looks at the pictures and makes an informed decision.

You are pretty close minded to think that because you peak at 150 hits / hour and have probably not taken steps to optimize your process that no other person could possibly do better.

My accuracy rate is approachign 90% becuase i do the job properly.

How they need to do it is to have build in testing routines that determine your ability to find picutres by throwing in ones that they already know the answer to and scanning your results.


If you would like to read a bit on how i achive this many hits / hour pleast go to http://www.blogger.com/comment.g?blogID=18803218&postID=113173309185877948

I have made 2 posts there on my technique. I assure you if you follow the same technique you will double your speed and still maintain the same level of quality you are providing now.

I started out doing 150/hr and i've worke don making it faster. You can too.. So before you point your fingers and claim that i'm wrong again. Follow the steps and try it first. You do not need 2 monitors you can use 2 seperate browser windows.

If you're unsure which scripts I am referring to and and how to set them up. Leave a message here and i will give you detailed instructions.

But watch while you're doing your entries. And count on a submission how much of the total time of 1 submission is spent either waiting for hte pictures to load or scrolling back to find the buttons. I eliminate all of that waiting time.

10:59 PM  
Blogger Sandoz said...

That link didn't post well here

it's on this blog

http://mturkey.blogspot.com/

and it's in the comments after the

post titled 'New Email'

http://tinyurl.com/8csub

11:02 PM  
Anonymous drekmonger said...

It is possible that you (sandoz) in particular actually have a high level of quality. You might have an uncanny, superhuman ability to process visual information. (however, again, your high acceptance rate actually speaks more towards a low quality....you are "voting" with the pack, and the pack is wrong on at least 10% of the HITs by my estimation.)

...let's agree there are zombies out there who perform the "job" mindlessly, and choose junk answers. Amazon will have to figure out a way to curtail this, or the whole mturk thing just won't work.

11:27 PM  
Anonymous drekmonger said...

Sorry for the double post, there's no way to edit my prior post:

another possiblity, sandoz, is that you have been verifying your own HITs, esp. since your volumn is so high. For example, when you answer the same HIT three times identically, you get 3 acceptances.

Note, this may have changed with the newest update. Your accuracy might fall back down to the average 85%, or maybe all of our accuracies will raise to 90% if bots and zombies are stymed by the update.

11:35 PM  
Blogger Sandoz said...

To be accurate my statement of 'nearing 90%'.. it actually sits at 87% accepted 13% rejected. and by 'nearing' i mean that 2 days ago it was at approximately 84% accepted and 16% rejected.. and thus .. was nearing 90%..

And again i can't stress enough how much time is wasted by waitin gfor pages to load twice.. (pre and post accept button)and thus can easily do 2x as many by having one fetching while you are doing one hit.

I have no actually gotten used tot he black box and it' snot slowing me down anymore. And i do understand the importance of weeding out bots and indiscriminate clickers. I want to see this program succeed as much as anyone cause i can actually seem to make it be worth a reasonable amount of money.

I guess they are under tight time constraints too.. and this is a dirty but quick fix to the problem. And i'm sure it is not the final solution to bots and that it is just a temporary measure.

And on a side note i'm glad to read your posts on your suspicions on how it works. Cause i've been putting the same puzzle together but rather than reading into it.. i spent my time optimizing my end.

anyways.. let the experiment continue


ps: someone needs to setup a forum for this rather than the comments on blogs back and forth... but beyond my technical ability

11:51 PM  
Anonymous Badlands said...

I'm impressed with your speed, sandoz! I was doing a little over 5 HITs a minute before the change, and a little under 5 since. That's in short bursts, though, as I generally don't go for extended periods of intense turking for fear of going looney (wait, too late!). I'll often do it in between doing other things or do it more leisurely.

I try to avoid places with lots of strip malls as they really slow you down, trying to see if you can see the business name in the distance (and if you can, will the others see it and match yours?). NYC is also a bit harder since they have so many businesses in each frame.

Another thing that slows me down is that I sometimes miss clicking the radio button on the first click and have to re-click it. I'm going to suggest to Amazon that they increase the size of the buttons, or better yet, fill in the box if you click on the photo.

12:11 AM  
Blogger Sandoz said...

There are 2 scripts available that run in Firefox if you use the Greasemonkey plugin.

1 of them will automatically click the 'Accept HIT' button for you.. which greatly increases your speed

The other one. Actually makes it so you can click directly on the image (skip finding the little radio button) and after clicking on the image it will actually submit the hit for you so it reduces each entry to 1 click rather than 3 or 4.. and reduces your load time by hitting accept wihtout waiting for the preview page to load all of the images.

This would take care of missing the radio buttons as well as greatly increase your efficiency

http://userscripts.org/home/boring_search?search=mechanical+turk

8:56 AM  

Post a Comment

<< Home