Sunday, November 13, 2005

Bots and Zombies Vs. The Black Box

Well everyone who has MTurked recently has noticed the new black box. And it seems to annoy the heck out of most of you.

First off let me say this the black box is good for us...It's good for everyone who isn't running a bot and who isn't a zombie. How so? Obviously by killing current bots (and trust me there are a number running) and it reduces the rate that HIT's disappear...Although HIT's haven't appeared to be in short supply. Secondly for those who aren't zombies when you do ID the business properly, you have a much better chance of it getting approved and not rejected because of "none of the above" votes from bots. And thirdly the system has stopped lagging so bad since the black box appeared. Fourthly it helps us all sorta slow down and make sure we complete the HIT correctly.

Of course this modification isn't hard for a good bot coder to get around. Simply scan the images for a large amount of black....Or randomly pick a radio button and hope for the best. But the scanning the images uses more computing power, time, and bandwidth. And randomly picking a radio button would drop profits for them considerably.

So Yes the black box was a good decision for Amazon, for right now, and perhaps for us in the long run. But I hope this is a temporary thing, there are many other ways that Amazon could stops bots....For instance simply suspending someone who chooses "none" 200 times in a row. Or just noticing the timing and how rhythmic it is. Two things I assumed they already had in place though I haven't heard anything on. But nevertheless the black box is a minor annoyance and if MTurk had it when it came out I wouldn't have minded....So let's try to be open minded and maybe pass a few better anti-bot measures along to Amazon via feedback

Let's remember too....The main reason nothing quite like this has been successful before....Is that bots and zombies have killed them....Let's bear with MTurk as it tries to rise above that. Oh ya and don't quit you day job....Not yet, at least.

12 Comments:

Blogger Sandoz said...

Yeah i've come to accept this solution. I realize that it is a quick and dirty fix to stop the bot'ers and indescriminate clickers. After a few hundred hits it hardly slows me down now.. Just changes the cadence.

I'm sure over time planted HITS will be used and other subtle, unseen techniques to weed out these people.

Vive le turk

11:53 PM  
Blogger Sandoz said...

PS: Someone needs to setup a PHP forum or something about this so we can keep threads going..

11:53 PM  
Anonymous Badlands said...

Put me in the crowd who doesn't like the new black box. It slows me down considerably. I've also noticed a reduction in my % of approved hits since the change. I was averaging 88% before the change (through 3000+ hits) and have been at 83% tonight.

I also noticed a change in the wording in the instructions, specifically the first line.

Here's what it said before: "Choose the picture that best represents the storefront or the building for the business."

Here's what it says now: "Choose the picture that best represents the storefront, building entrance, or address of the business."

This begs the question if we should click on a matching address on a building where the business name is not visible. I've always clicked the "none" option in that case. Then what if the description is "Turk's Pizza," but all you can see is a sign that says "Pizza?"

I wish Amazon would give more clear direction. Not only would it help us, but it would help them get the result they want.

11:56 PM  
Blogger Eric Cranston said...

I always go with if I see the address I click on that picture

12:10 AM  
Anonymous Badlands said...

FWIW, my approval rate since the change isn't as bad as it first appeared. I've had a bunch processed since my last post and I'm now up to 87% for the night.

BTW, some of the groups with a low # of HITs have the old school "none of the above" listings (possibly tie-breakers?).

12:55 AM  
Anonymous drekmonger said...

Good questions badlands.

I think amazon should have a training qualification for the A9 HITs....with examples of what to do in various sitations.

For example, if the left half of a storefront is in one frame, and the right half in another, do we select the best looking picture or "none"?

12:59 AM  
Blogger zoops said...

Is anyone else getting this error:

We encountered an internal error while accessing the payment management system.
We apologize for the inconvenience. Please try again later.

I've been getting this for the past several days.. really puts a dampening on my enthusiasm to do HITs.

7:00 AM  
Blogger Sandoz said...

I do not get this error, however i do find that if i try to access my account while using firefox i'm told that they 'page contains no data' and if i use internet explorer it works no problem.

Re: Need for training

I agree that more clearcut examples need to be given whith those situations explained. What if only 1/2 a business is visible. What if you see PIZZA but not TOM'S PIZZA that sort of thing.

Also wouldn't help to having a training series of HITS where the answers are already known by the mturk staff. And through answering those HITS it could be determined when someone is answering correctly or not and certify them to move on to real HITS.

I was also thinking it would be helpful if you could specify which direction from the current series is the proper location. As sometimes you see the business begining to appear in the last photo.. but not enough to select it In addition/instead of that you could have a box that allowed you to enter 1 visible street address so the program could auto-recalibrate the actual locations of each photo series.

8:45 AM  
Blogger Eric Cranston said...

Ya.....I especially think some training is needed for the HIT's where you write....the examples on them weren't worth a darn

Quite Simply put MTurk doesn't clearly state what it wants...which is important when your talking about thousands of HIT's. Also perhaps a long boring tutorial would keep people out who are just there to "check it out" and would put in bad HIT votes

10:19 AM  
Blogger cobaltgrc said...

Hey guys I set up a free PHP forum. It should tide us over for now. Here it is:

http://youngturk.proboards76.com/index.cgi

10:21 AM  
Anonymous Lauren said...

Hey I have been enjoying your site...there aren't many other bloggers out there talking about this besides you and lurker...So I decided to start my own photoblog of interesting/funny/etc images I and others find on MTurk.

Check it out...
http://www.mturk.laurenmayeux.com/blog/
and feel free to submit some of your own pictures.

12:45 PM  
Anonymous Anonymous said...

It seems like your forums arent working, so I set some up on my web hosting. Enjoy.
http://waltz2.com/mturk/index.php

2:23 AM  

Post a Comment

<< Home