Bots and Zombies Vs. The Black Box
Well everyone who has MTurked recently has noticed the new black box. And it seems to annoy the heck out of most of you.
First off let me say this the black box is good for us...It's good for everyone who isn't running a bot and who isn't a zombie. How so? Obviously by killing current bots (and trust me there are a number running) and it reduces the rate that HIT's disappear...Although HIT's haven't appeared to be in short supply. Secondly for those who aren't zombies when you do ID the business properly, you have a much better chance of it getting approved and not rejected because of "none of the above" votes from bots. And thirdly the system has stopped lagging so bad since the black box appeared. Fourthly it helps us all sorta slow down and make sure we complete the HIT correctly.
Of course this modification isn't hard for a good bot coder to get around. Simply scan the images for a large amount of black....Or randomly pick a radio button and hope for the best. But the scanning the images uses more computing power, time, and bandwidth. And randomly picking a radio button would drop profits for them considerably.
So Yes the black box was a good decision for Amazon, for right now, and perhaps for us in the long run. But I hope this is a temporary thing, there are many other ways that Amazon could stops bots....For instance simply suspending someone who chooses "none" 200 times in a row. Or just noticing the timing and how rhythmic it is. Two things I assumed they already had in place though I haven't heard anything on. But nevertheless the black box is a minor annoyance and if MTurk had it when it came out I wouldn't have minded....So let's try to be open minded and maybe pass a few better anti-bot measures along to Amazon via feedback
Let's remember too....The main reason nothing quite like this has been successful before....Is that bots and zombies have killed them....Let's bear with MTurk as it tries to rise above that. Oh ya and don't quit you day job....Not yet, at least.
First off let me say this the black box is good for us...It's good for everyone who isn't running a bot and who isn't a zombie. How so? Obviously by killing current bots (and trust me there are a number running) and it reduces the rate that HIT's disappear...Although HIT's haven't appeared to be in short supply. Secondly for those who aren't zombies when you do ID the business properly, you have a much better chance of it getting approved and not rejected because of "none of the above" votes from bots. And thirdly the system has stopped lagging so bad since the black box appeared. Fourthly it helps us all sorta slow down and make sure we complete the HIT correctly.
Of course this modification isn't hard for a good bot coder to get around. Simply scan the images for a large amount of black....Or randomly pick a radio button and hope for the best. But the scanning the images uses more computing power, time, and bandwidth. And randomly picking a radio button would drop profits for them considerably.
So Yes the black box was a good decision for Amazon, for right now, and perhaps for us in the long run. But I hope this is a temporary thing, there are many other ways that Amazon could stops bots....For instance simply suspending someone who chooses "none" 200 times in a row. Or just noticing the timing and how rhythmic it is. Two things I assumed they already had in place though I haven't heard anything on. But nevertheless the black box is a minor annoyance and if MTurk had it when it came out I wouldn't have minded....So let's try to be open minded and maybe pass a few better anti-bot measures along to Amazon via feedback
Let's remember too....The main reason nothing quite like this has been successful before....Is that bots and zombies have killed them....Let's bear with MTurk as it tries to rise above that. Oh ya and don't quit you day job....Not yet, at least.
posted by Eric Cranston at 11:12 PM
8 Comments:
Put me in the crowd who doesn't like the new black box. It slows me down considerably. I've also noticed a reduction in my % of approved hits since the change. I was averaging 88% before the change (through 3000+ hits) and have been at 83% tonight.
I also noticed a change in the wording in the instructions, specifically the first line.
Here's what it said before: "Choose the picture that best represents the storefront or the building for the business."
Here's what it says now: "Choose the picture that best represents the storefront, building entrance, or address of the business."
This begs the question if we should click on a matching address on a building where the business name is not visible. I've always clicked the "none" option in that case. Then what if the description is "Turk's Pizza," but all you can see is a sign that says "Pizza?"
I wish Amazon would give more clear direction. Not only would it help us, but it would help them get the result they want.
I always go with if I see the address I click on that picture
FWIW, my approval rate since the change isn't as bad as it first appeared. I've had a bunch processed since my last post and I'm now up to 87% for the night.
BTW, some of the groups with a low # of HITs have the old school "none of the above" listings (possibly tie-breakers?).
Good questions badlands.
I think amazon should have a training qualification for the A9 HITs....with examples of what to do in various sitations.
For example, if the left half of a storefront is in one frame, and the right half in another, do we select the best looking picture or "none"?
Ya.....I especially think some training is needed for the HIT's where you write....the examples on them weren't worth a darn
Quite Simply put MTurk doesn't clearly state what it wants...which is important when your talking about thousands of HIT's. Also perhaps a long boring tutorial would keep people out who are just there to "check it out" and would put in bad HIT votes
Hey guys I set up a free PHP forum. It should tide us over for now. Here it is:
http://youngturk.proboards76.com/index.cgi
Hey I have been enjoying your site...there aren't many other bloggers out there talking about this besides you and lurker...So I decided to start my own photoblog of interesting/funny/etc images I and others find on MTurk.
Check it out...
http://www.mturk.laurenmayeux.com/blog/
and feel free to submit some of your own pictures.
It seems like your forums arent working, so I set some up on my web hosting. Enjoy.
http://waltz2.com/mturk/index.php
Post a Comment
<< Home