Sunday, November 13, 2005

Some detailed Speculation on Approvals

I just wanted to draw attention to drekmonger's comment on how the HIT's are approved or rejected which has some very nice speculation and a few bits on the API

"drekmonger said...

I've already scanned through the API. The mechanism for approval is entirely up to the requester...

There's a call to retrieve a completed HIT, reject a completed HIT, and approve a completed HIT. Whether or not to approve a HIT is logic that is coded by the requester.

From my observations, the critia for approval and disapproval of the A9 HITs is evolving. As of late Saturday, early Sunday, the HITs seem to approve if X (possible 3 or 4) people click the same answer to a single HIT. Otherwise, the HIT is reguritated for a tie-breaker.

The system doesn't seem to keep track of who's done what HIT--at least it doesn't seem as such based on my quick read of the API. And HITs can be set to done multiple times.

So, it's possible that a single person (or bot) to receive the same HIT 3 or 4 times. If you answer it the same way each time, then you'll get that many HITs accepted.

I've noticed HITs I've completed end up in what I believe are the "tie-breaker" requests. These are the HIT groups for a city that contain just a few dozen HITs. When I complete such a HIT idetically to a prior HIT, I've noticed the number of HITs accepted in my dashboard increment by 2.

The trick would then be to guess how other people would answer the HIT, rather than try to answer the HIT itself. Unfortunately, I think there's a lot of people zipping through the HITs and not paying close attention--so even if a single person correctly identifies a business, he could be outvoted by 3 zombies clicking "none of the above"

4 Comments:

Anonymous drekmonger said...

Of course, the critia has changed once again with the latest update. The A9 HITS are no longer automatically accepting HITs with the (new) "none" choice. I have no clue how amazon intends to verify HITs now.

11:20 PM  
Blogger Sandoz said...

It seems like in the last 40 minutes or so they have started to approve again.

11:54 PM  
Anonymous Brian said...

I submitted 432 HITs between November 3-5. Currently 209 of those are still pending, nearly 50%.

As I understand the API now, there is no system to prevent an unscrupulous requester from never approving or rejecting submitted HITs. Amazon needs to implement some system that guarantees that the people doing the work gets paid if the requester fails to approve/reject a HIT in a reasonable period of time.

The system I envision adds a "Abandoned By Requester" category to Approved/Rejected/Pending. After a certain period of time (say, 7 days), a pending HIT becomes Abandonded By Requester and the HITter gets paid.

12:27 AM  
Blogger Eric Cranston said...

exactly. As well there needs to be some system to make sure that requestors aren't trying to scam us...rejecting HIT's that they use

Perhaps a view in which we can see the average rejection rate of the requestor...of course right now the only requestor so far has been amazon which I trust.

10:21 AM  

Post a Comment

<< Home