I spent the last couple hours melting faces with fireballs in Oblivion (well, mostly turning human foes against one another using illusion magic) to unwind from the brutal moderation experience I went through with Mechanical Turk. Let me tell you tale …
In order to streamline the FanSiter seed site creation, I stopped using their “batch” web mode (works just like a mail merge) since it is impossible to retry individual HIT’s. Thus you get an all-or-nothing kind of deal or you mediate it by adding redundant HIT’s. I have a contractor dedicated to solely fixing all the data this caused to get fragmented and thus the whole thing backfired rather magically. Sure, there are tons of individual sites, but you only see a portion of the bio’s I paid for.
So I spent significant time and development effort to learn the API and write a few PHP classes to help me make use of it. They work great and after a bit of mind-numbing testing I unleashed new code which would spawn HIT’s based on Landing-only entries in the plat. This did its job as expected, but certain parts of my moderation tools were broken and required repairs:
- Stray ampersands made Blogger puke, so approved HIT’s got disposed without a successful post being created. I updated the nifty RepairTags() function but then didn’t add code to call it. Face palm!
- Mid-way through modifications made against live I changed the moderate() method to return a string status rather than a boolean, but I didn’t correct the way it decided which status to return so it was always approving. Whoops! Set a bad precedent and I had to delete terrible data that I’d paid for.
- Adding another assignment when rejecting a previous one seemed like a good idea, but when people weren’t “getting” a particular name (Orange Avenue had 3 rejections, because people wrote about an album by the same name — WTF?! And for Tettix someone wrote a blurb on the Cicada bug instead of the musician!) I had to manually go in and blow away the whole HIT. If you’ve ever used Amazon’s “Manage HIT’s Individually” then you know how painful this is.
There were probably other, smaller issues, but those three got me frustrated. Add to that the quality of the data and I started to get really irritated. I had kept the wording simple, but specific in the HIT instructions and didn’t duplicate much between its description and the question text. Unfortunately, it appears many ignore the description since its not very noticable, and there was that confusion of writing about a bug or an album rather than a celebrity (despite that being in the title). I got a lot of completed assignments that were either literal copies of Wikipedia passages or only slightly re-arranged. There’s a tool called “Google” which is amazing at helping you find duplicate or plagiarized content. I guess when you spend very little time on something, you don’t care if you get away with it or not.
Having a length requirement (200+ words) allowed me to reject poorly written stuff on that point alone, but then that ended up being a sticky wicket. I failed to give a reason on a few of these, since I assumed the results being so bad would be apparent to the writer, and ended up with this fun email conversation:
Jim: may i ask why you rejected this?
Me: You’ll have to provide the contents of the HIT, the software I’m using disposes them once they’re rejected. Two possibilities that come to mind based on the past rejections: too short (requirement was 200+ words) and/or copied paragraphs from Wikipedia or other sources.
Jim: all were over 200 words and nothing was taking from anyone else. I have found that this site is just to good to be true, shame on me for trusting you. I have reported you to amazon.
Me: That’s fine. I found this, is it yours?
Devon Aoki is an actress and model, she is best known for her roles in Rosencrantz and Guildenstern Are Undead as Anna, Mutant Chronicles as Cpl. Valerie Duval, War as Kira, DOA: Dead or Alive as Sumi, Sin City as Miho, D.E.B.S. as Dominique and 2 Fast 2 Furious as Suki.
Devon Aoki has been doing television and movies since 2003. She has also done some modeling in 2006.
Devon Aoki is a beautiful charming actress that we see going places in the future.
That’s 84 words.
Jim: yes
He didn’t stop there, though, here’s another email thread about his one other assignment.
Jim: why did you not pay me? I did the work in my own words, i guess you are just another scam site, like most of the others on here, I will report you to amazon, as I did what you asked of me.
Me:
Jesse Eisenberg is best known for his roles in Kick the Can, The Social Network, Holy Rollers, Camp Hope, Some Boys Don’t Leave, Beyond All Boundaries, Solitary Man, Zombieland, Adventureland and The Hunting Party.
Jesse Eisenberg was born 5 October 1983, New York City, New York, USA.
Jesse Eisenberg has been in the Television and Motion Picture industries since 1999.
Jesse Eisenberg is currently working on 4 movies.
Our outlook for Jesse Eisenberg is watch out he is already a rising star, and we believe anything he touches will turn to gold.
92 words.
Ugh, those are just awful seed bio’s and then he lashes out to defend them. On the one hand I have the right to approve/reject whatever I want, that’s the rules of Turk, and I can do it based on quality alone. On the other I only set the award at 0.49 (0.50 shows up as 0.5) and that may be part of the problem. What I’ve found, though, is raising or lowering that amount still gets you same amount of cruft to sift! I did get a few really awesome blurbs today, but all in all probably approved only 20%.
I had wanted to avoid requiring qualifications or a quiz, but now I believe it’s going to be necessary. I have thought a lot about whether or not to just go to a singular writer or writing firm, but those just don’t scale the way I’d like. Plus you get less variety in the style since it’s the same person doing all of them. I like the idea of letting in random people, who have down time to make some bucks rather than screwing around on the Internet. The glaring problem here is the current crap I’m getting, both in content and sass.
So … what to do? Here’s my thoughts. I’m going to add some qualification requirements, which is more research and coding (argh!). Then in order to make it a worthwhile pain, I will set the award to something in the neighborhood of $5. Thus you can flip burgers or do blurb essays on Turk. Will it work? I don’t know, but I can’t continue the way I have. My sanity is at stake.