I ran across an intriguing concept while browsing yesterday - The Mechanical Turk.The original Mechanical Turk was supposedly a chess playing automaton. The Turk was able to dispatch human chess opponents with ease. The Turk was actually a scam - a human chess master was hiding inside of the machine.
You can read more about the original Mechanical Turk at wikipedia. It's worth a few minutes of your time. Fast forward to today. Amazon has created it's own version of the Turk.
Requesters post jobs. Workers perform the task. The requester reviews the work, and the workers get paid.
Most of the jobs at the Turk are fairly simple and straightforward. Write a quick review of a product. Post a comment to a blog post. Visit a web site. Do you like A or B better?
The Turk seems geared mostly towards companies that would like instant feedback to a new idea or survey. Does this product look better in blue or green?
I've posted a few jobs at the Turk out of curiosity. Chances are I will be publishing a few of the results. I've requested reviews of specific products, and even posted a job to request specific criticism of this website. The Turk workers that provide short criticisms for MeanPC.com get a nickel, and I get valuable information. If someone just enters "your site sux" or "nice site", I reject the work and they don't get paid.
The Turk is an interesting concept. There's lot of people with time on their hands out there. Let's put em to work.
Here's some examples of the Turk at Wurk. I offered to pay 5 Turk workers to write articles detailing their experience with the Turk to give MeanPC readers a deeper insight into the Turk machine. I offered to pay the Mechanical Turk 50 cents per article about..... the Turk. This first piece is pretty good. I think this person can write well, and surely should be getting more than 50 cents for her work. Anyway....read on.
The Mechanical Turk, also known as MTurk, provides a service for those looking for information which computers can’t provide: human intelligence. It also provides a way for people to make extra money, by performing those tasks, called HITs.
When it comes to MTurk there are both positives and negatives about the service, although my personal experience with MTurk, while brief, has been mostly positive. Mturk allows someone with an internet connection to earn extra income from home. I have earned some extra money that I plan to use to buy some extra things I would normally not buy. The pay is minimal, but the advantages are that you can work from home, or at any computer connected to the internet, there is no need to travel (so I subtract the time I would have used traveling to and from work from the time I have invested in MTurk, and the hourly pay increases), no cost in gas to commute, and a lot less headache because I’m not being stuck in traffic or at risk of being in an accident, etc.
The negative aspects of MTurk include a variance on the HITs available. It is not always reliable when it comes to the availability of steady-pay HITs, and when you do get on a roll with some of them you are always at risk that thousands of other people are doing them as well, and then those are soon dried up. So the fluctuation when it comes to the HIT availability is always going to be there, and will always depend on who is looking for information.
For instance, my first day on MTurk there was over a hundred thousand HITs available, but since then it has never even come close to that number again. It seems to hover around ten to twenty thousand, often less.
The other negative includes the low pay. Sometimes you can spend about a minute on a HIT that only pays a penny. That doesn’t add up very fast (sixty cents to a dollar, maybe, an hour), which means you are working for well below minimum wage. However, sometimes you can fly through the HITs that pay only a penny and get a little more. For the most part, however, you’re making less than you would at a steady job. However, if you do not have a steady job, or you have a job but need a little extra, and you have the time to devote yourself to a couple hours a day, or even a week, you can defiantly make a few extra bucks. As I mentioned before, it all has to do with the availability of the HITS, and how fast you can work through them.
Despite the negatives, I like MTurk. It is nice to have the option to do something from home. Being a caregiver, I am at home a lot and have often wanted ways which I could devote my time to something in return for monetary reward, and MTurk does provide an option for that. I only wish there were more options like this available!
OK.... overall a positive experience I think. Let's hear from another -
I am a new member of Mechanical Turk and so far my experience has been a good one. I like the fact that it gives you the opportunity to make some extra money at your own leasure. I can log on anytime it is convient for me and whatever hit I feel comfortable with. The hits that I have completed are easy and quite simple to complete, but you can choose what is best for you. Also, you can make as much or as little as you would like. The other great factor is that it is completely free to you and there is no quota or limit as to what you complete.
I personally like doing the photo tags. They consist of looking at pictures and describing what you see in the simplest form, there is no right or wrong way to do them. I work two jobs and Mechanical Turk is a great opportunity for me to make some extra money at my own leasure. I can sign on at 11:oo pm and work for an hour or 5 hours whatever I choose there is no limit. I also, like going to the dash board where you can view how much you made for hits completed and what hits are still pending approval.
Here's a slightly more disturbing view of the Turk....
I've been doing work at Mechanical Turk for nearly two years, and for the most part I have to say that my experience has not been very positive. I continue to do the work because even the small amount of money I've earned here is helpful to me. I'm disabled and on a fixed income, so it's difficult to pass up opportunities that I feel I can handle.
First, let me recount some of the good things about Mechanical Turk. I can work from home at any time of the day or night. If I don't feel well on any given day, I'm under no obligation to do any work. If I feel great that day I can do as much work as I'm able to find. That sums up the good.
Now, for the bad. Most of the work here is unappealing and the pay is a joke. There are only a handful of HITs (Human Intelligence Tasks) that are worthwhile and they're in very high demand and tend to disappear very quickly. Almost everything else seems to come from requesters who want to pay next to nothing for fairly time consuming tasks. For example, I've seen HITs that asked us to verify information in long documents, a task that could take ten to twenty minutes to be done correctly. The pay: one cent. What's worse, whether or not we get paid is entirely up to the requester's whim. There have been many, many instances when a requester has put up a HIT, gotten the work done as instructed but has then gone ahead and rejected the HIT on a technicality or for no known reason - meaning they got their task done for free. The way Mechanical Turk is currently set up it heavily favors requesters at the expense of workers.
Yes, over time such bad requesters do get weeded out because we learn through the grapevine who to trust and who not to trust, but we remain at a disadvantage. To be fair, there are some good requesters that pay low but fair amounts for their tasks and who are responsive to worker feedback. But they are the exception. Most of the requesters seem to view their workers as sub-human and never respond to any feedback given to them.
Surprisingly those that seem affiliated with Amazon are among the least responsive. For example, I've mostly done work for the requester NowNow, which is associated with Amazon's Askville service and their Kindle eBook reader. They have a "research question" HIT in which end-users will ask any question and Turkers will try to find the answer on the web and return it to them quickly. NowNow has set up their HIT in the form of a contest, with the workers who receive the largest number of "great" votes from end-users earn a bonus at the end of the week. There are many, many problems with the way this is set up, not least of which is the fact that end-users have no idea that we get paid based on "great" votes, so mostly they don't vote at all or sometimes submit "good" votes which are worthless to us. It can be very frustrating to do all that research and write up some really great answers and end up uncompensated for all that work, and this is exactly what happens most of the time. But since they offer a relatively large bonus, up to $100 per week, the NowNow HIT attracts a lot of interest and is likely Mechanical Turk's highest profile HIT.
In recent months the situation with that requester has deteriorated for a number of reasons and workers are openly expressing their dissatisfaction. The requester has made a number of changes which have ended up requiring more and more work from Turkers, all for the same low pay and the same uncertainty inherent in their set up. And they have appeared uncaring of workers' concerns. Virtually every email sent to them advising them of a problem or with some dissatisfaction with the way the service is run is ignored. NowNow continues to view Turkers as subroutines, not people. We don't even have the value of lab rats to them, we're simply bits of code to play with and manipulate, to experiment upon and tweak in an effort to change the result of a programming process. I don't think it's a conscious or malicious effort on their part, but it's a deep-seated aspect of their mentality and their approach to Mechanical Turk. It's only after there's an uproar over one of their actions that they step back for a moment and realize, heck, there's a real person being affected by this decision. And at that point we may get a very corporate sounding email apologizing for the way things were handled. But even then, nothing changes for the better and none of the problems are fixed. As a Turker, I find the treatment and response we've gotten to be dehumanizing. There are a number of other requesters that share that approach, and I only point out NowNow because it's the one I've done the most work for and am most familiar with. But they aren't alone in their practices. And unfortunately, their behavior isn't limited to requesters but is common among the people who work directly for Mechanical Turk.
Recently there was a change in the way payments are handled, but we received no advanced warning of the change and no explanation of what the change meant. When I wrote to inquire about what it all meant, they didn't seem to want to be bothered answering and it took nearly a week of repeated emails for me to get an adequate response. When other problems have turned up on the site, complaints go ignored. In the time I've been here, I've been surveyed several times by people who work at Mechanical Turk for ways of improving the site and about things I like and dislike about it. That makes it seem like they do care about their workers. But nothing has ever changed. What was the point in soliciting my feedback if they were just going to ignore it? There really seems to be no effort on their part to really take the workers' point of view into consideration.
Now, I'm aware that this review may come off as more of a rant than anything else. As I said at the start, I've been doing this for a while and I've been hoping that things would improve with time. They haven't. Just the opposite in fact. So my frustration level with the service has grown tremendously with time as has my cynicism. But I think I've been very fair to the service in giving it time to get its act together, and at this point I'm resigned in my belief that it never will. I'm also convinced most requesters view the service as a means of exploiting the underprivileged and that they have no compassion for us. Since they never have to see our faces or hear our voices, they feel comfortable in viewing us as inconsequential bits of code. It's shameful.
Another happy Turker :)
Ten days ago I was at home with flu and I was perusing some on-line computer magazines. A short article was describing how TagCow was tagging photos using humans instead of sophisticated programs. I'm a researcher in computer science so I was curious, both professionally and personally, and I started a session on Mturk. I already had an account on Amazon since, even if I'm Italian and I live in Italy, I read most of my fiction in English.
At a first glance the idea of "artificial artificial intelligence" appeared very nice and I was particularly happy to have guessed the meaning of the word "mturk" (when I was a kid I saw a TV transposition of an E.A.Poe's short story).
The site swarmed with HITs. There were at least 100,000 HITs from Amazon itself, asking to decide if two items from its catalog were indeed the same article or not. At 0.01$ = 0.0064 Euro / HIT, I thought that my time was a little too cheap, because at the beginning some pairs required more than few second to be decided. Anyway, with a little of experience I learnt some tricks of the trade and I was able to complete more than 1000 such HITs with less than 1% rejections in a reasonable time.
The problem is that I was becoming greedy. I obsessively listed all the 500 or so different kinds of HITs looking for some big fish. You must understand that I have a severe handicap: the language. I can read English very fast, about as fast as I read Italian, but, even if I (co)authored more that 50 scientific papers in English, I'm not very fluent in writing and I'm quite bad when it comes to understand spoken English. Thus, I could not accept the juicy HITs which required to transcribe audio and the HITs asking to rewrite of long articles on various arguments took more time than I was prepared to spend on Mturk.
But I can say with pride that I really shined writing some short blog-like entries of about 100 words each. In my little masterpieces I was an overprotecting mother discovering that her son has a girlfriend, a depressed husband on the verge of cheating his wife, a secretary exchanging secret messages with her lover across a room full of unknowing collegues and finally a no-nonsense husband with a soft spot for romantic inscriptions. Honestly, it was a challenge, but it was fun.
What about the photo tagging from which I started my acquaintance of Mturk? Well, I did some of these HITs, in which for $0.04 you were requested to tag 5 pictures, i.e., to describe in general terms what can be seen and all the text (if any). I abandoned quite soon this kind of HIT, for two reasons: I was too slow in describing some common situations because I could not force myself to find the right word and (more baleful) after a few tagging my natural inclination to bening mockery became more difficult to suppress from picture to picture. Instead of aseptic tags, my first impulse was to write things like "two intoxicated guys", "gorgeous bride plus so-so groom", "wannabe dancer ridiculizes himself", "another blurred one", "the sad case of the headless boyfriend" and so on.
Anyway, few days later there was a new kind of HITs: you had a large picture and several little shots of people and you have to determine who, among them, appeared in the first picture. I discovered that from a certain point, probably because of a software misconfiguration, they were showing always a subset of, say, a dozen of pictures and always asking if a certain guy appeared or not (he never appeared). So for a certain number of hours there was an endless flow of these HITs were you can very check all the "does not appear" boxes without even looking at the pictures. Since I'm a good guy I sent an e-mail to the Requester (I even tried to phone them) to inform them that they were giving away money for free, but since I'm not so a good guy I also made about 20 bucks in a couple of hours of desperate clicking.
In these last 10 days I was a little compulsive (as it happens often with new things) and I spent a lot of my spare time on Mturk. I completed about 3000 HITs and I made about $75. The Amazon HITs "Are these items different?" at first apparently endless, finished, returned, finished again and returned in this right moment (but not in so huge quantities as in my first day on Mturk), at least they give you something to do when more lucrative HITs are scarce. One thing that I would like to see answered in the FAQ is "how many days, at worst, must I wait before my HIT is paid or rejected?" because I have some HITs still pending after 9 days and I'm growing nervous... Another dream will be the possibility to transfer funds from my Amazon account to my Paypal account.
In conclusion, you may ask me if I'm here on Mturk for the fun or for the dough.
Well, you can ask me, but it will cost you another $0.01....
