Home > Crossway Blog > Mechanical Turk Recap

Mechanical Turk Recap

Our experiment with Mechanical Turk went better than we expected. We were able to approve 85% of submissions automatically, and we ultimately approved 98.3% of submissions. These figures came in higher than we planned: we thought we would approve 80% automatically and ultimately approve 90-95%.

We would (and probably will) use Mechanical Turk again for similar projects.

Background: What Is Mechanical Turk?

Computers can’t do everything. Consider the following sentences:

John saw Jane at the store. She said, “Hi.”

You can tell right away that Jane is the person speaking. A program would have to perform serious computational gymnastics to reach the same conclusion.

Amazon invented Mechanical Turk for situations just like the one above: to solve problems that humans find easy but computers find hard.

Our Approach

We’re always looking to improve the quality of the ESV text database, especially as we prepare an OSIS version. (OSIS is an XML format for the Bible.) OSIS allows us to indicate who speaks each direct quotation in the Bible.

We considered Mechanical Turk as we pondered the best way to go about building this database. For this first foray, we decided to use Mechanical Turk to verify what we created ourselves—in other words, to verify an internal database.

The ESV text contains about 7,100 quotations. Of those, 5,700 are top-level quotations (as opposed to quotes within quotes). Of those, we’ve already identified the quotes spoken by Jesus (indicated by red-letter text). Scratch another 643 quotes. So we have about 5,000 quotations to look at. No problem.

We had someone from Crossway go through each quotation using a web application we developed. This person spent about eight hours on the project over the course of a week and identified about 3,100 quotes in which the text identified by name the speaker of a quotation.

Next we wrote a few Perl scripts (including this module) to handle the uploading. We uploaded the 3,100 quotes over several hours on Tuesday, June 13, 2006, to give our blog readers plenty of time to respond to our invitation to participate.

We then wrote a program to compare the answers submitted by Mechanical Turk workers to our existing answers. About 85% of them were exact matches, so we approved those without even looking at them. We reviewed the remaining 500 or so by hand and only had to reject 54 of them for being wrong.

Pluses

  • Inexpensive. We got a database for about $75 that, as far as we can tell, no one has created before for the Bible.
  • Fast. We uploaded one HIT every five seconds over six hours. Workers performed these HITs almost as fast as they were uploaded. Seventy-eight workers participated.
  • High quality. We only rejected 1.7% of submissions, an excellent figure by any standard.

Minuses

  • No developer sandbox. We had to upload funds and grab a HIT ourselves to make sure everything worked OK. We would have liked a place to test our programs without having to expose them to the world (and without having to pay).
  • Funds have to come from a bank account. We had to get special authorization to withdraw funds, and it took a week after initiating the transfer for the funds to show up in our Mechanical Turk account. We would’ve preferred to pay with a credit card, even if that meant buying $20 blocks of Mechanical Turk credits at a time.
  • Limited formatting options. We would’ve liked to be able to put the quotation in bold; instead, we had to indicate it with brackets: [QUOTE BEGINS HERE]. We would prefer to use XHTML, as the limited formatting restricts the type of application we can develop with Mechanical Turk. Some enterprising individuals have worked around this limitation by asking people to visit a different website, answer the questions there, and enter a code from the other website. It works, but it’s not ideal.

Conclusion

We would use Mechanical Turk again, especially if they address the formatting restrictions to allow more interactive applications.

Creating metadata for the Bible is often detail-oriented and labor-intensive. Mechanical Turk presents a new and helpful way to spread the work inexpensively among many people (“crowdsourcing”). We estimate that Mechanical Turk cut our costs by about 60% for a comparable-quality result.

Update: The Amazon Web Services Blog has written about our experiment. We should add that we did find six instances in which our database was wrong and the Turkers were right.

June 15, 2006 | Posted in: Digital,ESV | Author: Crossway Staff @ 8:53 am | (17) Comments »

17 Comments

  1. [...] Those crazy ESV cats are geeks. [...]

    Pingback by The Boars Head Tavern » Blog Archive » — June 28, 2006 @ 2:50 pm

  2. [...] And it was used with great success on the Bible quotation project for the English Standard Version of the bible. According to their entry, it cost them $75 to confirm a database that before now, has never existed for the bible in the history of humankind. Not bad! [...]

    Pingback by uskollinen » Artificial Artifical Intelligence — June 28, 2006 @ 2:53 pm

  3. [...] In fact, the English Standard Version has started using it for building up their database. This is something I could probably get involved with. [...]

    Pingback by The Open Source Ministry Blog » ESV Used Mechanical Turk — June 28, 2006 @ 9:42 pm

  4. [...] Another cool use of the Mechanical Turk. This has to be the most innovative thing I’ve seen in that it is an api for manual labour. [...]

    Pingback by Damien Mulley » Blog Archive » fantasy is what people want, but reality is what people need - Links D’Fluffy 29th June 2006 — June 29, 2006 @ 4:12 am

  5. [...] In addition to being a great translation, the ESV guys are very high-tech…my favorite combination Christians and geeks They even have a blog. It’s terrific that they can weave Perl, XML, and Bible quotes in the same post! Check out their use of Amazon’s Mechanical Turk in this entry, ESV Bible Blog » Blog Archive » Mechanical Turk Recap. June 29th 2006 Posted to Technology, Philosophy [...]

    Pingback by .:: a few thoughts on the subject by rob wright ::. » ESV uses Amazon’s Mechanical Turk — June 29, 2006 @ 10:01 am

  6. [...] [...]

    Pingback by Musings of a Chicagoan — July 1, 2006 @ 1:19 am

  7. [...] The ESV is controlled by people who understand technology. In this sense, the ESV completely stands alone, and is a “future” translation–forward facing not only in translation method, but also in distribution methods. The ESV used Mechanical Turk to produce a dirt-cheap database that is unique to Christendom. The ESV has an excellent set of APIs, none of which are seen as a detriment to the hard copy on the shelf at Borders. [...]

    Pingback by Free, as in beer « Photography on Tap — November 6, 2006 @ 8:27 pm

  8. Great advice from Fred Wilson on bringing someone in from the outside to run your startup. Interesting concept video showing off the ability of an animation company. Thanks Andy. The rm *nix command. Overheard on the phone.

    Trackback by fantasy is what people want, but reality is what people need ... — January 18, 2007 @ 4:23 pm

  9. Bible quote database built with Amazon Mechanical Turk for $75 2 cents a quote, with a 98.3% success rate [via]. read more at Waxy.org Links. rss2lj.

    Trackback by Bible quote database built with Amazon Mechanical Turk for $75 — January 18, 2007 @ 4:23 pm

  10. The Standard Bible Society used the “crowdsourcing” of Amazon.com’s Mechanical Turk to produce an English Standard version of the bible at $.02 a verse and 98.3% accuracy. I had forgotten all about the mechanical turk until I read this

    Trackback by Bible quote database built with Amazon Mechanical Turk for $75 — January 18, 2007 @ 4:23 pm

  11. Bible quote database built with Amazon Mechanical Turk for $75. Okay the sheep were cute, but this is interesting.

    Trackback by Bible quote database built with Amazon Mechanical Turk for $75 — January 18, 2007 @ 4:23 pm

  12. I’m still working away (far too slowly for my impatient tastes) on the first complete first of New Testament Names, a semantic knowledgebase of named things in the New Testament and their relationships: you can get a sense of it from

    Trackback by Disambiguating Names in the New Testament — January 18, 2007 @ 4:23 pm

  13. Esv_blog There are some fascinating posts over at the ESV blog. ESV refers to the English Standard Version of the Bible. In order to increase the accuracy of their database of Biblical quotations, they used the Amazon Mechanical Turk.

    Trackback by Mechanical Turk for Metadata Collection — January 18, 2007 @ 4:23 pm

  14. From the blogroll… New Fink release. CameraHobby – Learning Modules and e-Books. Mechanical Turk Recap.

    Trackback by links for 2006-06-16 — January 18, 2007 @ 4:23 pm

  15. ’ve been meaning to talk about the ESV bible annotations done with the Amazon Mechanical Turk for ages, since I haven’t you’re just going to have to assume that I have and that I’ve noted how cool the ESV is in general…

    Trackback by Silent to the Dark — February 1, 2007 @ 10:38 am

  16. The ESV Bible team crowdsourced their interlinked search database to Amazon’s Mechanical Turk. That turned out quite well for them.

    Trackback by Latest links from delicious — February 1, 2007 @ 10:38 am

  17. I was messing around Mechanical Turk the other day and saw a ton of great ideas. When I first heard of it, I thought it was a way for Amazon to nail down some of its BETA projects but it goes much further than that. You can create your own HITs and build massive amounts of content on the cheap. Hmm. What to build?…

    Trackback by Brokekid.net — March 1, 2007 @ 7:42 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.