User Guide for the Amazon Mechanical Turk Command Line Tools v1.3.0

The Amazon Mechanical Turk Command Line Tools are a set of command line interfaces that allow you to easily build solutions leveraging Amazon Mechanical Turk without writing a line of code. The goals of the Command Line Tools are to:

The Command Line Tools can be run on the following environments:

What's New in Amazon Mechanical Turk Command Line Tools v1.3.0

We have added the following new features to this version of the Command Line Tools:

Amazon Mechanical Turk Concepts and Additional Information

Before getting started, you should first familiarize yourself with the key concepts of Amazon Mechanical Turk by reading the Mechanical Turk Concepts section in the Amazon Mechanical Turk Developer Guide.

For additional information, check out these resources:

  • The Amazon Mechanical Turk Developer Guide provides full details on all functionality available on Amazon Mechanical Turk.
  • The Amazon Mechanical Turk Getting Started Guide provides information about how to get started using Amazon Mechanical Turk. It includes step-by-step information about how to create a HIT using the Command Line Tools.
  • The Command Line Tools Tutorial provides video instructions to help you get started creating HITs.

    • Table of Contents

    1. Features and Benefits
    2. Installing the Command Line Tools
    3. Using the Amazon Mechanical Turk Developer Sandbox for Testing
    4. Overview of the Sample Applications
    5. Running the Hello World Sample Application
    6. Running the Simple Survey Sample Application
    7. Running the Best Image Sample Application
    8. Running the Image Categorization Sample Application
    9. Running the Image Tagging Sample Application
    10. Running the External HIT Sample Application
    11. Running the Web Site Filtering Sample Application
    12. Running the Web Site Filtering Qualification Sample Application
    13. Running the Assign Qualification Sample Application
    14. Running the Quiz Qualification Sample Application
    15. Running the Java Qualification Sample Application
    16. Description of Files Used by the Command Line Tools
    17. Overview of Command Line Tool Operations
    18. Running the Get Balance Operation
    19. Running the Load HITs Operation
    20. Running the Get Results Operation
    21. Running the Approve Work Operation
    22. Running the Reject Work Operation
    23. Running the Review Results Operation
    24. Running the Delete HITs Operation
    25. Running the Update HITs Operation
    26. Running the Extend HITs Operation
    27. Running the Grant Bonus Operation
    28. Running the Block Worker Operation
    29. Running the Unblock Worker Operation
    30. Running the Create Qualification Type Operation
    31. Running the Update Qualification Operation
    32. Running the Assign Qualification Operation
    33. Running the Get Qualification Requests Operation
    34. Running the Approve Qualification Requests Operation
    35. Running the Reject Qualification Requests Operation
    36. Running the Evaluate Qualification Requests Operation
    37. Running the Update Qualification Score Operation
    38. Running the Revoke Qualification Operation
    39. Running the Reset Account Operation
    40. Running the Make Template Operation
    41. Logging and Debugging
    42. Portability of Convenience Files
    43. Comments, Questions, or Feedback

    Features and Benefits

    The Amazon Mechanical Turk Command Line Tools offers you the following features and benefits:

    Feature Benefit
    Bulk load HITs via a Question Template File, an Input File, and a HIT Properties File. No coding necessary. Only simple text file manipulation, using Microsoft Excel to manage your input and simple XML editing.
    Retrieve submitted results as a file. Results can be easily viewed in a text editor or Excel and can be easily parsed for integration to your backend system or into another HIT workflow.
    Bulk approve/reject assignments Results can be easily approved/rejected using files.
    Change properties for HITs that are live Change properties (i.e. reward) for HITs that are already in production.
    Preview your HIT as an HTML file. Designing your HITs takes less time since you can preview the look and feel without loading into Amazon Mechanical Turk.
    Manage your Workers Create and maintain your qualifications to manage your Workers. Prevent undesirable Workers from completing your HITs by using the Block Worker command.

    Return to Table of Contents

    Installing the Command Line Tools

    The installation of the Command Line Tools and other required components is simple and requires minimal configuration.

    If you downloaded and ran the mech-turk-setup.exe file, you have installed and configured the Command Line Tools. When this documentation refers to [Command Line Tools Installation Directory], it assumes the following directory structure:
    Directory] is [Command Line Tools Installation Directory] = mech-turk-tools-1.3.0

    If you downloaded a different file, use the following instructions to install and configure the Command Line Tools.

    The instructions below assume that the downloaded file was uncompressed in a directory that you specified. When the instructions refer to [Command Line Tools Installation Directory], it assumes the following directory structure:
    [Command Line Tools Installation Directory] = [The directory that the downloaded file was uncompressed into]\aws-mturk-clt-1.3.0
    1. Sign up for an Amazon Web Services (AWS) account at the AWS web site.
    2. Sign up for an Amazon Mechanical Turk Requester account at the Requester web site.
    3. Configure the Command Line Tools to use your AWS access identifiers information.

         a. Open the following file: [Command Line Tools Installation Directory]\bin\mturk.properties    b. Set the following properties to reflect your AWS access identifiers:

      access_key=[Your AWS Access Key]
      secret_key=[Your Secret Key]          c. Save and close the file.

    4. If you downloaded the Unix distribution, you must set the MTURK_CMD_HOME environment variable to point to your Command Line Tools installation location.
    5. If you downloaded the Unix distribution, you must set the JAVA_HOME environment variable to point to your Java Standard Edition Runtime Environment (JRE) installation location.
    For advanced users, you can optionally set the MTURK_CMD_HOME environment variable to point to your Command Line Tools installation location so that they can be run from any directory.

    Note: If you did not download a version of the Command Line Tools that contains the Java Standard Edition Runtime Environment (Only available for Windows distribution), you must have the Java JRE 1.5.X (JRE 5). The Command Line Tools are not 100% compatible with JRE 6. You can download the JRE at the following website: http://java.sun.com/javase/downloads/index_jdk5.jsp.

    If you have installed the JRE and elect not to use the included JRE (Windows distribution only), you should set the JAVA_HOME environment variable to point to your JRE installation.

    Return to Table of Contents

    Using the Amazon Mechanical Turk Developer Sandbox for Testing

    By default, the Command Line Tools are configured to work against the production site (http://www.mturk.com). When creating HITs on the Production site, you will need to have a funded Amazon Mechanical Turk account. Visit the following link to fund your account: http://requester.mturk.com/mturk/youraccount.

    Optionally, you can configure the Command Line tools to work against the Developer Sandbox environment. The Developer Sandbox is a simulated environment that allows developers to test their Amazon Mechanical Turk solutions for free. To view HITs created in the sandbox, go to the Developer Sandbox worker site ( http://workersandbox.mturk.com).

    To configure the Command Line Tools to work against the Developer Sandbox:

    1. Open the following file in a text editor:
      [Command Line Tools Installation Directory]\bin\mturk.properties
    2. Comment out the Production site service_url with a "#":
      #service_url=http://mechanicalturk.amazonaws.com/?Service=AWSMechanicalTurkRequester
    3. Uncomment the Developer Sandbox service_url by removing the "#":
      service_url=http://mechanicalturk.sandbox.amazonaws.com/?Service=AWSMechanicalTurkRequester
    4. Save and close the file.
    You can also temporarily override your configuration in your mturk.properties file and work against the Developer Sandbox by using the -sandbox parameter with any of the commands. For example:

    loadHITs -sandbox [other options] This will execute the loadHITs command against the Developer Sandbox environment even if you have configured your command line tools to run against production in your mturk.properties file.

    Return to Table of Contents

    Overview of the Sample Applications

    The Command Line Tools include a set of sample applications that demonstrate various use cases of Amazon Mechanical Turk.

    There are 4 create HIT sample applications and 7 Qualification related sample applications.

    Each create HIT sample application is defined by 3 files:

    Each create HIT sample described in subsequent sections consist of the following steps:
    1. For each row in the Input File, the fields will be inserted into the placeholders of the Question Template File and the HIT will be created in the production site with the properties defined in the HIT Properties File. The # of rows in the Input File will determine how many HITs will be created.
    2. A Success File is generated containing identifiers to the HITs that were created.
    3. The HITs are accepted, worked on and submitted on the Worker sandbox site.
    4. The Success File is used to retrieve the results of the HITs which are then written to a Results File.
    5. The Success File is then used to automatically approve all the HITs and then properly dispose of them.
    Details on the Qualification related sample applications can be found in their respective section.

    For more details on the above files, see section Description of Files Used by the Command Line Tools.

    The following samples have been provided:

    Sample Name Description
    Hello World Create HITs that ask a few simple questions such as "What is the weather like right now in Seattle, WA?"
    Simple Survey Create a HIT that conducts a quick survey on a worker's political party preferences.
    Best Image Create a HIT that asks a worker to choose the best image of three given a set of criteria. Before loading the HIT into Amazon Mechanical Turk, this sample will also demonstrate how to preview the HIT before loading.
    Image Categorization Create HITs asking workers to categorize images.
    Image Tagging Create HITs asking workers to create tags for various images.
    External HIT Create HITs that uses content served on a third party website as the question answer form to ask workers to categorize websites.
    Web Site Filtering Create HITs asking workers to filter out inappropriate websites.
    Assign Qualification Create a simple qualification that does not require a test and assign this qualification to a worker.
    Quiz Qualification Create a qualification with a test that is automatically graded by Amazon Mechanical Turk with an answer key.
    Java Qualification Create a qualification that tests a worker's knowledge of Java where the answers will be scored by you instead of AmazonMechanical Turk. You will retrieve qualification requests from workers and score their tests using an answer key that you define.
    Web Site Filtering Qualification Create a qualification that tests a worker's ability to filter out inappropriate websites.

    Return to Table of Contents

    Running the "Hello World" Sample Application

    With this sample application, you create a few HIT that ask a few simple questions, such as "What is the weather like right now in Seattle, WA?"

    To run this sample:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\helloworld
    2. Then run the following command:
      For Microsoft Windows: run
      For Unix: ./run.sh       You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      --[Initializing]----------
 Input: ..\samples\helloworld\helloworld.input
 Properties: ..\samples\helloworld\helloworld.properties
 Question File: ..\samples\helloworld\helloworld.question
 Preview mode disabled
--[Loading HITs]----------
  Start time: Wed Aug 22 11:18:43 PDT 2007
Created HIT 1: HITId=HXVZ28HXQXQ6VBNA6ZS0
Created HIT 2: HITId=0XBEXK5AN0TZW0GP7VC0
Created HIT 3: HITId=2GCM1PYFZJAPHWYDHZQZ
Created HIT 4: HITId=CAYZVKHRRZRCWMC5XBEZ
Created HIT 5: HITId=EA1ZQ300GK06TQG70VDZYou may see your HIT(s) with HITTypeId 'VWCZXMY21ARZ0KZHYXN0' here:

http://www.mturk.com/mturk/preview?groupId=VWCZXMY21ARZ0KZHYXN0

  End time: Wed Aug 22 11:18:44 PDT 2007
--[Done Loading HITs]----------
  Total load time: 1 seconds.
  Successfully loaded 5 HITs.
    Your HIT has been successfully loaded. You can visit the Amazon Mechanical Turk site with the URL provided in the output to view the HIT.

    To retrieve the results after workers have submitted answers:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\helloworld
    2. Then run the following command:
      For Microsoft Windows: getResults
      For Unix: ./getResults.sh
    3. Open the file called helloworld.results to view the results. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file. The submitted results can be found at the last columns in the file.
    To approve and delete the results:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\helloworld
    2. Then run the following command:
      For Microsoft Windows: approveAndDeleteResults
      For Unix: ./approveAndDeleteResults.sh
    3. You should see the following output:
      You are about to delete 5 HITs.
      To confirm this operation, please press ENTER (or press Ctrl+C to abort):
    4. Press enter.
    5. You should see the following output:
      [0XBEXK5AN0TZW0GP7VC0] Successfully deleted HIT (1/5)
[HXVZ28HXQXQ6VBNA6ZS0] Successfully deleted HIT (2/5)
[2GCM1PYFZJAPHWYDHZQZ] Successfully deleted HIT (3/5)
[CAYZVKHRRZRCWMC5XBEZ] Successfully deleted HIT (4/5)
[EA1ZQ300GK06TQG70VDZ] Successfully deleted HIT (5/5)
--- Finished to delete HITs ---

    Concepts Covered in this Sample

    In this sample, you have successfully loaded your first HITs into Amazon Mechanical Turk using the Command Line Tools.

    The questions are defined in the Input File:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\helloworld
    2. Open the file called helloworld.input in a text editor and you should see the following text: question
      What is the weather like right now in Seattle, WA?
What company makes the Windows operating system?
How many glasses of water do you drink per day?
What's the one thing you've bought that you're happiest with?
What are you thinking right now?
    The HIT layout is defined in the Question Template File:
    1. Open the file called helloworld.question in a text editor.
    2. Note the following section of the XML file containing QuestionForm content: <QuestionContent>
      <Text>${question}</Text>
      </QuestionContent>
    3. The ${question} line of text represents the variable defined in the Input File.
    The HIT properties are defined in the HIT Properties File:
    1. Open the file called helloworld.properties in a text editor.
    2. Note the properties defined in this file which correspond to the properties of the HIT that was created.
    After successfully loading the HITs, a record of the operation is stored in the Success File:
    1. Open the file called helloworld.input.success in a text editor and you should see something like the following text:
       hitid	hittypeid
HXVZ28HXQXQ6VBNA6ZS0	1APZGQSEFW5ZHPDA3YW0
0XBEXK5AN0TZW0GP7VC0	1APZGQSEFW5ZHPDA3YW0
2GCM1PYFZJAPHWYDHZQZ	1APZGQSEFW5ZHPDA3YW0
CAYZVKHRRZRCWMC5XBEZ	1APZGQSEFW5ZHPDA3YW0
EA1ZQ300GK06TQG70VDZ	1APZGQSEFW5ZHPDA3YW0
    2. This file contains the HIT IDs and HIT Type ID that can be used to retrieve your results.

    Return to Table of Contents

    Running the "Simple Survey" Sample Application

    With this sample application, you create a HIT that conducts a quick survey on a worker's political party preferences.

    To run this sample:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\simple_survey
    2. Then run the following command:
      For Microsoft Windows: run
      For Unix: ./run.sh       You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      --[Initializing]----------
 Input: ..\samples\simple_survey\simple_survey.input
 Properties: ..\samples\simple_survey\simple_survey.properties
 Question File: ..\samples\simple_survey\simple_survey.question
 Preview mode disabled
--[Loading HITs]----------
  Start time: Wed Aug 22 11:30:19 PDT 2007
Created HIT 1: HITId=ZVRZHDYPHWDZXV01YXBZ
You may see your HIT(s) with HITTypeId 'XSVZVC432ZXZVFCCDKGZ' here:

 http://www.mturk.com/mturk/preview?groupId=XSVZVC432ZXZVFCCDKGZ

  End time: Wed Aug 22 11:30:20 PDT 2007
--[Done Loading HITs]----------
  Total load time: 1 seconds.
    Your HIT has been successfully loaded. You can visit the Amazon Mechanical Turk site with the URL provided in the output to view the HIT.

    To retrieve the results after workers have submitted their answers:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\simple_survey
    2. Then run the following command:
      For Microsoft Windows: getResults
      For Unix: ./getResults.sh
    3. Open the file called simple_survey.results to view the results. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file. The submitted results can be found at the last columns in the file.
    To approve and delete the results:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\simple_survey
    2. Then run the following command:
      For Microsoft Windows: approveAndDeleteResults
      For Unix: ./approveAndDeleteResults.sh
    3. You should see the following output:
      You are about to delete 1 HITs.
      To confirm this operation, please press ENTER (or press Ctrl+C to abort):
    4. Press enter.
    5. You should see the following output:
      [1] Success Deleting HIT (ZVRZHDYPHWDZXV01YXBZ)

    Concepts Covered in this Sample

    In this sample, a     locale qualification (What is a qualification?) was used:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\simple_survey
    2. Open the file called simple_survey.properties in a text editor.
    3. Note the following section of text:
      qualification.1:00000000000000000071
      qualification.comparator.1:EqualTo
      qualification.locale.1:US
      qualification.private.1:false
    4. The Qualification Requirement above specifies that the worker must be located in the US and that the HIT can be previewed by workers who do not meet this Qualification Requirement.

    Return to Table of Contents

    Running the "Best Image" Sample Application

    With this sample application, you create a HIT that asks a worker to choose the best image of three given a set of criteria. Before loading the HIT into Amazon Mechanical Turk, this sample will also demonstrate how to preview the HIT before loading.

    To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\best_image
    2. Then run the following command to preview the HIT: For Microsoft Windows: run -preview -previewfile ..\samples\best_image\preview.html
      For Unix: ./run.sh -preview -previewfile ../samples/best_image/preview.html

      You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):

      --[Initializing]----------
 Input: ..\samples\best_image\best_image.input
 Properties: ..\samples\best_image\best_image.properties
 Question File: ..\samples\best_image\best_image.question
 Preview mode enabled
Preview file: ..\samples\best_image\preview.html
--[Previewing HITs]----------
    3. Open the preview.html file in a browser. You will be presented with a preview of what the HIT will look like when posted on Amazon Mechanical Turk.
    To actually load this HIT:
    1. Run the following command: For Microsoft Windows: run
      For Unix: ./run.sh
       You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      --[Initializing]----------
 Input: ..\samples\best_image\best_image.input
 Properties: ..\samples\best_image\best_image.properties
 Question File: ..\samples\best_image\best_image.question
 Preview mode disabled
--[Loading HITs]----------
  Start time: Mon Jun 01 14:00:00 PDT 2007
Created HIT 1: HITId=NWQMG5ZCBYQZW6CV4W10
You may see your HIT(s) with HITTypeId 'JXTJ6DXRVYE4VEY3CX60' here:

http://www.mturk.com/mturk/preview?groupId=JXTJ6DXRVYE4VEY3CX60

  End time: Mon Jun 01 14:00:02 PDT 2007
--[Done Loading HITs]----------
  Total load time: 2 seconds.
    Your HIT has been successfully loaded. You can visit the Amazon Mechanical Turk site with the URL provided in the output to view the HIT.

    To retrieve the results after workers have submitted answers:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\best_image
    2. Then run the following command:
      For Microsoft Windows: getResults
      For Unix: ./getResults.sh
    3. Open the file called best_image.results to view the results. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file. The submitted results can be found at the last columns in the file.
    To approve and delete the results:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\best_image
    2. Then run the following command:
      For Microsoft Windows: approveAndDeleteResults
      For Unix: ./approveAndDeleteResults.sh
    3. You should see the following output:
      You are about to delete 1 HITs.
      To confirm this operation, please press ENTER (or press Ctrl+C to abort):
    4. Press enter.
    5. You should see the following output:
      [1] Success Deleting HIT (NYXZHZ062WAZQAHG2A4Z)

    Concepts Covered in this Sample

    In this sample, the     <FormattedContent> functionality in the QuestionForm was used:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\best_image
    2. Open the file called best_image.question in a text editor.
    3. Note the following section of text:
      <FormattedContent><![CDATA[
      ... some HTML that conforms to the XHTML specification
      ]]></FormattedContent>
    4. A subset of HTML tags are allowed to be placed within the FormattedContent tags. Click here for a list of what is allowed.
    Additionally, a system qualification (What is a qualification?) was used:
    1. Open the file called best_image.properties in a text editor.
    2. Note the following section of text:
      qualification.1:000000000000000000L0
      qualification.comparator.1:greaterthan
      qualification.value.1:25
      qualification.private.1:false
    3. The Qualification Requirement above specifies that the worker must have an approval rate of 25% or greater and that the HIT can be previewed by workers who do not meet this Qualification Requirement.

    Return to Table of Contents

    Running the "External HIT" Sample Application

    With this sample application, you create HITs asking workers to categorize websites using a HIT hosted on an external website.

    To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\external_hit
    2. Run the following command: For Microsoft Windows: run
      For Unix: ./run.sh       You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      --[Initializing]----------
 Input: ..\samples\external_hit\external_hit.input
 Properties: ..\samples\external_hit\external_hit.properties
 Question File: ..\samples\external_hit\external_hit.question
 Preview mode disabled
--[Loading HITs]----------
  Start time: Mon Jun 01 14:00:00 PDT 2007
Created HIT 1: HITId=DS3JHQHKTYZGHRZQ13YZ
Created HIT 2: HITId=GWTZYPZEHXWAZRZVMYN0
Created HIT 3: HITId=5GD8YRY52T5ZZDYKRSC0
Created HIT 4: HITId=FSG80S1FYZSZV0ZE4JNZ
Created HIT 5: HITId=Q8QM70ZNR3DZRC43CZZZ
You may see your HIT(s) with HITTypeId 'Y0ZAWKWKRYCZNMRE7W2Z' here:

http://www.mturk.com/mturk/preview?groupId=Y0ZAWKWKRYCZNMRE7W2Z

  End time: Mon Jun 01 14:00:03 PDT 2007
--[Done Loading HITs]----------
  Total load time: 3 seconds.
  Successfully loaded 5 HITs.
    Your HITs have been successfully loaded. You can visit the Amazon Mechanical Turk site with the URL provided in the output to view the HITs.

    To retrieve the results after workers have submitted their answers:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\external_hit
    2. Then run the following command:
      For Microsoft Windows: getResults
      For Unix: ./getResults.sh
    3. Open the file called external_hit.results to view the results. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file. The submitted results can be found at the last columns in the file.
    To approve and delete the results:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\external_hit
    2. Then run the following command:
      For Microsoft Windows: approveAndDeleteResults
      For Unix: ./approveAndDeleteResults.sh
    3. You should see the following output:
      You are about to delete 5 HITs.
      To confirm this operation, please press ENTER (or press Ctrl+C to abort):
    4. Press enter.
    5. You should see the following output:
      [1] Success Deleting HIT (NZFRWN8QBXAZ41151WQZ)
      [2] Success Deleting HIT (TJ3MPGYNNZEZQRZRKWC0)
      [3] Success Deleting HIT (KT9ZVZYM1W2ZYBYD3Y5Z)
      [4] Success Deleting HIT (1J0ERTZQYX2671Z6BZW0)
      [5] Success Deleting HIT (0AXZNTHEVXH8XKYZPY50)

    Concepts Covered in this Sample

    In this sample, the HIT itself is hosted externally (    ExternalQuestion) outside of Amazon Mechanical Turk.
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\external_hit
    2. Open the file called external_hit.question in a text editor.
    3. Note the following section of text:
      <ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
      <ExternalURL>http://s3.amazonaws.com/mturk/samples/sitecategory/externalpage.htm?url=${helper.urlencode($urls)}</ExternalURL>
      <FrameHeight>400</FrameHeight>
      </ExternalQuestion>
    4. In this example, the externalpage.htm web page is being configured by the request parameter called url.
    5. The ${helper.urlencode($urls)} is a special feature that can be used to encode the raw URL from your input file (i.e. $urls) so that it can be included as a request parameter.
    6. Open the file called externalpage.htm in an HTML editor or a text editor to view the source. This simple HTML file demonstrates how to submit results back to Amazon Mechanical Turk as well as implementing the Javascript necessary to have the same file work in the Developers Sandbox and the Production system.

    Return to Table of Contents

    Running the "Image Categorization" Sample Application

    With this sample application, you create HITs asking workers to categorize images based on a list of pre-defined categories. This sample comes with a list of 200 images and asks 3 workers to evaluate each image. You will be creating 200 HITs with 3 assignments each with a total of up to 600 assignments.

    To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\image_category
    2. Run the following command: For Microsoft Windows: run
      For Unix: ./run.sh       You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      --[Initializing]----------
 Input: ..\samples\image_category\image_category.input
 Properties: ..\samples\image_category\image_category.properties
 Question File: ..\samples\image_category\image_category.question
 Preview mode disabled
--[Loading HITs]----------
  Start time: Fri Nov 23 15:52:14 PST 2007
Created HIT 1: HITId=J9CT5ZC4JWF6QKC7DRN0
Created HIT 2: HITId=0WQJY6XD5YXZV5YK9Z2Z
Created HIT 3: HITId=J2R8RS5MKZXZR69WCZ9Z
Created HIT 4: HITId=KRMTVGY0HVXGPTZ37GRZ
Created HIT 5: HITId=XHFPY48G5XTARGNDP1M0
Created HIT 6: HITId=J9CRZKZ28J0ZT0XGMG4Z
...

You may see your HIT(s) with HITTypeId '1HFZ6K8MJ8EZYQYBH0YZ' here:

http://www.mturk.com/mturk/preview?groupId=1HFZ6K8MJ8EZYQYBH0YZ

  End time: Fri Nov 23 15:52:20 PST 2007
--[Done Loading HITs]----------
  Total load time: 82 seconds.
  Successfully loaded 200 HITs.
    Your HITs have been successfully loaded. You can visit the Amazon Mechanical Turk site with the URL provided in the output to view the HITs.

    To retrieve the results after workers have submitted their answers:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\image_category
    2. Run the following command:
      For Microsoft Windows: getResults
      For Unix: ./getResults.sh
    3. Open the file called image_category.results to view the results. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file. The submitted results can be found at the last columns in the file.
    4. To view a summary report of the results, run the following command:
      For Microsoft Windows: generateResultsSummary
      For Unix: ./generateResultsSummary.sh
    5. Open the file called image_category.summary to view the summarized results. This file shows the majority answer for each question, the percentage of workers who selected this answer, and how many workers selected the answer out of the number who worked on the HIT. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file.
    To approve and delete the results:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\image_category
    2. Then run the following command:
      For Microsoft Windows: approveAndDeleteResults
      For Unix: ./approveAndDeleteResults.sh
    3. You should see the following output:
      You are about to delete 200 HITs.
      To confirm this operation, please press ENTER (or press Ctrl+C to abort):
    4. Press enter.
    5. You should see the following output:
      [1] Success Deleting HIT (NZFRWN8QBXAZ41151WQZ)
      [2] Success Deleting HIT (TJ3MPGYNNZEZQRZRKWC0)
      [3] Success Deleting HIT (KT9ZVZYM1W2ZYBYD3Y5Z)
      [4] Success Deleting HIT (1J0ERTZQYX2671Z6BZW0)
      [5] Success Deleting HIT (0AXZNTHEVXH8XKYZPY50)
      ...

    Concepts Covered in this Sample

    The image categorization sample is a good example of how you can use multiple assignments to determine the correct answer for a HIT.

    The number of assignments per HIT is defined in the .property file:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\image_category
    2. Open the file called image_category.properties in a text editor and you should see the following text: 
      assignments:3
    3. When your HIT asks questions that requires multiple choice answers, you can ask several workers to complete the same HIT so you can easily compare their answers.

    Return to Table of Contents

    Running the "Image Tagging" Sample Application

    With this sample application, you create HITs asking workers to create tags for a list of images. This sample comes with a list of 250 images. Given an image, a worker will be asked to give tags, or keywords, that are relevant to the image. Workers are required to give at least 3 tags but they can optionally assign up to 5 tags for each image. You will be creating a total of 250 HITs with this sample.

    To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\image_tagging
    2. Run the following command: For Microsoft Windows: run
      For Unix: ./run.sh       You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      --[Initializing]----------
 Input: ..\samples\image_tagging\image_tagging.input
 Properties: ..\samples\image_tagging\image_tagging.properties
 Question File: ..\samples\image_tagging\image_tagging.question
 Preview mode disabled
--[Loading HITs]----------
  Start time: Fri Nov 23 15:52:14 PST 2007
Created HIT 1: HITId=Y08ZRHY35VKWXJ0MJZ3Z
Created HIT 2: HITId=RKFZSGWZ9XSTR01ERKR0
Created HIT 3: HITId=AZZZPGY97RV65FZ8Q1FZ
Created HIT 4: HITId=6KHZQAZA8T9ASAY9Q330
Created HIT 5: HITId=71HJTJY0CXNEX6D5EKE0
Created HIT 6: HITId=5XFGW4YAY9RTR7C89WTZ
...

You may see your HIT(s) with HITTypeId 'SKZZTCZDDY8ZZTYA2YCZ' here:

http://www.mturk.com/mturk/preview?groupId='SKZZTCZDDY8ZZTYA2YCZ'

--[Done Loading HITs]----------
  Total load time: 102 seconds.
  Successfully loaded 250 HITs.
    Your HITs have been successfully loaded. You can visit the Amazon Mechanical Turk site with the URL provided in the output to view the HITs.

    To retrieve the results after workers have submitted their answers:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\image_tagging
    2. Then run the following command:
      For Microsoft Windows: getResults
      For Unix: ./getResults.sh
    3. Open the file called image_tagging.results to view the results. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file. The submitted results can be found at the last columns in the file.
    To approve and delete the results:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\image_tagging
    2. Then run the following command:
      For Microsoft Windows: approveAndDeleteResults
      For Unix: ./approveAndDeleteResults.sh
    3. You should see the following output:
      You are about to delete 250 HITs.
      To confirm this operation, please press ENTER (or press Ctrl+C to abort):
    4. Press enter.
    5. You should see the following output:
      [1] Success Deleting HIT (Y08ZRHY35VKWXJ0MJZ3Z)
      [2] Success Deleting HIT (RKFZSGWZ9XSTR01ERKR0)
      [3] Success Deleting HIT (AZZZPGY97RV65FZ8Q1FZ)
      [4] Success Deleting HIT (6KHZQAZA8T9ASAY9Q330)
      [5] Success Deleting HIT (71HJTJY0CXNEX6D5EKE0)
      ...

    Concepts Covered in this Sample

    In this sample, the HIT is designed to ask workers to supply at least 3 tags up to 5 tags. In the HIT, the first three questions are required and the last two questions are optional.
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\image_tagging
    2. Open the file called image_tagging.question in a text editor.
    3. Note the following section of:

      #set( $minimumNumberOfTags = 3 )
      #foreach( $tagNum in [1..5] )
      <Question>
      <QuestionIdentifier>tag${tagNum}</QuestionIdentifier>
      #if( $tagNum <= $minimumNumberOfTags)
      <IsRequired>true</IsRequired>
      #else
      <IsRequired>false</IsRequired>
      #end
    4. This code segment takes advantage of a free open source templating language called Velocity. In this example, we use a simple if-else statement to determine whether a question should be required. This creates 5 questions where the first 3 are required. You can re-use this code segment for your own application and change the numbers as appropriate.
    5. For more information about how to use Velocity, go to the Velocity Reference Guide.

    Return to Table of Contents

    Running the "Website Filtering" Sample Application

    With this sample application, you create HITs asking workers to identify adult porn sites. There are 100 websites for workers to evaluate. Each HIT asks 3 different workers to evaluate one website. This sample creates a total of 100 HITs with 3 assignments each. Workers are required to have passed the qualification test for Qualified to Identify Adult Porn Websites before they can work on the HITs.

    To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\site_filter
    2. Run the following command: For Microsoft Windows: run
      For Unix: ./run.sh       You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      --[Initializing]----------
 Input: ..\samples\site_filter\site_filter.input
 Properties: ..\samples\site_filter\site_filter.properties
 Question File: ..\samples\site_filter\site_filter.question
 Preview mode disabled
--[Loading HITs]----------

Created HIT 1: HITId=KYSRWE4Z8Y5ZZ708YZP0
Created HIT 2: HITId=F9EZXC4082PEZAZFNYZZ
Created HIT 3: HITId=JYF6T3ZM2KYZY14S1XKZ
Created HIT 4: HITId=R0FZQ5ZY1X0Z0D9KDWCZ
Created HIT 5: HITId=EXDYN7YGR0S22ZS1FJP0
Created HIT 6: HITId=W08T2Q9QXW7ZXBY4VHF0
...

You may see your HIT(s) with HITTypeId 'WBTP3SDX5TB8SYZFCZ0Z' here:

http://www.mturk.com/mturk/preview?groupId='WBTP3SDX5TB8SYZFCZ0Z'

--[Done Loading HITs]----------
  Total load time: 41 seconds.
  Successfully loaded 100 HITs.
    Your HITs have been successfully loaded. You can visit the Amazon Mechanical Turk site with the URL provided in the output to view the HITs.

    To retrieve the results after workers have submitted their answers:

    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\site_filter
    2. Then run the following command:
      For Microsoft Windows: getResults
      For Unix: ./getResults.sh
    3. Open the file called site_filter.results to view the results. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file. The submitted results can be found at the last columns in the file.
    4. To view a summary report of the results, run the following command:
      For Microsoft Windows: generateResultsSummary
      For Unix: ./generateResultsSummary.sh
    5. Open the file called site_filter.summary to view the summarized results. This file shows the majority answer for each question, the percentage of workers who selected this answer, and how many workers selected the answer out of the number who worked on the HIT. The file contains tab-delimited data so it is best viewed in Microsoft Excel or any other application suitable for viewing this kind of file.
    To approve and delete the results:
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\site_filter
    2. Then run the following command:
      For Microsoft Windows: approveAndDeleteResults
      For Unix: ./approveAndDeleteResults.sh
    3. You should see the following output:
      You are about to delete 100 HITs.
      To confirm this operation, please press ENTER (or press Ctrl+C to abort):
    4. Press enter.
    5. You should see the following output:
      [1] Success Deleting HIT (KYSRWE4Z8Y5ZZ708YZP0)
      [2] Success Deleting HIT (F9EZXC4082PEZAZFNYZZ)
      [3] Success Deleting HIT (JYF6T3ZM2KYZY14S1XKZ)
      [4] Success Deleting HIT (R0FZQ5ZY1X0Z0D9KDWCZ)
      [5] Success Deleting HIT (EXDYN7YGR0S22ZS1FJP0)
      ...

    Concepts Covered in this Sample

    In this sample, the website to be evaluated is embedded in the HIT using the <FormattedContent> functionality in the QuestionForm.
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\site_filter
    2. Open the file called site_filter.question in a text editor.
    3. Note the following section of text:
      <FormattedContent><![CDATA[
      <iframe src="${urls}" width="100%" height="400">This text is necessary to ensure proper XML validation. </iframe>
      ]]></FormattedContent>
    4. The <iframe> tag allows you to display content from another website directly in the HIT itself. There must be some text in between the <iframe> and </iframe> to conform with XML validation.

    Return to Table of Contents

    Running the "Assign Qualification" Sample Application

    With this sample application, you will create a simple Qualification that does not require a test and assign this Qualification to a Worker. This is the easiest way to use Qualifications to track the performance of Workers and manage who can complete your HITs.

    Note: By default, qualifications created in the samples are in the Developer Sandbox instead of the Production environment.

    To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\assign_qualification
    2. The first step is to create the Qualification that you will be assigning to workers.
    3. Run the following command: For Microsoft Windows: run
      For Unix: ./run.sh
    4. You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      Created qualification type: KYJ4GZZ5G3M6TDCEWYF0
      You can request the qualification here: http://workersandbox.mturk.com/mturk/requestqualification?qualificationId=KYJ4GZZ5G3M6TDCEWYF0
    Your Qualification has been successfully created. You can visit the Amazon Mechanical Turk     Developer Sandbox worker site with the URL provided in the output to view the Qualification.

    The next step in this sample is to assign this newly created Qualification to a worker. Before proceeding further, you should have a worker ID that can be used. You can find the worker ID by opening one of the results files from any of the create HIT samples above. To assign the Qualification to the worker with an initial score of 100:

    1. Run the following command: For Microsoft Windows: assignQual -workerid [your workerid] -score 100
      For Unix: ./assignQual -workerid [your workerid] -score 100
    2. You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      Assigned qualification KYJ4GZZ5G3M6TDCEWYF0 to A3C4G8DMXFG5PQ with value 100

    Concepts Covered in this Sample

    In this sample, you created a Qualification that did not require a test.
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\assign_qualification
    2. Open the file called qualification.properties in a text editor.
    3. Note the following section of text:
      name=Qualification Assign
      description=This qualification is assigned to workers
      keywords=assignqual
    4. The name and description parameters are the only required parameters to create a basic Qualification.
    5. When the Qualification was successfully created, a Qualification Success file was generated. Open the file called qualification.properties.success
    6. The file contains the Qualification type ID that was created: qualtypeid
      KYJ4GZZ5G3M6TDCEWYF0

    Return to Table of Contents

    Running the "Quiz Qualification" Sample Application

    With this sample application, you create will create a Qualification with a test that is automatically graded by Amazon Mechanical Turk with an answer key.

    Note: By default, qualifications created in the samples are in the Developer Sandbox instead of the Production environment.

    To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\quiz_qual
    2. Run the following command: For Microsoft Windows: run
      For Unix: ./run.sh
    3. You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      Created qualification type: 9WKZ0KXWCBAE3WZDFZP0
      You can take the test here: http://workersandbox.mturk.com/mturk/requestqualification?qualificationId=9WKZ0KXWCBAE3WZDFZP0
    Your Qualification has been successfully created. You can visit the Amazon Mechanical Turk     Developer Sandbox worker site with the URL provided in the output to view the Qualification.

    Since this Qualification has a test as well as an answer key, you will be able to complete the Qualification test and be immediately notified of your results.

    Concepts Covered in this Sample

    In this sample, you created a Qualification that required a test.
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\quiz_qual
    2. Open the file called qualification.question in a text editor.
    3. This file contains the QuestionForm XML for the test. Notice that the format of this file is the same as the Question file used to create HITs.
    4. Open the file called qualification.properties in a text editor.
    5. Note the following section of text:
      retrydelayinseconds=3600

# Workers will have 15 minutes to complete this test. 15 minutes = 60 seconds * 15 minutes = 900
testdurationinseconds=900
    6. The retrydelayinseconds parameter defines how much time needs to pass before a worker can retry the test.
    7. The testdurationinseconds parameter defines how much time a worker has to complete the test.
    Additionally, in this sample, you also provided an answer key for the test above which allows Amazon Mechanical Turk to automatically grade the test and assign the worker a Qualification score.
    1. Open the file called qualification.answer in a text editor.
    2. This file contains the XML for the answer key. Notice that the format of this file is similar to the QuestionForm.
    3. Note the following section of text:
      <Question>
<QuestionIdentifier>question1</QuestionIdentifier>
<AnswerOption>
  <SelectionIdentifier>1b</SelectionIdentifier>
  <AnswerScore>10</AnswerScore>
</AnswerOption>
</Question>
    4. The above snippet indicates that the correct answer for question1 is 1b and will be assigned a score of 10.
    5. There is an answer that corresponds to each question. For this sample, each answer is assigned a score of 10. The total score that will be assigned to a worker is the sum of all correctly answered questions.

    Return to Table of Contents

    Running the "Java Qualification" Sample Application

    With this sample application, you will create a Qualification that tests a worker's knowledge of Java where the answers will be scored by you instead of Amazon Mechanical Turk. You will retrieve Qualification requests from workers and score their tests using an answer key that you define.

    Note: By default, qualifications created in the samples are in the Developer Sandbox instead of the Production environment.

    This sample uses the Command Line Tools' simple EvaluateQualificationRequests command that will allow you to configure an answer key that the worker must answer all correctly in order to be granted the Qualification. To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\java_qual_test
    2. Run the following command: For Microsoft Windows: run
      For Unix: ./run.sh
    3. You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      Created qualification type: YTQJWWYVWTFZG2YZXY20
      You can take the test here: http://workersandbox.mturk.com/mturk/requestqualification?qualificationId=YTQJWWYVWTFZG2YZXY20
    Your Qualification has been successfully created. You can visit the Amazon Mechanical Turk     Developer Sandbox worker site with the URL provided in the output to view the Qualification.

    Since this Qualification has a test, you should complete the Qualification test so that a Qualification request is created.

    The next step in this sample is to grade the Qualification request:

    1. Run the following command to preview the outcome of grading. This step will not actually grant or reject the Qualification requests : For Microsoft Windows: evaluateRequests -preview
      For Unix: ./evaluateRequests.sh -preview
    2. You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      Preview flag is set. Qualification requests will not be approved or rejected.
---[Worker A3C4G8DMFSG5PQ]--------------------------------------------------------------
Question question1:CORRECT [The answer key 'false' matches the given answer 'false']
Question question2:CORRECT [The answer key 'true' matches the given answer 'true']
Question question3:CORRECT [The answer key '1' matches the given answer '1']
Question question4:CORRECT [The answer key 'true' matches the given answer 'true']
Question question5:CORRECT [The answer key '0' matches the given answer '0']
Worker A3C4G8DMFSG5PQ has PASSED your test and scored 100
---------------------------------------------------------------------------------------
    3. If the preview results look good, you can turn preview off and grant/reject the Qualification requests by running the following command: For Microsoft Windows: evaluateRequests
      For Unix: ./evaluateRequests.sh
    4. You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output): 
      ---[Worker A3C4G8DMFSG5PQ]--------------------------------------------------------------
Question question1:CORRECT [The answer key 'false' matches the given answer 'false']
Question question2:CORRECT [The answer key 'true' matches the given answer 'true']
Question question3:CORRECT [The answer key '1' matches the given answer '1']
Question question4:CORRECT [The answer key 'true' matches the given answer 'true']
Question question5:CORRECT [The answer key '0' matches the given answer '0']
Worker A3C4G8DMFSG5PQ has PASSED your test and scored 100
Qualification request granted.
---------------------------------------------------------------------------------------
    5. The worker above has been granted the Qualification.

    Concepts Covered in this Sample

    In this sample, you created a Qualification that required a test where the results are scored by you instead of Amazon Mechanical Turk.
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\java_qual_test
    2. Open the file called qualification.answerkey in a text editor.
    3. Note the following section of text:
      question1=false
      question2=true
      question3=1
      question4=true
      question5=0
    4. The format of the file is simply [QuestionIdentifier]=[correct SelectionIdentifier] for each question in the qualification.question file.
    5. This method allows you to score answers that are provided in the FreeTextAnswer format.

    Return to Table of Contents

    Running the "Website Filtering Qualification" Sample Application

    With this sample application, you will create a Qualification that tests a worker's ability to identify adult porn sites. This qualification has a test with an answer key that can be automatically graded by Amazon Mechanical Turk

    Note: By default, qualifications created in the samples are in the Developer Sandbox instead of the Production environment.

    To run this sample:

    1. Navigate to the following directory: [Command Line Tools Installation Directory]\samples\site_filter_qual
    2. Run the following command: For Microsoft Windows: createQualification
      For Unix: ./createQualification.sh
    3. You should see the following output (Note: any IDs, dates or times are illustrative and will differ from your output):
      Created qualification type: G2KZ1182ZRHZRDYJA880
      You can take the test here: http://workersandbox.mturk.com/mturk/requestqualification?qualificationId=G2KZ1182ZRHZRDYJA880
    Your Qualification has been successfully created. You can visit the Amazon Mechanical Turk     Developer Sandbox worker site with the URL provided in the output to view the Qualification.

    Since this Qualification has a test as well as an answer key, you will be able to complete the Qualification test and be immediately notified of your results.

    Concepts Covered in this Sample

    In this sample, similar to the Quiz Qualification, you have created a qualification that requires a test.
    1. Navigate to the following directory:
      [Command Line Tools Installation Directory]\samples\site_filter_qual
    2. Open the file called site_filter_qual.question in a text editor.
    3. This file contains the QuestionForm XML for the test. Notice that the format of this file is the same as the Question file used to create HITs.
    4. In this test, you're essentially asking workers to do the same thing as they would when they work on your Website Filtering HITs. You're requiring workers to evaluate 10 websites and determine whether they're adult porn sites.
    5. A worker's performance on this qualification test should act as a good indicator as to whether they will understand your instructions when they work on your Website Filtering HITs.

    Return to Table of Contents

    Description of Files Used by the Command Line Tools

    The following table describes the key files used by the Command Line Tools.

    Input File Filename Description
    MTurk Properties mturk.properties
    • A text file containing specific information on your AWS Identifiers as well as the Amazon Mechanical Turk service endpoint.
    • This file must exist in the current directory from where you are running the Command Line Tools.
    Input File [your filename].input
    • A tab delimited text file containing the dynamic fields that is merged with your Question Template and HIT Properties files.
    • The first row of the file contains the field headings.
    • Subsequent rows represents the custom field values for the HITs to be loaded.
    • Each row represents one HIT. (e.g. 1000 data rows = 1000 HITs)
    HIT Properties File [your filename].properties
    • A text file defining the properties of the HITs to be created.
    • title parameter defines the title of the HIT.
    • description parameter defines the description of the HIT.
    • keywords parameter defines the key words to associate with the HIT.
    • reward parameter defines the reward for the HIT in $0.00 USD.
    • assignments parameter defines the maximum number of assignments to make available for this HIT.
    • annotation parameter defines a value that you can use to uniquely identify this HIT from the others.
    • The annotation parameter can be merged with values from the Input File by using the following syntax "${the field name}". Apache Velocity is used to perform the merge. Refer to http://velocity.apache.org for advanced merge syntax.
    • assignmentduration parameter defines how much time a worker has to complete the assignment.
    • hitlifetime parameter defines how long the HIT will be active before it expires.
    • autoapprovaldelay parameter defines how much time will pass before a submitted assignment is automatically approved.
    • To specify Qualification Requirements for your HIT, you can use the following convention:
      • qualification.1 parameter defines the Qualification Type ID.
      • qualification.comparator.1 parameter defines the comparator to use.
      • qualification.value.1 parameter defines the integer value for the comparator to use.
      • qualification.locale.1 parameter defines the locale value to use.
      • qualification.private.1 parameter defines if the HIT can be previewed by workers who not meet the qualification requirements.
      • To specify more Qualification requirements, simply increment the .1 suffix to .2 up to .X.
      • For more details on the parameters of a HIT, refer to the Developer Guide.
    Question Template File [your filename].question
    • A text file containing the QuestionForm XML that defines your HIT.
    • Your QuestionForm can be made into a template and merged with values from your Input File by using the following syntax "${the field name}". Apache Velocity is used to perform the merge. Refer to http://velocity.apache.org for advanced merge syntax.
    Success File [your Input File file name].success
    • A tab delimited text file containing the HIT IDs and the HIT Type IDs that were successfully loaded into Amazon Mechanical Turk.
    • This file is autogenerated when HITs are loaded using the Command Line Tools.
    • This file is used to retrieve the results of the HITs that have been loaded.
    Failure File [your Input File file name].failure
    • A tab delimited text file containing the rows from the Input File that failed to load into Amazon Mechanical Turk.
    • This file is autogenerated when HITs are loaded using the Command Line Tools.
    • This file can be fixed, renamed and used to load the HITs that failed.
    Output File [your Output file name]
    • A tab delimited text file containing the results of submitted HITs that are retrieved from Amazon Mechanical Turk.
    • Contains all data related to the HIT and assignments submitted.
    • The last column contains a tab delimited set of question/answer pairs for each question field defined in your QuestionForm.
    Qualification Properties File [your qualification file name].properties
    • A text file defining the properties of the Qualification to be created.
    • name parameter defines the name of the Qualication.
    • description parameter defines the description of the Qualification.
    • keywords parameter defines the key words to associate with the Qualification.
    • retrydelayinseconds parameter defines the minimum amount of time required to pass before a Worker can re-request the Qualification.
    • testdurationinseconds parameter defines the time allowed for the Worker to complete the Qualification test (if a test exists).
    • autogranted parameter defines if the Qualification should be autogranted upon request. Valid values are "true" and "false". This parameter is only valid if a Qualification does not have a test associated with it.
    • autograntedvalue parameter defines the default Qualification score to be assigned to the Worker if the Qualification is to be auto granted.
    Qualification Question File [your qualification file name].question
    • A text file containing the QuestionForm XML that defines your Qualification test.
    • This file is similar to the HIT Question File except that templating is not applicable.
    Qualification Answer File [your qualification file name].answer
    • A text file containing the AnswerKey XML that defines the answer key for your Qualification test that will be automatically scored by the Amazon Mechanical Turk system.
    Qualification Command Line Tools Answer Key File [your qualification file name].answerkey
    • A text file containing name/value pairs of your questions and expected answer.
    • The format is simply questionid=expected answer value
    • This answer key provides another method to evaluate Qualification requests by giving you control over when requests are granted. This format also provides more flexibility by allowing you to evaluate FreeTextAnswer answers which can not currently be auto graded by Amazon Mechanical Turk.

    Return to Table of Contents

    Overview of Command Line Tool Operations

    The Amazon Mechanical Turk Command Line Tools offers you the following features and benefits:

    Operation Description
    Get Balance Returns your Requester account balance.
    Load HITs Loads HITs into Amazon Mechanical Turk.
    Get Results Retrieves the results of submitted HITs from Amazon Mechanical Turk.
    Approve Work Approves assignments submitted by Workers via Amazon Mechanical Turk.
    Reject Work Rejects assignments submitted by Workers via Amazon Mechanical Turk.
    Review Results Approves or rejects assignments from a file.
    Delete HITs Deletes your HITs created on Amazon Mechanical Turk.
    Update HITs Updates certain properties of HITs (i.e. reward) that are already on Amazon Mechanical Turk.
    Extend HITs Add time or assignments to your HITs.
    Grant Bonus Give a bonus to a Worker who has done work for you.
    Block Worker Blocks a Worker from working on your HITs.
    Unblock Worker Unblocks a Worker from working on your HITs.
    Create Qualification Type Creates a Qualification that can be used for your HITs on Amazon Mechanical Turk.
    Update Qualification Type Updates the properties of your Qualification.
    Assign Qualification Assigns your Qualification to a Worker without them requesting the Qualification.
    Get Qualification Requests Retrieve the Qualification requests from Workers for your Qualifications.
    Approve Qualification Requests Approve Workers Qualification requests.
    Reject Qualification Requests Reject Workers Qualification requests.
    Evaluate Qualification Requests Evaluate the answers submitted to approve/reject Qualification requests.
    Update Qualification Scores Update the Qualification scores for Workers.
    Revoke Qualification Revoke a Qualification from a Worker.
    Reset Account Delete all existing HITs from your account.
    Make Template Make a copy of a sample application.

    Return to Table of Contents

    Running the Get Balance Operation

    This operation returns your Requester account balance.

    To run the getBalance operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: getBalance
      For Unix: ./getBalance.sh

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox

    Return to Table of Contents

    Running the Load HITs Operation

    This operation loads HITs into Amazon Mechanical Turk.

    To run the loadHITs operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: loadHITs [your options]
      For Unix: ./loadHITs.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -label [label name] Define a label to attach to this set of HITs that are being loaded. no -label 2006-12-12-contracts
    -input [Input File filename] Specify the Input File. yes -input helloworld.input
    -question [Question File filename] Specify the Question File. yes -question helloworld.question
    -properties [HIT Properties File filename] Specify the HIT Properties File. yes -properties helloworld.properties
    -preview Creates an HTML preview of the HIT instead of loading it into Amazon Mechanical Turk. If -previewfile is not specified, this operation will create a file called preview.html in the current directory. no -preview
    -previewfile [filename of preview HTML file] Specify the file name of the preview HTML file. no. Only useful if -preview is used -previewfile helloworld.html
    -maxhits Specify the maximum number of HITs to create (used for testing purposes if you don't want to load the entire Input File). no -maxhits 1
    -help or -h Print Help for this operation. no -help or -h

    Note: When you run LoadHITs multiple times in the same directory, the latest .success file will override the .success file from the previous run that shares the same name. You can use the -label parameter to give a unique name to the .success file for each run so that you can retain the .success files from all runs.

    Return to Table of Contents

    Running the Get Results Operation

    This operation retrieves the results of submitted HITs from Amazon Mechanical Turk.

    To run the getResults operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: getResults [your options]
      For Unix: ./getResults.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -successfile [Success File filename] Specify the Success File containing the HITs to be retrieved. yes -successfile helloworld.success
    -outputfile [Output File filename] Specify the Output File. yes -outputfile helloworld.results
    -namevaluepair Specify the answers to be comma separated name value pairs for multiple questions. This is the old output format. yes -outputfile helloworld.results
    -help or -h Print Help for this operation. no -help or -h

    When you run GetResults multiple times in the same directory, the latest .results file will override the .results file from the previous GetResults run.

    Each HIT in this file has a link to your Manage HITs page in the Requester web site.. Use these links if you want to manually reject assignments, pay bonuses, or send emails to Workers.

    Note: When a HIT has multiple questions for each assignment, each answer is returned in its own column, rather than the old format of a comma separate list of answers unless you specify the -namevaluepair parameter.

    This operation returns metrics for the results in the file. These metrics show:

    You will see output similar to the following on the screen.
    Results have been written to file '..\samples\site_filter\site_filter.results'.
    Assignments completed: 30/30 (100%)
    Time elapsed: 0:05:16 (h:mm:ss)
    Average submit time: 13.7 seconds

    Return to Table of Contents

    Running the Approve Work Operation

    This operation approves assignments submitted by workers via Amazon Mechanical Turk.

    To run the approveWork operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: approveWork [your options]
      For Unix: ./approveWork.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -assignment [assignment IDs] Specify the assignment ID to approve. For multiple assignments, separate each assignment ID with a comma and enclosed in quotation marks. yes if -approvefile isn't chosen -assignment "SYSZH6HTMKFG2ZDECWS0,KYRZSX0F7154T054AZT0"
    -approvefile [filename containing list of assignment IDs] Specify the filename of a text file containing a list of assignment IDs and optionally approval comments.
    • First row must contain column header named "assignmentIdToApprove"
    • Optional column header following named "assignmentIdToApproveComment"
    • Fields should be separated with a tab and comments should be enclosed in quotes ("")
    		 yes if -assignment isn't chosen. -approvefile helloworld_approve.txt
    -successfile Specify the filename of a .success file that contains a list of HIT IDs and HIT type IDs. The operation will approve all of the assignments for the HITs in the file. no -successfile image_category.success
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Reject Work Operation

    This operation rejects assignments submitted by workers via Amazon Mechanical Turk.

    To run the rejectWork operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: rejectWork [your options]
      For Unix: ./rejectWork.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -assignment [assignment IDs] Specify the assignment ID to reject. For multiple assignments, separate each assignment ID with a comma and enclosed in quotation marks. yes if -rejectfile isn't chosen -assignment "SYSZH6HTMKFG2ZDECWS0,KYRZSX0F7154T054AZT0"
    -rejectfile [filename containing list of assignment IDs] Specify the filename of a text file containing a list of assignment IDs and optionally rejection comments.
    • First row must contain column header named "assignmentIdToReject"
    • Optional column header following named "assignmentIdToRejectComment"
    • Fields should be separated with a tab and comments should be enclosed in ""
    		 yes if -assignment isn't chosen. -rejectfile helloworld_reject.txt
    -successfile Specify the filename of a .success file that contains a list of HIT IDs and HIT type IDs. The operation will reject all of the assignments for the HITs in the file. no -successfile image_category.success
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Review Results Operation

    This operation allows you to accept and reject multiple assignments. Use the results file produced by the Get Results operation to specify which assignments to reject. All other assignments in the file will be approved.

    To run the reviewResults operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the getResults command:
      For Microsoft Windows: getResults
      For Unix: ./getResults.sh
    3. Open the output file.
    4. For each assignment that you want to reject, type any character in the "reject" column for that assignment.
    5. Specify this file as the -resultsfile option in the command.
    6. Run the reviewResults command:
      For Microsoft Windows: reviewResults [your options]
      For Unix: ./reviewResults.sh [your options]
    Note: Any mark in the "reject" column causes the assignment to be rejected. Any unmarked assignments are accepted.

    Options

    Option Definition Required Example
    -help or -h Print Help for this operation. no --help or -h
    -resultsfile [filename] Specifies the results file that was generated by a call to the getResults command. You must mark the "reject" column of the assignments to reject. All other assignments will be approved. yes -resultsfile helloworld.results
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox

    Return to Table of Contents

    Running the Delete HITs Operation

    This operation deletes your HITs created on Amazon Mechanical Turk.

    To run the deleteHITs operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: deleteHITs [your options]
      For Unix: ./deleteHITs.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -successfile [Success File filename] Specify the Success File containing the HITs to be deleted. yes -successfile helloworld.success
    -force Bypasses any confirmations around deleting the HITs. HITs are deleted immediately. no -force
    -approve Automatically approves any assignments that have been submitted and have not been approved or rejected. Assignments that are in the review or reviewing states cannot be deleted. no -approve
    -expire Automatically expires any HITs that are still available to workers on Amazon Mechanical Turk. Live HITs cannot be deleted. no -expire
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Update HITs Operation

    This operation updates properties of HITs that are already on Amazon Mechanical Turk.

    All properties of a HIT Type (except for Reward) can be modified for a HIT. Properties such as:

    The reward property can only be updated for HITs that do not have any assignments that have been accepted, submitted, approved or rejected.

    To run the updateHITs operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: updateHITs [your options]
      For Unix: ./updateHITs.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -input [Input file filename] A file containing the HIT ids to be updated. The file must contain a column labeled hitid. The Success File can be used as the input file for this operation yes -input helloworld.input.success
    -properties The HIT Properties file that contains the new properties to update the HITs with. The HITs will be updated with all the values defined in the properties file. yes -properties helloworld.properties
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Extend HITs Operation

    This operation can allows HITs to be active on Amazon Mechanical Turk for a longer period of time by extending the expiration date of existing HITs. It can also allow more Workers to work on a HIT by adding more assignments to existing HITs.

    When you add assignments to existing HITs you must have enough balance to pay for those additional assignments.

    To run the extendHITs operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: extendHITs [your options]
      For Unix: ./extendHITs.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -successfile [.success filename] A .success file containing the HIT ids to be updated. The file must contain a column labeled hitid. yes -successfile helloworld.success
    -hours How many hours to extend the expiration date of the HITs. No -hours 24
    -assignments [number of assignments] How many assignments to add to the HITs. No -assignments 3
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Grant Bonus Operation

    This operation grants a bonus to a Worker. You can pay a bonus to any Worker who has submitted an assignment for you. You must have enough balance in your account to pay for the bonus.

    To run the grantBonus operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: grantBonus [your options]
      For Unix: ./grantBonus.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -workerid [worker ID] The ID (i.e. A3C4G8DMXFG5PQ) of the worker you want to pay the worker. yes -workerid A3C4G8DMXFG5PQ
    -amount [amount to pay] The amount of bonus you want to give to the worker in dollars. yes -amount 0.25
    -assignment [assignment for this bonus] The assignment associated with this bonus. You can find the assignment ID in the .results file. yes -assignment ZYJZWSCAT0DZRFY5KYP00S0ZS8Y5H0NZR9YAMY1Z
    -reason The reason you're paying the bonus. The reason string should be enclosed by quotes "" yes -reason "Good job, you did the most work last week!"
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Block Worker Operation

    This operation blocks a Worker from working on your HITs. This operation should be used sparingly in order to prevent Workers who consistently submit low quality work from completing your HITs. This operation should not replace the Qualifications capabilities that are available for managing your Workers. Corrective action will be taken against Workers who are consistently blocked by Requesters.

    To run the blockWorker operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: blockWorker [your options]
      For Unix: ./blockWorker.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -workerid [worker ID] The ID (i.e. A3C4G8DMXFG5PQ) of the worker you want to block. yes -workerid A3C4G8DMXFG5PQ
    -reason The reason why the Worker is being blocked. This reason is logged in our system for auditing purposes and can be used to determine if corrective action against the Worker is necessary. The reason string should be enclosed by quotes "" yes -reason "After several warnings, the Worker continued to submit answers without reading the instructions carefully."
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Unblock Worker Operation

    This operation unblocks a Worker that has been blocked from working on your HITs.

    To run the unblockWorker operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: unblockWorker [your options]
      For Unix: ./unblockWorker.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -workerid [worker ID] The ID (i.e. A3C4G8DMXFG5PQ) of the worker you want to unblock. yes -workerid A3C4G8DMXFG5PQ
    -reason The reason why the Worker is being unblocked. This reason is logged in our system for auditing purposes. The reason string should be enclosed by quotes "" yes -reason "Made a mistake. Blocked the wrong Worker ID"
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Create Qualification Type Operation

    This operation creates a Qualification that can be used for your HITs on Amazon Mechanical Turk.

    To run the createQualificationType operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: createQualificationType [your options]
      For Unix: ./createQualificationType.sh [your options]
    This operation will output a file containing the QualificationTypeID of the Qualification that was just created called [Qualifications Properties File].success.

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -properties [Qualification Properties File filename] Specify the Qualification Properties File. yes -properties qualification.properties
    -question [Qualification Question File filename] Specify the Qualification Question File that contains the Qualification test. no -question qualification.question
    -answer [Qualification Answer File filename] Specify the Qualification Answer File that contains the AnswerKey XML for the Qualification test so that the Amazon Mechanical Turk system can automatically grade a Qualification request. no -answer qualification.answer
    -noretry Specify that Workers cannot retry requesting the Qualification. Workers can only request the Qualification once. no -noretry
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Update Qualification Type Operation

    This operation updates the properties of your Qualification.

    To run the updateQualificationType operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: updateQualificationType [your options]
      For Unix: ./updateQualificationType.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -qualtypeid [Qualification Type ID] Specify the Qualification Type ID of the Qualification you want to update. This ID can be found in a .success file generated after you created the Qualification with the createQualificationType operation. yes -qualtypeid RWFZTKZ55ZPZXN1C8TDZ
    -properties [Qualification Properties File filename] Specify the Qualification Properties File that contains the new properties for the Qualification. The Qualification will be updated with all the properties defined in the file. Note: Amazon Mechanical Turk currently does not allow you to modify the title of a Qualification. This will be fixed in a future release of Amazon Mechanical Turk. no -properties qualification.properties
    -question [Qualification Question File filename] Specify the Qualification Question File that contains the new Qualification test. no -question qualification.question
    -answer [Qualification Answer File filename] Specify the Qualification Answer File that contains the new AnswerKey XML for the Qualification test so that the Amazon Mechanical Turk system can automatically evaluate and score a Qualification request. no -answer qualification.answer
    -status [Active or Inactive] Specify that the Qualification is Active or Inactive. Inactive Qualifications are no longer available for Workers and cannot be used for new HITs. Valid values are "Active" or "Inactive". no -status Active
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Assign Qualification Operation

    This operation assigns your Qualification to a Worker without them requesting the Qualification.

    To run the assignQualification operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: assignQualification [your options]
      For Unix: ./assignQualification.sh [your options]
    There are 2 ways to call this operation:

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -input [File containing the Qualification ID] Specify the file that contains the Qualification ID you want to assign to workers. This ID can be found in a .success file generated after you created the Qualification with the createQualificationType operation. The file must have a column header called qualtypeid yes. Only if the qualtypeid parameter is not specified. -input qualification.properties.success
    -scorefile [File containing Worker IDs and Qualification scores] Specify the file that contains a tab delimited list of Worker IDs and Qualification scores to assign the Qualification to. The file must contain a column called workerid. The file can optionally contain an additional column called score containing the Qualification scores to assign to each Worker. If the score is not specified, a default value of 1 will be assigned. yes. Only if the workerid parameter is not specified. -scorefile workerstoassign.txt
    -qualtypeid [Qualification Type ID] Specify the Qualification Type ID of the Qualification you want to assign to the worker. This ID can be found in a .success file generated after you created the Qualification with the createQualificationType operation. yes. Only if the input parameter is not specified. -qualtypeid RWFZTKZ55ZPZXN1C8TDZ
    -workerid [Worker ID] Specify the Worker ID to assign the Qualification to. yes. Only if the scorefile parameter is not specified. -workerid A3C4G8DMXFG5PQ
    -score [Qualification score to assign] Specify the default Qualification score to assign to workers. This optional parameter can be used in both cases. If this is not specified, and the score has not been defined in the scorefile, the default score value of 1 will be used. no -score 100
    -donotnotify Specifies that the Worker will not be notified that they have been assigned the Qualification. no -donotnotify
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Get Qualification Requests Operation

    This operation retrieves the Qualification requests from Workers for your Qualifications.

    To run the getQualificationRequests operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: getQualificationRequests [your options]
      For Unix: ./getQualificationRequests.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -outputfile [Filename for output file] Specify the output filename that is a tab delimited text file containing the details on the Qualification requests made by Workers. Qualification Requests are to be approved or rejected by using the ApproveQualificationRequests and RejectQualificationRequests operations respectively. yes -outputfile qualification_requests.txt
    -qualtypeid [Qualification Type ID] Specify the Qualification Type ID of the Qualification you want to retrieve Qualification requests for. This ID can be found in a .success file generated after you created the Qualification with the createQualificationType operation. If this is not specified, all Qualification requests for Qualifications you own will be included in the output file. no -qualtypeid RWFZTKZ55ZPZXN1C8TDZ
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Approve Qualification Requests Operation

    This operation approves Worker Qualification requests.

    To run the approveQualificationRequests operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: approveQualificationRequests [your options]
      For Unix: ./approveQualificationRequests.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -approvefile [file containing Qualification requests to approve] Specify the tab delimited text file containing the list of Qualification requests to approve. The file must contain a column titled qualificationRequestToApprove containing the Qualification request IDs. An optional column called qualificationRequestToApproveValue can be specified containing the Qualification score to assign for each Qualification request. yes. Only if qualRequest parameter is not specified. -approvefile qualification_requests_toapprove.txt
    -score [integer value for default Qualification score] Specify the default score to assign for each approved Qualification request. Scores defined in the approve file will override this default score. no -score 100
    -qualRequest [Qualification request ID] Specify the Qualification request ID(s) to approve. Multiple Qualification request IDs should be comma-separated. yes. Only if approvefile parameter is not specified. -qualRequest "TA3ZJBYP2Y7ZJSX2BBN0TZ8ZTM4F6H4ZVQ4DE8FZ,TA3ZJBYP2Y7ZJSX2CCN0TZ8ZTM4F6H4ZVQ4DE8FZ"
    -force Specify NOT to prompt for manual confirmation before performing the reject operation. This parameter should only be used by advanced users. no -force
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Reject Qualification Requests Operation

    This operation rejects Worker Qualification requests.

    To run the rejectQualificationRequests operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: rejectQualificationRequests [your options]
      For Unix: ./rejectQualificationRequests.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -rejectfile [file containing Qualification requests to approve] Specify the tab delimited text file containing the list of Qualification requests to reject. The file must contain a column titled qualificationRequestToReject containing the Qualification request IDs. An optional column called qualificationRequestToRejectComment can be specified containing comments you want to provide to the Worker regarding the rejection. yes. Only if qualRequest parameter is not specified. -rejectfile qualification_requests_toreject.txt
    -qualRequest [Qualification request ID] Specify the Qualification request ID(s) to reject. Multiple Qualification request IDs should be comma-separated. yes. Only if rejectfile parameter is not specified. -qualRequest "TA3ZJBYP2Y7ZJSX2BBN0TZ8ZTM4F6H4ZVQ4DE8FZ,TA3ZJBYP2Y7ZJSX2CCN0TZ8ZTM4F6H4ZVQ4DE8FZ"
    -force Specify NOT to prompt for manual confirmation before performing the reject operation. This parameter should only be used by advanced users. no -force
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Evaluate Qualification Requests Operation

    This operation evaluates the answers submitted by Workers in order to approve/reject Qualification requests.

    This operation uses an answer key file for the Qualification test to automatically review the test results submitted by Workers and determine if the Qualification request should be approved or rejected. This method allows you to specify free text answers that can be automatically processed. This overcomes a current limitation of the Amazon Mechanical Turk system where FreeTextAnswer type answers cannot be auto graded.

    All questions must be answered correctly in order for the Qualification request to be approved. A Qualification score of 100 will be assigned for approvals.

    To run the evaluateQualificationRequests operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: evaluateQualificationRequests [your options]
      For Unix: ./evaluateQualificationRequests.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -answers [Qualification Command Line Tools Answer Key File] Specify the Qualification Command Line Tools Answer Key File (see Description of Files Used by the Command Line Tools for more details on file format) that contains the answer key. yes -answers qualification.answerkey
    -input [File containing the Qualification Type ID] Specify the file that contains the Qualification ID you want to assign to workers. This ID can be found in a .success file generated after you created the Qualification with the createQualificationType operation. The file must have a column header called qualtypeid. yes. Only if the qualtypeid parameter is not specified. -input qualification.properties.success
    -qualtypeid [Qualification Type ID] Specify the Qualification Type ID to evaluate Qualification requests for. yes. Only if the input parameter is not specified. -qualtypeid RWFZTKZ55ZPZXN1C8TDZ
    -preview Specify if you want to preview the outcome of the Qualification request evaluations before submitting the decisions to Amazon Mechanical Turk. no -preview
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Update Qualification Scores Operation

    This operation updates the Qualification scores for Workers.

    To run the updateQualificationScore operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: updateQualificationScore [your options]
      For Unix: ./updateQualificationScore.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -input [File containing Worker IDs and scores] Specify the tab delimited file that contains the Worker IDs and desired Qualification scores to assign. The file must have a column header called workerid containing the Worker IDs. An optional column called score can be specified to assign specific scores to Workers. yes. Only if the workerid parameter is not specified. -input worker_qual_scores.txt
    -qualtypeid [Qualification Type ID] Specify the Qualification Type ID to evaluate Qualification requests for. yes -qualtypeid RWFZTKZ55ZPZXN1C8TDZ
    -score [Integer value for score] Specify the score to assign to each Worker for the given Qualification Type ID. yes. Only if the score has NOT been defined in the input file. -score 50
    -workerid [Worker ID] Specify the Worker ID of the Worker that you want to assign the score to. yes. Only if the input parameter is not specified. -workerid A3C4G8DMXFG5PQ
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Revoke Qualification Operation

    This operation revokes a Qualification from a Worker.

    To run the revokeQualification operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: revokeQualification [your options]
      For Unix: ./revokeQualification.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -qualtypeid [Qualification Type ID] Specify the Qualification Type ID of the Qualification to revoke the Worker from. yes -qualtypeid RWFZTKZ55ZPZXN1C8TDZ
    -workerid [Worker ID] Specify the Worker ID of the Worker that you want to revoke the Qualification from. yes -workerid A3C4G8DMXFG5PQ
    -reason [reason for revoking Qualification] Specify the reason for revoking the Qualification from the Worker. The reason should be enclosed by "". no -reason "I'm sorry but your work quality is below acceptable standards."
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Reset Account Operation

    This operation deletes all of your existing HITs from Amazon Mechanical Turk. If any of the HITs are still being worked on by workers then those HITs are not deleted. If any of the assignments for the HITs are submitted but not yet approved, they will be automatically approved before the HITs are deleted.

    To run the resetAccount operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\bin
    2. Run the command:
      For Microsoft Windows: resetAccount [your options]
      For Unix: ./resetAccount.sh [your options]

    Options

    Option Definition Required Example
    -sandbox Configures the command line tools to run against the Developer Sandbox environment. Overrides the setting in mturk.properties file. no -sandbox
    -force If the -force option is specified then the user is not prompted for confirmation when running the resetAccount command. no -force
    -help or -h Print Help for this operation. no -help or -h

    Return to Table of Contents

    Running the Make Template Operation

    This operation creates a copy of a sample applciation. After you run the sample applications that the Amazon Mechanical Turk Command Line Tools provide, you might want to create your own application, probably one that is similar to one of the sample applications. The Make Template operation makes it easy for you to replicate one of the samples in a different directory. You can make a template of a HIT-generating sample, such as the Hello World sample or the Best Image sample. For these templates, run the Make Template operation from the [Command Line Tools installation directory]\hits directory.

    You can also make copies of a Qualification sample by running the Make Template operation from the [Command Line Tools installation directory]\qualifications directory.

    To run the Make Template operation:

    1. Navigate to the following directory:
      [Command Line Tools installation directory]\hits or to the following directory: [Command Line Tools installation directory]\qualifications
    2. Run the command:
      For Microsoft Windows: makeTemplate [your options]
      For Unix: ./makeTemplate.sh [your options]

    Options for MakeTemplate for HITs

    Option Definition Required Example
    -target [directory name] Specifies the name of the directory to create. This is the directory where the new files will be copied. This name is also used for the files that are created in this directory. Note: this directory has to be under the hits or qualifications directory. yes -target my_survey
    -template [directory name] The name of the sample directory from which you are making the template. yes -template helloworld
    -help or -h Print Help for this operation. no -help or -h

    When you run this operation for HIT-generating samples, the Command Line Tools creates the new directory and copies the .input, .question, and .properties files from the specified sample. These files are created with the name specified in the -target option. If you are using Windows, the Command Line Tools creates new run.cmd, getResults.cmd, and approveAndDeleteResults.cmd files. If you are using Unix, the new files are run.sh, getResults.sh, and approveAndDeleteResults.sh.

    Options for MakeTemplate for Qualifications

    When you run the makeTemplate command for qualifications, you can simply pass in the name of the target directory, e.g. makeTemplate myQual. The makeTemplate command for qualifications only copies over one sample qualification unlike the makeTemplate for HITs operation.

    When you run this operation for Qualification-generating samples, the Command Line Tools creates the new directory and copies the .answer, .question, and .properties files from the specified sample. These files are created with the name of the target directory you specified. If you are using Windows, the Command Line Tools creates new createQualification.cmd, updateQualification.cmd, and deactivateQualification.cmd files. If you are using Unix, the new files are createQualification.sh, updateQualification.sh, and deactivateQualification.sh.

    Return to Table of Contents

    Logging and Debugging

    By default, logging has been enabled for all operations performed by the Command Line Tools. The logs will contain significant details on each web request made to Amazon Mechanical Turk as well as details on the responses received back from the system.

    These logs can be helpful in troubleshooting any issues you encounter with the Command Line Tools and contains sufficient details to allow us to diagnose them.

    You can access the logs here: [Command Line Tools Installation Directory]\log\java-aws-mturk.log Since this file currently logs all requests, it will grow in size very quickly. You can safely delete this file and not impact the functionality of the Command Line Tools.

    The logs are configured to only take up a maximum of 10 MB. When a log file exceeds 2 MB, another log file is created (i.e. java-aws-mturk.2.log) up to 5 log files. Once the maximum of 10 MB is reached, the first log file will be overwritten and will continue as a rolling process.

    For advanced users, the level of logging can be specified. To set the level of logging:

    1. Open your mturk.properties in a text editor.
    2. Add the following line as a property in the file:
      log_level=[your desired log level]
    3. Save and close the file.
    The Command Line Tools uses     Log4J for logging. The following log level definitions are available:
    For more information on what details are logged for each log level, access the Log4J documentation on log levels.

    Return to Table of Contents

    Portability of Convenience Files

    In general, the convenience files (i.e. mturk.properties, input files, question template, HIT template, etc.) used by the Amazon Mechanical Turk Command Line tools are portable to the Amazon Mechanical Turk SDK for Java.

    However, the convenience files are not currently portable to other Amazon Mechanical Turk SDKs in other programming languages (i.e. Ruby, C#, PERL, etc.).

    Portability across SDKs on different programming platforms is a desired goal for future releases.

    Return to Table of Contents

    Comments, Questions or Feedback

    If you have any comments, questions or feedback on the Amazon Mechanical Turk SDK for Java or the service in general, please visit the Amazon Mechanical Turk discussion forums. Alternatively, you can visit the open source project home page.

    Return to Table of Contents