Surge API
Welcome to the Surge API! You can use the API to programmatically create projects and run tasks. All you need is your API key, and funds in your account if you want to have tasks completed by our workforce.
Authentication
To authorize, use this code.
require 'httparty'
# For POST requests
HTTParty.post("{{API_ENDPOINT}}",
basic_auth: { username: "{{YOUR_API_KEY}}" },
body: {
... # Parameters in the POST request
}
)
# For GET requests
HTTParty.get("{{API_ENDPOINT}}",
basic_auth: { username: "{{YOUR_API_KEY}}" }
)
import requests
# For POST requests
requests.post("{{API_ENDPOINT}}",
auth = ("{{YOUR_API_KEY}}", ""),
json = {
... # Parameters in the POST request
}
)
# For GET requests
requests.get("{{API_ENDPOINT}}",
auth = ("{{YOUR_API_KEY}}", "")
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
# Or set the API key as an environment variable:
# export SURGE_API_KEY={{YOUR_API_KEY}}
# Note that the colon follows the API key and is not part of it.
curl {{API_ENDPOINT}} \
-u {{YOUR_API_KEY}}:
Surge uses API keys to allow access to the API. You can find your API key in your profile.
Authentication is performed via HTTP Basic Auth. Your API key is your basic auth username, and you do not need to provide a password.
If you need to authenticate via bearer auth (e.g., for a cross-origin request), use -H "Authorization: {{YOUR_API_KEY}}:"
instead of -u {{YOUR_API_KEY}}:
.
Example
Let's walk through an example using Surge to categorize companies. If you'd like to run the example yourself, find your API key in your profile.
require 'httparty'
# First, we create a "categorize company" project.
response = HTTParty.post("https://app.surgehq.ai/api/projects",
basic_auth: { username: "DJ34KFX24M2G3JRHGXNZVCPE" },
body: {
name: "Categorize this company",
payment_per_response: 0.1,
questions: [
{
"text": "What is this company's website?"
},
{
"text": "What category does this company belong to?",
"options": ["Tech", "Sports", "Gaming"]
}
]
}
)
# Extract the project ID.
project_id = JSON.parse(response.body)["id"]
# Post a task to the API.
HTTParty.post("https://app.surgehq.ai/api/projects/#{project_id}/tasks",
basic_auth: { username: "DJ34KFX24M2G3JRHGXNZVCPE" },
body: {
"fields": {
"website": "google.com"
}
}
)
# Retrieve the tasks and any responses.
response = HTTParty.get("https://app.surgehq.ai/api/projects/#{project_id}/tasks",
basic_auth: { username: "DJ34KFX24M2G3JRHGXNZVCPE" }
)
JSON.parse(response.body).each do |task|
puts task["responses"][0]["response_data"]
end
import requests
requests.post("https://app.surgehq.ai/api/projects",
auth = ("{{YOUR_API_KEY}}", ""),
json = {
"name": "Categorize this company",
"payment_per_response": 0.1,
"questions": [
{
"text": "What is this company's website?"
},
{
"text": "What category does this company belong to?",
"options": ["Tech", "Sports", "Gaming"]
}
]
}
)
import surge
from surge.questions import FreeResponseQuestion, MultipleChoiceQuestion
surge.api_key = "{{YOUR_API_KEY}}"
# Create questions to add to the project
free_response_q = FreeResponseQuestion(
text="What is this company's website?")
multiple_choice_q = MultipleChoiceQuestion(
text="What category does this company belong to?",
options=["Tech", "Sports", "Gaming"])
project = surge.Project.create(
name="Categorize this company",
payment_per_response=0.1,
questions=[free_response_q, multiple_choice_q])
Projects
Create a project
Definition
POST https://app.surgehq.ai/api/projects
Example Request
curl https://app.surgehq.ai/api/projects \
-u {{YOUR_API_KEY}}: \
-d name="Categorize this company" \
-d payment_per_response=0.1 \
-d questions=... (can't do this in curl)
require 'httparty'
HTTParty.post("https://app.surgehq.ai/api/projects",
basic_auth: { username: "{{YOUR_API_KEY}}" },
body: {
name: "Categorize this company",
payment_per_response: 0.1,
instructions: "You will be asked to categorize a company.",
questions: [
{
"type": "free_response",
"text": "What is this company's website?"
},
{
"type": "multiple_choice",
"text": "What category does this company belong to?",
"options": ["Tech", "Sports", "Gaming"]
},
{
"type": "checkbox",
"text": "Check all the social media accounts this company has",
"options": ["Facebook", "Twitter", "Pinterest", "Google+"]
}
]
}
)
import requests
requests.post("https://app.surgehq.ai/api/projects",
auth = ("{{YOUR_API_KEY}}", ""),
json = {
"name": "Categorize this company",
"payment_per_response": 0.1,
"instructions": "You will be asked to categorize a company.",
"questions": [
{
"type": "free_response",
"text": "What is this company's website?"
},
{
"type": "multiple_choice",
"text": "What category does this company belong to?",
"options": ["Tech", "Sports", "Gaming"]
},
{
"type": "checkbox",
"text": "Check all the social media accounts this company has",
"options": ["Facebook", "Twitter", "Pinterest", "Google+"]
}
]
}
)
import surge
from surge.questions import (FreeResponseQuestion, MultipleChoiceQuestion,
CheckboxQuestion)
surge.api_key = "{{YOUR_API_KEY}}"
# Create questions to add to the project
free_response_q = FreeResponseQuestion(
text="What is this company's website?")
multiple_choice_q = MultipleChoiceQuestion(
text="What category does this company belong to?",
options=["Tech", "Sports", "Gaming"])
checkbox_q = CheckboxQuestion(
text="Check all the social media accounts this company has",
options=["Facebook", "Twitter", "Pinterest", "Google+"])
project = surge.Project.create(
name="Categorize this company",
payment_per_response=0.1,
instructions="You will be asked to categorize a company.",
questions=[free_response_q, multiple_choice_q, checkbox_q])
Example Response
Project({
"id": "17e323f1-f7e4-427c-a2d5-456743aba8",
"name": "Categorize this company",
"num_tasks": 0,
"num_tasks_completed": 0,
"num_workers_per_task": 1,
"payment_per_response": 0.1,
"status": "unlaunched"
})
Creates a project. It is also possible to create a project through the web UI (instead of the API), and you will still be able to use the API to post tasks to the project.
Parameters
Parameter | Type | Description |
---|---|---|
name | string | Required. Name to give your project. |
private_workforce | boolean | Optional (default false). If true, the project won't be shown to the Surge workforce and will only be available to workers who receive the link to your task. If false, |
payment_per_response | float | Optional, only required for tasks being sent to the Surge workforce. Amount in dollars to pay workers for each response. You will need enough funds in your account to pay for all of the responses required by the project before launching it. |
instructions | string | Optional. Instructions shown to workers describing how they should complete the task. |
fields_template | string | Optional. A template describing how fields are shown to workers working on the task. For example, if fields_template is <a href="{{url}}">{{company_name}}</a> , then workers will be shown a link to the company. |
questions | array | Required. An array of question objects describing the questions to be answered. Each question object contains a text field (the text of the question), an optional options array (for multiple choice and checkbox questions, listing the answer options), a type field (one of free_response , multiple_choice , or checkbox ), and a required boolean field (describing whether or not workers must answer the question). See the Question Types section for a list of available question types. |
num_workers_per_task | integer | Optional (default 1). Number of workers who work on each task. |
Question Types
This table describes each of the available questions you can send to workers
Question Type | Description | Screenshot of worker view |
---|---|---|
Multiple Choice | The worker is presented with a multiple choice question and can only select one answer. SDK Object: MultipleChoiceQuestion(text, options, required=True) |
![]() |
Checkbox | The worker is presented with a series of options and can check as many as they want. SDK Object: CheckboxQuestion(text, options) |
![]() |
Free Response | The worker is presented with a text area input they can type into. SDK Object: FreeResponseQuestion(text, required=True) |
![]() |
Text Tagging (Named Entity Recognition) | The worker is presented text and a list of options. They can highlight spans of text to tag them. SDK Object: TextTaggingQuestion(text, options) |
![]() |
Retrieve a project
Definition
GET https://app.surgehq.ai/api/projects/{{PROJECT_ID}}
Example Request
require 'httparty'
HTTParty.get("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8",
basic_auth: { username: "{{YOUR_API_KEY}}" }
)
import requests
requests.get("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8",
auth = ("{{YOUR_API_KEY}}", "")
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
project = surge.Project.retrieve("17e323f1-f7e4-427c-a2d5-456743aba8")
curl https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8 \
-u {{YOUR_API_KEY}}:
Example Response
Project({
"id": "17e323f1-f7e4-427c-a2d5-456743aba8",
"name": "Categorize this company",
"num_tasks": 1000,
"num_tasks_completed": 239,
"num_workers_per_task": 1,
"payment_per_response": 0.1,
"status": "unlaunched"
})
Retrieves a specific project you have created.
Parameter | Type | Description |
---|---|---|
id | string | Unique identifier for project. |
created_at | DateTime | When the project was created. |
fields_template | string | A template describing how fields are shown to workers working on the task. For example, if fields_template is <a href="{{url}}">{{company_name}}</a> , then workers will be shown a link to the company. |
instructions | string | Instructions shown to workers describing how they should complete the task. |
name | string | Name of the project. |
link_to_work_on_task | string | URL for working on the task. If you are using the private workforce option, this is the URL you will want to send to your workforce. |
num_tasks | integer | Number of tasks in the project. |
num_tasks_completed | integer | Number of completed tasks. |
num_workers_per_task | integer | How many workers work on each task (i.e., how many responses per task). |
payment_per_response | float | How much a worker is paid (in US dollars) for an individual response. |
questions | array | An array of question objects describing the questions to be answered. Each question object contains a text field (the text of the question), an optional options array (for multiple choice and checkbox questions, listing the answer options), a type field (one of free_response , multiple_choice , or checkbox ), and a required boolean field (describing whether or not workers must answer the question). See the Question Types section for a list of available question types. |
status | string | One of in_progress , completed , canceled , or paused . |
interrater_agreement | Dictionary | A dictionary mapping each question to the its interrater agreement score. |
avg_gold_standard_score | Float | The average correctness of the project's gold standard responses, ranging from 0-100. For gold standards set multiple choice questions, this is the percent of workers who answered correctly. For checkbox questions, this is the proportion of checkboxes where the worker correctly checked it or left it unchecked. For text tagging questions this is the average of a similarity score between the gold standard tags and worker tags. |
List all projects
Definition
GET https://app.surgehq.ai/api/projects
GET https://app.surgehq.ai/api/projects?page=n
Example Request
require 'httparty'
HTTParty.get("https://app.surgehq.ai/api/projects?page=2",
basic_auth: { username: "{{YOUR_API_KEY}}" }
)
import requests
requests.get("https://app.surgehq.ai/api/projects",
auth = ("{{YOUR_API_KEY}}", "")
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
projects = surge.Project.list(page=1)
curl https://app.surgehq.ai/api/projects \
-u {{YOUR_API_KEY}}:
Example Response
[
Project({
"id": "17e323f1-f7e4-427c-a2d5-456743aba8",
"name": "Categorize this company",
"num_tasks": 1000,
"num_tasks_completed": 239,
"num_workers_per_task": 1,
"payment_per_response": 0.1,
"status": "unlaunched"
}),
Project({
"id": "81533541-0359-4d4b-a545-af38a2cb3e8c",
"name": "Label product images",
"num_tasks": 5000,
"num_tasks_completed": 4315,
"num_workers_per_task": 1,
"payment_per_response": 0.25,
"status": "in_progress"
})
]
Get a list of all the projects you have created. Returns a list of Project objects.
Parameters
Parameter | Type | Description |
---|---|---|
page | integer | Projects are returned in descending created_at order. Each page contains a maximum of 25 projects. Pages start at 1. (optional, default 1) |
Pause a project
Definition
POST https://app.surgehq.ai/api/projects/{{PROJECT_ID}}/pause
Example Request
require 'httparty'
HTTParty.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/pause",
basic_auth: { username: "{{YOUR_API_KEY}}" }
)
import requests
requests.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/pause",
auth = ("{{YOUR_API_KEY}}", "")
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
project = surge.Project.retrieve("17e323f1-f7e4-427c-a2d5-456743aba8")
project = project.pause()
curl https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/pause \
-u {{YOUR_API_KEY}}: \
-X POST
Example Response
Project({
"id": "17e323f1-f7e4-427c-a2d5-456743aba8",
"name": "Categorize this company",
"num_tasks": 1000,
"num_tasks_completed": 239,
"num_workers_per_task": 1,
"payment_per_response": 0.1,
"statu)s": "paused"
}
Pauses a project. Tasks added to the project will not be worked on until you resume the project.
Resume a project
Definition
POST https://app.surgehq.ai/api/projects/{{PROJECT_ID}}/resume
Example Request
require 'httparty'
HTTParty.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/resume",
basic_auth: { username: "{{YOUR_API_KEY}}" }
)
import requests
requests.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/resume",
auth = ("{{YOUR_API_KEY}}", "")
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
project = surge.Project.retrieve("17e323f1-f7e4-427c-a2d5-456743aba8")
project = project.resume()
curl https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/resume \
-u {{YOUR_API_KEY}}: \
-X POST
Example Response
Project({
"id": "17e323f1-f7e4-427c-a2d5-456743aba8",
"name": "Categorize this company",
"num_tasks": 1000,
"num_tasks_completed": 239,
"num_workers_per_task": 1,
"payment_per_response": 0.1,
"status": "in_progress"
})
Resumes a paused project.
Cancel a project
Definition
POST https://app.surgehq.ai/api/projects/{{PROJECT_ID}}/cancel
Example Request
require 'httparty'
HTTParty.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/cancel",
basic_auth: { username: "{{YOUR_API_KEY}}" }
)
import requests
requests.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/cancel",
auth = ("{{YOUR_API_KEY}}", "")
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
project = surge.Project.retrieve("17e323f1-f7e4-427c-a2d5-456743aba8")
project = project.cancel()
curl https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/cancel \
-u {{YOUR_API_KEY}}: \
-X POST
Example Response
Project({
"id": "17e323f1-f7e4-427c-a2d5-456743aba8",
"name": "Categorize this company",
"num_tasks": 1000,
"num_tasks_completed": 239,
"num_workers_per_task": 1,
"payment_per_response": 0.1,
"status": "canceled"
})
Cancels a project.
Tasks
The task object
Example Response
Task({
"id": "38da6bc5-a644-41b9-a964-4678bc7375c6",
"project_id": "b31ede78-afdf-4938-b775-8813453c7fc5",
"created_at": "2016-11-01T18:56:56.000Z",
"is_complete": true,
"fields": {
"company": "Surge",
"city": "San Francisco",
"state": "CA"
},
"responses": [
TaskResponse({
"id": "1befb37b-8e4f-42b6-8a61-2c946ad4b5ce",
"data": {
"website": "https://app.surgehq.ai",
"category": "Technology"
},
"time_spent_in_secs": 2,
"completed_at": "2016-11-01T23:29:17.971Z",
"worker_id": "P94T3ATCZX4X"
})
]
})
Parameter | Type | Description |
---|---|---|
id | string | Unique identifier for the task. |
created_at | DateTime | When the task was created. |
project_id | string | ID of the project that this task belongs to. |
is_complete | boolean | Whether or not this task has received the desired number of responses (equal to the project's num_workers_per_task field). |
fields | dictionary | A dictionary of named fields that get shown (according to the project template) to workers when working on the task. |
responses | array | An array of TaskResponses to the task. Each TaskResponse object contains an id , time_spent_in_secs field, completed_at field, worker_id field, and a data dictionary that maps questions to their answers. |
Create a task
Definition
POST https://app.surgehq.ai/api/projects/{{PROJECT_ID}}/tasks
Example Request
require 'httparty'
HTTParty.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/tasks",
basic_auth: { username: "{{YOUR_API_KEY}}" },
body: {
"fields": {
"company": "Surge",
"city": "San Francisco",
"state": "CA"
}
}
)
import requests
requests.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/tasks",
auth = ("{{YOUR_API_KEY}}", ""),
json = {
"fields": {
"company": "Surge",
"city": "San Francisco",
"state": "CA"
}
}
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
project = surge.Project.retrieve("17e323f1-f7e4-427c-a2d5-456743aba8")
tasks = project.create_tasks([{
"company": "Surge",
"city": "San Francisco",
"state": "CA"
}])
# You can also create Tasks in bulk via CSV file.
# The header of the CSV file must specify the fields that are used in your Tasks.
file_path = './companies_to_classify.csv'
tasks = project.create_tasks_from_csv(file_path)
curl https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/tasks \
-u {{YOUR_API_KEY}}: \
-d data=... (can't do this in curl)
Example Response
Task({
"id": "38da6bc5-a644-41b9-a964-4678bc7375c6",
"project_id": "b31ede78-afdf-4938-b775-8813453c7fc5",
"created_at": "2016-11-01T18:56:56.000Z",
"is_complete": false,
"fields": {
"company": "Surge",
"city": "San Francisco",
"state": "CA"
},
"responses": []
})
Creates a task.
Whenever a task is created, the project is automatically launched if it is not already in progress.
Parameters
Parameter | Type | Description |
---|---|---|
data | dictionary | A dictionary of named fields that get shown (according to the project template) to workers when working on the task. (required) |
List all tasks
Definition
GET https://app.surgehq.ai/api/projects/{{PROJECT_ID}}/tasks
GET https://app.surgehq.ai/api/projects/{{PROJECT_ID}}/tasks?page=n
Example Request
require 'httparty'
HTTParty.get("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/tasks",
basic_auth: { username: "{{YOUR_API_KEY}}" }
)
import requests
requests.post("https://app.surgehq.ai/api/projects/17e323f1-f7e4-427c-a2d5-456743aba8/tasks",
auth = ("{{YOUR_API_KEY}}", "")
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
project = surge.Project.retrieve("17e323f1-f7e4-427c-a2d5-456743aba8")
tasks = project.list_tasks(page=1)
curl https://app.surgehq.ai/api/projects/b31ede78-afdf-4938-b775-8813453c7fc5/tasks \
-u {{YOUR_API_KEY}}: \
...
Example Response
[
Task({
"id": "38da6bc5-a644-41b9-a964-4678bc7375c6",
"project_id": "b31ede78-afdf-4938-b775-8813453c7fc5",
"created_at": "2016-11-01T18:56:56.000Z",
"is_complete": true,
"fields": {
"company": "Surge",
"city": "San Francisco",
"state": "CA"
},
"responses": [
TaskResponse({
"id": "1befb37b-8e4f-42b6-8a61-2c946ad4b5ce",
"data": {
"website": "https://app.surgehq.ai",
"category": "Technology"
},
"time_spent_in_secs": 2,
"completed_at": "2016-11-01T23:29:17.971Z",
"worker_id": "P94T3ATCZX4X"
})
]
}),
Task({
"id": "dd66e80a-88b3-4c4f-9efc-2aca8eed73db",
"project_id": "b31ede78-afdf-4938-b775-8813453c7fc5",
"created_at": "2016-11-01T23:23:26.016Z",
"is_complete": true,
"fields": {
"company": "Google",
"city": "Mountain View",
"state": "CA"
},
"responses": [
TaskResponse({
"id": "cc9f4263-1d0f-4730-a97e-199851b6ade5",
"data": {
"website": "https://www.google.com",
"category": "Technology"
},
"time_spent_in_secs": 3,
"completed_at": "2016-11-01T23:22:19.124Z",
"worker_id": "P94T3ATCZX4X"
})
]
})
]
Lists all tasks belonging to a project.
Query Parameters
Parameters
Parameter | Type | Description |
---|---|---|
page | integer | Tasks are returned in ascending created_at order. Each page contains a maximum of 25 tasks. Pages start at 1. (optional, default 1) |
Retrieve a task
Definition
GET https://app.surgehq.ai/api/tasks/{{TASK_ID}}
Example Request
require 'httparty'
HTTParty.get("https://app.surgehq.ai/api/tasks/38da6bc5-a644-41b9-a964-4678bc7375c6",
basic_auth: { username: "{{YOUR_API_KEY}}" }
)
import requests
requests.post("https://app.surgehq.ai/api/tasks/38da6bc5-a644-41b9-a964-4678bc7375c6",
auth = ("{{YOUR_API_KEY}}", "")
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
task = surge.Task.retrieve(task_id = "eaa44510-c8f6-4480-b746-28a6c8defd4c")
curl https://app.surgehq.ai/api/tasks/38da6bc5-a644-41b9-a964-4678bc7375c6 \
-u {{YOUR_API_KEY}}:
Example Response
Task({
"id": "38da6bc5-a644-41b9-a964-4678bc7375c6",
"project_id": "b31ede78-afdf-4938-b775-8813453c7fc5",
"created_at": "2016-11-01T18:56:56.000Z",
"is_complete": true,
"fields": {
"company": "Surge",
"city": "San Francisco",
"state": "CA"
},
"responses": [
TaskResponse({
"id": "1befb37b-8e4f-42b6-8a61-2c946ad4b5ce",
"data": {
"website": "https://app.surgehq.ai",
"category": "Technology"
},
"time_spent_in_secs": 2,
"completed_at": "2016-11-01T23:29:17.971Z",
"worker_id": "P94T3ATCZX4X"
})
]
})
Retrieves a specific task you have created.
Set a gold standard answer
Definition
GET https://app.surgehq.ai/api/tasks/{{TASK_ID}}/gold-standards
Example Request
require 'httparty'
HTTParty.get("https://app.surgehq.ai/api/tasks/38da6bc5-a644-41b9-a964-4678bc7375c6/gold-standards",
basic_auth: { username: "{{YOUR_API_KEY}}" },
body: {
"answers": [
"https://surgehq.ai",
'Tech',
['Twitter']
]
}
)
import requests
requests.post("https://app.surgehq.ai/api/tasks/38da6bc5-a644-41b9-a964-4678bc7375c6/gold-standards",
auth = ("{{YOUR_API_KEY}}", ""),
json = {
"answers": [
"https://surgehq.ai",
'Tech',
['Twitter']
]
}
)
import surge
surge.api_key = "{{YOUR_API_KEY}}"
task = surge.Task.retrieve(task_id = "eaa44510-c8f6-4480-b746-28a6c8defd4c")
task.set_gold_standard(['https://surgehq.ai', 'Tech', ['Twitter']])
curl https://app.surgehq.ai/api/tasks/38da6bc5-a644-41b9-a964-4678bc7375c6/gold-standards \
-u {{YOUR_API_KEY}}
-d data=... (can't do this in curl)
Example Response
Task({
"id": "38da6bc5-a644-41b9-a964-4678bc7375c6",
"project_id": "b31ede78-afdf-4938-b775-8813453c7fc5",
"created_at": "2016-11-01T18:56:56.000Z",
"is_complete": false,
"fields": {
"company": "Surge",
"city": "San Francisco",
"state": "CA"
},
"gold_standard_data": "{'ner': {},'multiple_choice': {'b6337c97-6498-4c8c-a9ea-fcdc45782067': '3366ef4e-fe7f-4606-92bf-fba9a60a3d19'},'checkbox': {'f3feb6b0-daf9-42e3-9154-fc66fe10802d': ['44f050e2-e936-4241-8a1b-991bdb64a53e']},'free_response': {'85470cbd-8309-42ab-be70-500eb644736d': 'https://surgehq.ai'}}"
})
Gold standards are used to assess worker quality. After a gold standard answer is uploaded, every single worker will be given that piece of data so they can all be evaluated based on their performance. The average correctness across all gold standards in a project are in the gold_standard_score
field, and you can see additional analytics on the site, including how each worker is performing on gold standards and aggregated stats indicating which gold standards are answered incorrectly most often.
Parameters
Parameter | Type | Description |
---|---|---|
answers | List[String] or None | A list of the ground truth answers for this task, one for each question in the project. If you don't want to set an answer for one of the questions, you can leave it blank by passing an empty string. For example, if you wanted to modify the example on the right to only set the company categorization multiple choice question as a gold standard then you could set answers to ['', 'Tech', ''] . |
answers | boolean | This param indicates whether this task is a gold standard. You can toggle gold standards on or off without modifying the answers by setting this value to true or false and omitting the answers array. |
Task Responses
The TaskResponse object
Example Response
TaskResponse({
"id": "1befb37b-8e4f-42b6-8a61-2c946ad4b5ce",
"data": {
"website": "https://app.surgehq.ai",
"category": "Technology"
},
"time_spent_in_secs": 2,
"completed_at": "2016-11-01T23:29:17.971Z",
"worker_id": "P94T3ATCZX4X"
})
Parameter | Type | Description |
---|---|---|
id | string | A unique identifier for the task response. |
data | dictionary | A dictionary that maps each question to the worker's response. |
time_spent_in_secs | int | Time (in seconds) that the worker spent to complete this task response. |
completed_at | string | Timestamp of the response completion. |
worker_id | string | ID of the worker that completed this task response. |