The Work
You will create realistic, reusable evaluation scenarios for AI agents that perform real-world tasks. You will define the correct path, acceptable agent behavior, expected outputs, edge cases, and scoring rules.
You will also review agent outputs, improve scenarios for clarity and coverage, and work with developers and other contributors to test and refine evaluation frameworks.
- Design structured evaluation scenarios for LLM-based agents.
- Define gold-standard behavior and acceptable alternatives.
- Annotate task steps, expected outputs, edge cases, and scoring logic.
- Review agent responses and revise scenarios to improve coverage.
- Use structured formats such as JSON and YAML.
What It Pays and Takes
This is an intermediate-level, part-time contractor role for someone who combines careful writing with software testing or quality assurance thinking.
- Pay: $18 to $24 per hour.
- Time: 20 or more hours per week.
- Location: Open worldwide.
- Language: Strong written English.
- Required: Basic Python and JavaScript experience.
- Required: Analytical thinking and comfort designing structured evaluations.
- Preferred: Experience in software testing, QA, data analysis, or NLP annotation.
- Work type: Part-time contract.
How It Works
Apply on OpenTrain. The employer reviews applications there.
About AI Training Work
AI training work is the human work behind modern artificial intelligence, including writing tests, reviewing model responses, and rating whether systems behave correctly. People with technical and analytical skills are paid to provide the careful examples and judgments that help improve AI systems.