The work
You will assess coding agents that handle real software engineering tasks. Using codebases and realistic tickets, you will judge whether their implementations work and meet reasonable engineering standards.
The role centers on coding agent evaluation. Review the duties and qualifications carefully before applying.
- Assign coding agents real engineering tickets and review their output like a teammate’s pull request.
- Run submitted code and decide whether it solves the task correctly.
- Work step by step with an AI agent on problems from familiar codebases.
- Find, explain, and correct errors in the agent’s implementation or reasoning.
- Use software engineering judgment to assess difficult tasks and explain where the agent falls short.
What it pays and takes
This contract supports remote work from the United States. The schedule is flexible, but the role requires a substantial weekly commitment.
- Pay: Up to $150 per hour, with weekly payment.
- Hours: 20 or more hours per week.
- Location: Open to candidates in the United States.
- Language: Strong written English and clear explanations of technical decisions.
- Experience: At least four years of professional software engineering experience.
- Education: Bachelor’s or master’s degree in computer science or a related field.
- Helpful background: Prior AI experience is not required. You should be able to run code, review implementations, diagnose defects, and decide whether a solution meets a real ticket’s requirements.
- Work arrangement: Remote contract work with flexible scheduling.
How it works
Apply on OpenTrain with your resume, then complete the application on the hiring site.