The work
You will evaluate AI-generated code and coding-agent behavior using real-world software repositories. Your reviews will help AI systems improve through structured human judgments and clear technical feedback.
- Review model outputs and source materials against project guidelines.
- Apply evaluation rubrics and make evidence-based judgments.
- Explain why an implementation is correct or incorrect.
- Document technical decisions in clear written feedback.
- Help maintain consistent quality across repeated review tasks.
- Support evaluation processes for AI coding agents.
What it pays and takes
This is a part-time contractor role. The work requires strong real-world software engineering knowledge, but prior AI training experience is not required.
- Pay: $100 to $150 per hour.
- Time: 20 or more hours per week.
- Location: Open to candidates in the United States and Canada.
- Language: Professional fluency in English.
- Experience level: Entry level.
- Experience: At least 5 years of hands-on software engineering experience.
- Programming: Strong proficiency in Python, TypeScript or JavaScript, Go, or another major production language.
- Codebases: Experience working in substantial real-world codebases.
- Core skills: Strong code review, technical judgment, and written communication.
- Technical knowledge: Familiarity with modern large language models or AI coding tools.
- Helpful experience: LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training.
How it works
Apply on OpenTrain with your resume, then complete the application on the hiring site.