The work
Create and assess coding datasets used to train and benchmark large language models. Use production engineering judgment to evaluate correctness, efficiency, scalability, reliability, and software quality.
- Curate code examples and develop precise solutions for training datasets.
- Correct and refine code in Python, C/C++, Rust, Go, Java, and JavaScript, including ReactJS.
- Assess AI-generated implementations for systems-level correctness, performance, scalability, and reliability.
- Build verification agents to identify error patterns and assess infrastructure code quality.
- Evaluate model capabilities across prototyping, architecture, API design, production implementation, monitoring, and operational maintenance.
- Design ways to automatically verify solutions to software engineering tasks and work with technical teams to improve coding results against performance benchmarks.
What it takes
- Strong expertise in systems programming, infrastructure, or backend development.
- Experience building and deploying scalable, production-grade software.
- Deep knowledge of software architecture, development, debugging, and code-quality review.
- Proficiency in one or more of Python, C/C++, Rust, Go, Java, or JavaScript.
- Ability to provide clear, structured written and spoken evaluation rationales.
Schedule and engagement
- Location: Open to candidates based in the United States.
- Schedule: Flexible, with a commitment of 20 or more hours per week.
- Engagement: Individual contractor, part-time.
- Initial term: One month, with potential extensions based on performance and fit.