Senior Software Engineer Ruby LLM Evaluation Repo Validation

You will create and evaluate software engineering tasks from public code repositories for AI training datasets. The work includes analyzing open-source code, triaging issues, and testing how well large language models handle real bug-fixing and development tasks.

You will also work with researchers to choose challenging repositories and issues, and lead junior engineers on collaborative projects.

• Analyze and triage issues across trending open-source libraries.
• Set up code repositories, including Docker environments and other local configuration.
• Review unit test coverage and assess its quality.
• Run and modify codebases locally to evaluate model performance on bug-fixing tasks.
• Help researchers identify repositories and issues that challenge large language models.
• Lead junior engineers on collaborative projects.

This is a remote contractor assignment with part-time options. The listed experience level is entry level, and the role also requires at least three years of software engineering experience.

• Open to candidates in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, and Mexico.
• Commit to at least 20 hours per week.
• Choose 20, 30, or 40 hours per week.
• Maintain four hours of overlap with Pacific Standard Time.
• Have strong Ruby experience.
• Be proficient with Git, Docker, and basic software pipeline setup.
• Understand and navigate complex codebases.
• Be comfortable running, modifying, and testing real-world projects locally.
• Have professional fluency in English.

Experience contributing to or evaluating open-source projects can help. Background in large language model research or evaluation, developer tools, or automation agents is also useful.

• Open-source project contribution or evaluation experience.
• Previous work on large language model research or evaluation projects.
• Experience building or testing developer tools or automation agents.

Apply on OpenTrain with your resume, then complete the application on the hiring site.

AI training work uses human judgment to help artificial intelligence systems learn from examples, including code, text, images, and model responses. OpenTrain helps people find these projects, build a profile, and apply to roles across the AI training industry.