LLM Annotator, Master's Degree

You will help evaluate and improve large language models by testing how well they retrieve, analyze, and reason over structured data. You will review AI-generated answers, identify failures, and explain your findings using clear evidence.

The work includes prompt design, response evaluation, dataset validation, and annotation according to project guidelines. You will focus on factual accuracy, logical reasoning, completeness, hallucinations, inconsistencies, and gaps in reasoning.

• Create challenging prompts that test model performance on structured data.
• Assess AI-generated responses for factual accuracy, reasoning quality, and completeness.
• Identify hallucinations, inconsistencies, reasoning gaps, and other model failures.
• Validate outputs against datasets and supporting evidence.
• Write clear explanations that justify your evaluations.
• Follow annotation guidelines and maintain consistent quality standards.

This is a remote contractor assignment with a four-week duration. The role is listed as entry level, while the requirements include a master's degree or higher and at least three years of professional, research, or teaching experience.

• Pay: Not provided in the listing.
• Schedule: Choose 20, 30, or 40 hours per week.
• Time zone: At least four hours of overlap with Pacific Time is required.
• Location: Remote and open worldwide.
• Education: Master's degree or higher in any discipline.
• Experience: At least three years of professional, research, or teaching experience.
• Communication: Excellent written English.
• Skills: Strong analytical and critical-thinking abilities, careful attention to detail, and the ability to validate information against source data.
• Helpful background: Large language models, generative AI, prompt engineering, AI evaluation, data annotation, model testing, or structured datasets such as CSV files, spreadsheets, and databases.
• Additional strength: Ability to identify edge cases and design prompts that expose model limitations.

Apply on OpenTrain with your resume, then complete the application on the hiring site.

AI training is the human work behind systems that learn from examples, including writing prompts, rating model responses, and checking facts. People with strong analytical skills or relevant experience help make these systems more accurate and useful, while OpenTrain provides a free place to find projects and build an AI training career.