AI Evaluation Analyst (Remote)
Crewlia
What the role involves
- Rating AI responses
- Comparing paired outputs
- Checking scoring consistency
What they ask for
At least 18 years old; able to communicate in English; reside in an eligible country; reliable high-speed internet and suitable home-working setup; able to compare responses and explain preferences with text-based evidence
About the job
- Role: AI Evaluation Analyst (Remote)
- Location: Remote (Australia, Canada, Ireland, New Zealand, United Kingdom, United States)
- Pay: US$18 per hour
Applicants must currently reside in Australia, Canada, Ireland, New Zealand, the United Kingdom or the United States.
Overview
We are recruiting analytical contributors to rate and compare AI responses across defined performance dimensions. You will apply scoring rules, analyze why outputs differ, and produce structured evidence for each evaluation decision.
Key Responsibilities
- Rate responses separately for correctness, relevance, completeness, reasoning quality, and instruction adherence.
- Compare paired outputs and select the stronger response using the stated tie-breaking rules.
- Identify the failure mode that most influenced a low score or preference decision.
- Check rating consistency against calibration items and revise decisions when the rubric requires it.
- Summarize evaluation evidence without speculating about how the underlying system was built.
Requirements
- Must be at least 18 years old and able to communicate in English.
- Ability to keep multiple scoring dimensions separate while forming an overall evaluation.
- Ability to compare responses and explain a preference with text-based evidence.
- Reliable high-speed internet and a suitable home-working setup.
- Previous experience with AI evaluation, analytical review, or rubric-based scoring is helpful but not required.
Equal Opportunity
We welcome applications from qualified individuals of all backgrounds. Applicants are considered based on relevant skills, qualifications and project eligibility requirements, without discrimination on grounds protected by applicable law.
Apply Now!
Applying for this job
What to show on your CV
- For the work it describes, like "Rating AI responses", give an example from your own experience, with a result you can talk about at an interview.
Our guides to the New Zealand CV format, cover letters and interview questions cover the rest.