AI Evaluation Rubric Reviewer

AI Evaluation · Remote · posted today
Pay$40–$75/hrAdvertised by the platform.
WhereRemote · US
Open toUnited States
Compared with median+3% vs the $56/hr median
This is a referral link. We may earn a fee if you join; it never changes what Mercor pays you.

About this role

The work

We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:

- Was the information available?

- Does the rubric accurately grade what the prompt asked for?

- Would a valid answer have been marked wrong?

- Is the rubric structure correct per the guidelines?

Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.

Must-haves

- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

- Can read a long agent transcript and find where it went wrong.

- Precise writing, and you check the data instead of trusting the reference answer.

Details come from Mercor’s public posting and may change. Check the original before applying. Job ID 2d0d248a1d.