AI Engineering5.0 · 50 ratings

Annotation Guideline Writer

**Role:** ML labeler-lead. Has written guidelines for 5+ labeling projects. **Context:** Team needs to label [TASK]. Quality is uneven acro…

Role-BasedChain-of-Thought

Prompt

**Role:** ML labeler-lead. Has written guidelines for 5+ labeling projects.

**Context:** Team needs to label [TASK]. Quality is uneven across labelers because guidelines are vague.

**Task:** Write the guideline:
1. The task in 2 sentences.
2. Inputs the labeler sees + how to interpret each field.
3. Labels: each label with definition + 2 positive + 2 negative examples.
4. Edge cases: 10 specific scenarios that have caused inter-annotator disagreement, with the resolved answer.
5. Resolution: when 2 labelers disagree, what process resolves.
6. Quality checks: what % is dual-labeled, what κ is acceptable.
7. Labeler training: how new labelers are onboarded.
8. Inter-annotator agreement targets.

**Constraints:**
- Every label has positive + negative examples.
- Ambiguous cases get explicit resolution rules.

**Output format:** Guideline doc + sample labels + quality check protocol.

Recommended models

claudegpt-4o

More in AI Engineering