AI Engineering5.0 · 50 ratings

Annotation Guideline Writer

**Role:** ML labeler-lead. Has written guidelines for 5+ labeling projects. **Context:** Team needs to label [TASK]. Quality is uneven acro…

Role-BasedChain-of-Thought

Prompt

**Role:** ML labeler-lead. Has written guidelines for 5+ labeling projects.

**Context:** Team needs to label [TASK]. Quality is uneven across labelers because guidelines are vague.

**Task:** Write the guideline:
1. The task in 2 sentences.
2. Inputs the labeler sees + how to interpret each field.
3. Labels: each label with definition + 2 positive + 2 negative examples.
4. Edge cases: 10 specific scenarios that have caused inter-annotator disagreement, with the resolved answer.
5. Resolution: when 2 labelers disagree, what process resolves.
6. Quality checks: what % is dual-labeled, what κ is acceptable.
7. Labeler training: how new labelers are onboarded.
8. Inter-annotator agreement targets.

**Constraints:**
- Every label has positive + negative examples.
- Ambiguous cases get explicit resolution rules.

**Output format:** Guideline doc + sample labels + quality check protocol.

How to use this prompt

  1. 1

    Copy the prompt above and paste it into ChatGPT, Claude, or Gemini — or open it in the visual Studio to edit each part on a canvas and run it with your own key.

  2. 2

    Replace any bracketed placeholders with your specifics. The more concrete your context and constraints, the sharper the result — see the 5-part prompt structure.

  3. 3

    Run it, then refine. Ask the model to critique and improve its own answer with self-critique prompting.

Techniques in this prompt

Role-Based

Assigns the model an expert persona so it adopts the right vocabulary, depth, and standards for the task.

Learn this technique
Chain-of-Thought

Asks the model to reason step by step before answering — ideal for multi-step, logical, or analytical tasks.

Learn this technique

Recommended models

claudegpt-4o

Build on this prompt

Open it in the visual Studio to wire it into a full workflow with your own API key — or learn the craft behind prompts like this.

More in AI Engineering