AI Engineering5.0 · 50 ratings
Annotation Guideline Writer
**Role:** ML labeler-lead. Has written guidelines for 5+ labeling projects. **Context:** Team needs to label [TASK]. Quality is uneven acro…
Role-BasedChain-of-Thought
Prompt
**Role:** ML labeler-lead. Has written guidelines for 5+ labeling projects. **Context:** Team needs to label [TASK]. Quality is uneven across labelers because guidelines are vague. **Task:** Write the guideline: 1. The task in 2 sentences. 2. Inputs the labeler sees + how to interpret each field. 3. Labels: each label with definition + 2 positive + 2 negative examples. 4. Edge cases: 10 specific scenarios that have caused inter-annotator disagreement, with the resolved answer. 5. Resolution: when 2 labelers disagree, what process resolves. 6. Quality checks: what % is dual-labeled, what κ is acceptable. 7. Labeler training: how new labelers are onboarded. 8. Inter-annotator agreement targets. **Constraints:** - Every label has positive + negative examples. - Ambiguous cases get explicit resolution rules. **Output format:** Guideline doc + sample labels + quality check protocol.
Recommended models
claudegpt-4o
More in AI Engineering
RAG vs Fine-tune Decision Memo
**Role:** You are a senior AI engineer who has shipped both RAG-based and fine-tuned LLM products at production scale. You believe most team…
Read prompt
Evals Harness Design for [Domain]
**Role:** AI engineer who has built evals suites that have caught 30+ production regressions before they shipped. You believe vibes-based "t…
Read prompt
System Prompt Audit
**Role:** Senior prompt engineer who has audited 100+ production system prompts. You read prompts the way an editor reads prose — for the me…
Read prompt
Agent Loop Halt-Condition Design
**Role:** Applied AI engineer who has shipped agents that completed millions of tool-calling iterations in production. You believe most agen…
Read prompt