Skills in high demand

AI Experts

Instruction & Prompt Guidelines for Reliable Labeling

A practical guide for writing clear instructions and prompts that produce consistent labels. Use these templates, examples, and quality checks to reduce ambiguity, raise agreement, and keep datasets stable across reviewers and releases.

Hire Talent

Who This Is For

Product, research, and data leaders who need reproducible labeling and evaluation across text, image, audio, video, documents, code, and multimodal tasks.

Core Principles for Reliable Instructions

Write for the exact decision you want. One instruction should map to one decision.
Define terms before you ask reviewers to use them.
Prefer objective rules over subjective language.
Show examples and counterexamples that match real difficulty.
State how to handle ambiguity and when to escalate.
Version guidelines and track changes so everyone stays aligned.

Labeling Guideline Template

Use this outline for any labeling or evaluation task.

1. Task Definition

Describe the goal in one or two sentences. State the expected input and the expected output.

2. Label Set And Definitions

List each label. Give a short definition and one clear example.

3. Decision Rules

Write step-by-step rules that decide the label. Keep each step a single action.

4. Examples And Counterexamples

Include three to five examples for each label. Add one tricky counterexample for edge cases.

5. Ambiguity Policy

State what to do when the rules do not apply. Tell reviewers to choose a default label or to escalate.

6. Escalation Path

Name who to ask and how to log a question. Include a one-line service level for responses.

7. Quality Targets

Set inter-annotator agreement targets and gold-test minimums. Give a short note on how you measure them.

8. Submission Checklist

Add a brief checklist that reviewers run before they submit.

9. Tooling Conventions

Describe hotkeys, file naming, and required fields.

10. Privacy And Security

List redaction rules and any restricted content.

11. Version And Ownership

Add the version number, date, and the owner of the guideline.

Prompt Writing Basics for Labeling and Evaluation

Begin with the task and the expected output format.
Specify the label set or the rubric dimension.
Ask for a short rationale when it helps catch mistakes.
Set limits on length, time, and scope.
Remind reviewers what to do when they are unsure.

Good Prompt Template

“Read the text and assign one label from {Positive, Neutral, Negative}. Quote the phrase that drove your decision in one short sentence. If you are unsure, choose Neutral and write ‘unsure’ as the rationale.”

Weak Prompt Example

“Read and label the text.”

Rubrics for RLHF and Preference Work

Use a simple, repeatable rubric that focuses on the real task. 

Helpfulness: Does the answer solve the user’s request and follow the instruction.
Correctness: Is the content accurate and free of unsupported claims.
Safety and Policy: Does the answer follow policy and avoid risky content.
Style and Tone: Does the answer match the desired tone and format.

Ask raters to rank two answers side by side. Require a one-line rationale that cites the rule they used. Keep scales short and clear. For most tasks a three-point or five-point scale is enough.

Positive and Negative Instruction Examples

Clear Instruction

“Mark an entity as PERSON if the name refers to a human. Do not mark organizations or products. For ‘Jordan,’ mark as PERSON only if a human is meant. If you cannot tell, select Unknown and add a note.”

Unclear Instruction

“Tag people and things that feel like people.”

Edge Cases and Ambiguity

Add a short section for known tricky cases. State the expected behavior in one sentence per case. Ask reviewers to log new edge cases with a short note and a suggested rule. Review these notes weekly and update the guideline.

Calibration and Change Management

Run a small pilot with a gold set before full production.
Review disagreements and update rules that cause confusion.
Re-run the pilot until agreement meets the target.
Announce changes with a short summary and a new version number.
Keep old versions for reference and audit.

Quality Gates and Metrics

Gold-Test Pass Rate: Use a small gold set in every batch.
Inter-Annotator Agreement: Track agreement and coach when it drops.
Escalation Resolution Time: Measure how quickly questions get answered.
Re-Label Rate: Watch how often labels are corrected after review.
Release Gates: Block training or release if a gate falls below the target.

Prompts for Synthetic Generation and Safe Use

Use synthetic data to expand coverage, not to replace human judgment.

Generation Prompt Template

“Create ten realistic examples for the task described below. Vary difficulty and include at least two edge cases. Do not repeat phrases. Return in JSON with fields input, expected_output, and notes.”

Safety Checks

Filter near duplicates. Remove artifacts and policy violations. Label a sample with humans and only promote batches that pass evaluation.

Frequently Asked Questions

Do I always need rationales in prompts?

Use rationales when they improve quality or uncover confusion. Skip them for simple tasks with clear rules.

How often should I update instructions?

Review weekly during early production and monthly once metrics stabilize. Version every change and summarize what shifted.

Can I reuse prompts across modalities?

You can reuse structure, but you should write examples that match each modality. Reviewers will perform better when examples look like the real data.

Ready to raise label quality?

Tell us your tasks, tools, languages, and timelines. We will provide you with the right staff who can write clear instructions and prompts, set strong quality gates, and keep everything consistent.

Hire Talent