VerifiedarXiv:2212.1056032 min
LLMs · Synthetic instruction data

Self-Instruct: Aligning Language Models with Self-Generated Instructions

GPT-3 writes 52,000 tasks for itself, starting from 175 written by people, and finetuning on those tasks teaches it to follow instructions.

A 2022 paper from the University of Washington and collaborators: prompt a plain pretrained model to invent tasks and solve them, discard the near-duplicates and malformed ones, and finetune the same model on what is left.

Explaining the paperSelf-Instruct: Aligning Language Models with Self-Generated InstructionsWang, Kordi, Mishra, Liu, Smith, Khashabi, Hajishirzi · ACL 2023 · arXiv:2212.10560 ↗

On 119 held-out NLP tasks, finetuning on the generated data took GPT3 from 6.8 to 39.9 ROUGE-L, 0.9 below OpenAI's text-davinci-001. Generating the data cost about $600, and an author judged only 54% of 200 sampled examples correct in every field.

Why instruction data was the bottleneck

A pretrained language model such as GPT-3 is trained to continue text. Given the line "Write a haiku about autumn.", the 175B-parameter davinci model may continue with more writing prompts, a forum reply, or a list, because those are what often follow such a line on the web. On the Super-NaturalInstructions benchmark used in this paper, plain GPT3 scores 6.8 ROUGE-L (a word-overlap score from 0 to 100, defined below), and the authors found that it "usually generates irrelevant and repetitive text, and does not know when to stop generation."

Instruction tuning addresses this: finetune the model on many tasks, each written as an instruction with the desired response as the target, and it learns to treat a new instruction as a request. FLAN and T0 built their tuning data by rewriting existing NLP datasets through hand-written templates. Super-NaturalInstructions (SuperNI) collected task definitions for 1,600+ tasks from contributors. InstructGPT used prompts and demonstrations written by contractors, together with prompts from OpenAI's API users. All of this is human-written, which makes it expensive, and the public collections lean toward the tasks NLP researchers already study, mostly classification, while users ask for things like a polite email declining a job offer or the CSS code for a named color.

Self-Instruct tests whether the pretrained model can write the tuning data itself. People write 175 seed tasks. GPT3 writes 52,445 new instructions and 82,439 input-output examples for them, heuristic filters remove invalid and repeated ones, and the same GPT3 is finetuned on the result. No person labels the generated data, and no stronger model is involved.

What counts as an instruction task

The paper defines a task tt by an instruction ItI_t in natural language and nt≥1n_t \ge 1 instances, each an input Xt,iX_{t,i} and an output Yt,iY_{t,i}. A model MM follows the instruction if

M(It, Xt,i)  =  Yt,i,i∈{1,…,nt}M(I_t,\, X_{t,i}) \;=\; Y_{t,i}, \qquad i \in \{1, \ldots, n_t\}
(1)

The input may be empty. "Write an essay about school safety" works as an instruction with no input, and the same task can be split into the instruction "Write an essay about the following topic" and the input "school safety". The paper allows both forms on purpose, to vary the data's format. Three rows from the released dataset:

instructioninputoutput
Given two integers, find their average.2, 3(2 + 3) / 2 = 2.5
Write a letter to your boss asking for an increase in salary.(empty)Dear [Boss], I am writing to you today to request a raise in my salary. ...
Given an article title, tell me if it's fake news or not.Article title: The new iPhone is here!Not fake news

Of the 82,439 generated instances, 35,878 (43.5%) have an empty input. The 175 seed tasks were written by the authors and their labmates at the University of Washington, each with one instruction and one instance, without reference to existing datasets or to the test sets. The paper counts 25 of them as classification tasks, meaning tasks whose output comes from a small fixed set of labels.

How Self-Instruct generates its own data

The pipeline has four steps:

  1. prompt GPT3 with 8 instructions from the task pool and let it write more;
  2. ask GPT3 whether each new instruction is a classification task;
  3. ask GPT3 to write inputs and outputs for each instruction, in an order that depends on the answer to step 2;
  4. filter out invalid or near-duplicate instructions and instances.

Steps 1 and 4 form a loop. Each request to the model puts 8 instructions in the prompt: 6 sampled from the human seeds and 2 sampled from the instructions GPT3 has already generated (the paper says the generated ones are there "to promote diversity"). The model continues the numbered list, every new line goes through the filters, and the lines that pass join the pool, where later requests can sample them. The pool starts with the 175 seeds and grows only through the filter. In the released code the loop runs until it has the requested number of instructions (the script asks for 50,000), and steps 2 and 3 then run once over the finished pool.

Figure 1 · the bootstrap loop, request by request
Come up with a series of tasks:
1. Question: in which year did Yao Ming retire? Answer
2. Rank the following universities.
3. Create a survey that asks about eating and cooking habits.
4. Come up with a question and a funny answer as a joke.
5. Give some examples of national dishes from a given country.
6. Give a detailed description of a method for brainstorming an essay.
7. Write a funny interview question. Why did you choose this question?
8. Given the name of an exercise, explain how to do it.
Press play or drag the slider. Left: the pool, 175 seeds (teal) and generated instructions (amber); outlined squares are the 8 sampled into the current prompt. Middle: the prompt. Right: the new lines GPT3 writes, and the filter verdict on each. The text below the figure is the prompt itself. Illustrative: the seed instructions and the kept lines are real ones from the paper's data, but these exact prompts did not produce those lines, and the rejected lines were written for this figure (their ROUGE-L scores are computed). The first request has no generated instructions to sample, so it uses 8 seeds.

Watch the counter under "task pool": each request adds 3 to 5 instructions in this run, and a prompt in request 2 already contains two of the instructions kept in request 1. The full run repeated this until the pool held 52,445 generated instructions, about 300 times the seed set. The whole pipeline, as pseudocode that follows the released scripts:

pool = seed_instructions                        # 175 written by people
machine = []
while len(machine) < 50_000:                    # scripts/generate_instructions.sh
    shots = sample(machine, min(2, len(machine)))
    shots += sample(pool, 8 - len(shots))       # 6 seeds once 2 machine exist
    text = davinci(numbered_list(shots), temperature=0.7, top_p=0.5,
                   presence_penalty=2, stop=["\n\n", "\n16", "16.", "16 ."])
    for inst in split_numbered(text):           # up to 7 lines: items 9-15
        if bad_length(inst) or bad_keyword(inst) or bad_start(inst):
            continue
        if max(rouge_l(inst, p) for p in pool + machine) > 0.7:
            continue
        machine.append(inst)                    # later prompts can sample it

for inst in machine:                            # once, after the loop
    clf = davinci(CLF_PROMPT + inst, temperature=0, max_tokens=3) == "Yes"
    tmpl = OUTPUT_FIRST if clf else INPUT_FIRST
    raw = davinci(tmpl + inst, temperature=0, top_p=0, presence_penalty=1.5)
    instances[inst] = filter_instances(parse(raw, clf))[:5]

Step 1: generating new instructions

The prompt is a numbered list with a one-line header, and the model's job is to continue it. Request 2 of Figure 1, as sent to the model (the code strips a trailing colon from each instruction):

Come up with a series of tasks:
1. Decide if the person likes dogs or cats.
2. Create a survey that asks about eating and cooking habits.
3. Explain the questions like I am 5.
4. Suggest some games that can be played by a group of people.
5. What is an important lesson that life taught you?
6. Select the oldest person from the list.
7. How do you answer this question in a job interview?
8. Generate a haiku using the following word
9.

GPT3 writes item 9 and keeps going. Decoding uses temperature 0.7 and top-p 0.5: each token is sampled, but only from the smallest set of candidates that holds half the probability. A presence penalty of 2 lowers the score of every token the model has already written in this completion, which pushes each new instruction toward words the earlier ones did not use. Generation stops at a blank line or when the model starts item 16, so one request returns at most 7 new instructions (items 9 to 15). A response that hits the 1,024-token limit is thrown away whole, since its last item is likely cut off.

Why sample from the pool instead of always showing the same 8 seeds? A fixed prompt would produce variations on 8 fixed tasks. Sampling different seeds each time spreads the requests over all 175, and the 2 generated instructions move the prompt toward tasks the seeds do not cover.

Step 2: classification or not

The next step needs to know whether an instruction is a classification task, because the two kinds get different instance prompts. The paper asks GPT3 itself, few-shot: the prompt lists 31 seed instructions, each followed by "Is it classification? Yes" or "No" (12 Yes, 19 No), then the new instruction. Decoding is greedy (temperature 0) with at most 3 tokens:

Can the following task be regarded as a classification task with finite output
labels?

Task: Given my personality and the job, tell me if I would be suitable.
Is it classification? Yes

Task: Give me an example of a time when you had to use your sense of humor.
Is it classification? No
... (31 examples in all: 12 Yes, 19 No)

Task: Given a word, return its suffix.
Is it classification?

The code counts a completion as a yes only if it is exactly "Yes", "yes" or "YES". Of the final 52,445 instructions, 11,584 (22%) were labeled classification and 40,861 not.

Step 3: writing inputs and outputs

For each instruction the model writes up to 5 instances. The obvious order is the one a user follows: write an input, then answer it. The paper calls this input-first and uses it for non-classification tasks. The prompt holds 12 fixed example tasks, each with its examples written out, then the new instruction. For classification tasks it uses output-first: the prompt holds 17 fixed classification tasks, each listing a class label followed by an input of that class, so the model first writes a label and then an input that belongs to it. One example task from each prompt:

# input-first (non-classification), one of 12 fixed examples in the prompt
Task: Sort the given list ascendingly.
Example 1
List: [10, 92, 2, 5, -4, 92, 5, 101]
Output: [-4, 2, 5, 5, 10, 92, 92, 101]

# output-first (classification), one of 17 fixed examples in the prompt
Task: Tell me if the following email is a promotion email or not.
Class label: Promotion
Email: Check out our amazing new sale! We've got discounts on all of your
favorite products.
Class label: Not Promotion
Email: We hope you are doing well. Let us know if you need any help.

The authors found that input-first produces inputs biased toward one label. For a grammar-checking task, the model writes the input before any label exists, so the input comes from what the model expects a sentence to look like, which is a grammatical one. Every instance then gets the label "grammatical", and finetuning on those instances pushes the model toward always answering "grammatical". Output-first writes the label first and conditions the input on it, so "ungrammatical" gets an input written to be ungrammatical.

Figure 2 · label balance, input-first against output-first
A toy model, not data from the paper: 24 grammar-checking tasks, one column each. Input-first, each of 5 instances is grammatical with the probability set by the slider. Output-first writes one instance per label. Drag the slider to 1.0 to see every input-first task end up with one label; switch modes, or resample the draws.

At the default 0.85, about 15% of input-first instances carry the minority label, and the chance that a task's 5 instances all share one label is 0.855+0.155≈0.440.85^5 + 0.15^5 \approx 0.44, so about 11 of the 24 tasks teach a single answer. Output-first has 50% minority labels and no single-label tasks by construction. The paper gives no measurement of the imbalance it saw, and its limitations section still lists unbalanced labels as a problem it did not solve.

Instance generation is greedy (temperature 0, top-p 0) with a presence penalty of 1.5 and stops at "Example 6" or at the next "Task:". Greedy decoding gives one fixed completion per instruction. In the released data the average is 1.57 instances per instruction, and 29,964 of the 52,445 instructions kept a single instance.

Step 4: the ROUGE-L novelty filter

A model prompted with 8 example instructions can write a copy of one with a word changed. The filter rejects those with ROUGE-L, a text-overlap score from summarization evaluation (Lin, 2004). It splits both texts into lowercase words, finds the longest common subsequence (LCS: the longest list of words that appears in both texts in the same order, gaps allowed), and combines precision and recall:

P=LCS(c,r)∣c∣,R=LCS(c,r)∣r∣,F=2PRP+RP = \frac{\mathrm{LCS}(c, r)}{|c|}, \quad R = \frac{\mathrm{LCS}(c, r)}{|r|}, \quad F = \frac{2PR}{P + R}P=LCS(c,r)∣c∣,R=LCS(c,r)∣r∣,F=2PRP+R\begin{gathered} P = \frac{\mathrm{LCS}(c, r)}{|c|}, \quad R = \frac{\mathrm{LCS}(c, r)}{|r|}, \\ F = \frac{2PR}{P + R} \end{gathered}
(2)

In (2), cc is the candidate instruction, rr an instruction already in the pool, and ∣⋅∣|\cdot| a word count. The candidate enters the pool only if F≤0.7F \le 0.7 against every instruction in the pool, seeds and generated alike. Two worked cases:

Figure 3 · ROUGE-L against ten seed instructions
Pick a candidate or type your own. Top: the candidate's words and those of the closest seed, with the longest common subsequence highlighted and linked. Bottom: F against each of the ten real seed instructions, with the threshold as a dashed line. The code rejects F above the threshold: at 0 any shared word rejects a candidate, and at 1 nothing is rejected, not even an exact copy.

The paraphrase preset, "Is the following sentence grammatically correct?", scores 0.26 against the grammar seed and passes, although it asks for the same task. ROUGE-L compares words in order, not meaning, so it removes lexical near-copies and lets paraphrases through. The paper does not say why it chose ROUGE-L over a similarity computed from sentence embeddings; ROUGE-L needs no extra model and targets the copy-with-a-word-changed failure that the prompt format invites.

Each candidate is compared with every instruction in the pool, so near the end of the run one candidate costs about 52,000 LCS computations; the code spreads them over 4 processes. Cheaper checks run first. The code drops an instruction if it has 3 words or fewer or more than 150, if it contains one of the keywords image, images, graph, graphs, picture, pictures, file, files, map, maps, draw, plot, or "go to" (tasks a text-only model cannot do), if it starts with "Write a program" (the model added that phrase to many instructions, which made it unclear whether the answer should be code or the result), or if it starts with punctuation or a non-ASCII character.

Instances have their own filters. An instance is dropped if its output is empty, if its output equals its input, or if its input or output ends in a colon (usually a cut-off generation). Exact duplicate instances are merged. If two instances of one task share an input but have different outputs, the released code drops every instance of that task, so the instruction leaves the dataset; the paper describes this as filtering out the conflicting instances.

Finetuning GPT3 on its own data

Each instance becomes one prompt-completion pair. The instruction and input are concatenated into the prompt through one of 7 templates (6 when the input is empty), chosen at random per instance, so the model does not learn to depend on one format. Two of the seven, as (prompt, completion) pairs from the code:

("Task: {instruction}\nInput: {input}\nOutput:", " {output}<|endoftext|>")   # 3
("{instruction}\n\n{input}\n\n", "{output}<|endoftext|>")                   # 5

Half the time, an input with a single field name such as "Article title:" loses the name. The completion is the output followed by <|endoftext|>, which teaches the model where to stop.

# prepare_for_finetuning.py, then the OpenAI fine-tunes API
for (inst, x, y) in generated_instances + seed_instances:   # 82,439 + 175
    if random() < 0.5:
        x = drop_single_field_name(x)       # "Article title: ..." -> "..."
        x = x.replace("\n\n", "\n")
    prompt_tmpl, completion_tmpl = choice(TEMPLATES_WITH_INPUT if x
                                          else TEMPLATES_WITHOUT_INPUT)
    write(prompt=prompt_tmpl.format(inst, x),
          completion=completion_tmpl.format(y))         # ends in <|endoftext|>
# dedupe exact (prompt, completion) pairs -> 82,612 lines, shuffle
# openai api fine_tunes.create --model davinci --n_epochs 2 \
#        --prompt_loss_weight 0

The authors finetuned davinci through OpenAI's fine-tunes API for 2 epochs with the default settings except one: prompt_loss_weight set to 0. The API trains with the next-token loss over the whole sequence, and that parameter scales the loss on prompt tokens. Schematically, for prompt tokens xx and completion tokens yy:

L(θ)=−∑j=1∣y∣log⁡pθ(yj∣x,y<j)  −  w∑k=1∣x∣log⁡pθ(xk∣x<k)\mathcal{L}(\theta) = -\sum_{j=1}^{|y|} \log p_\theta(y_j \mid x, y_{<j}) \;-\; w \sum_{k=1}^{|x|} \log p_\theta(x_k \mid x_{<k})L(θ)=−∑j=1∣y∣log⁡pθ(yj∣x,y<j)−w∑k=1∣x∣log⁡pθ(xk∣x<k)\begin{aligned} \mathcal{L}(\theta) &= -\sum_{j=1}^{|y|} \log p_\theta(y_j \mid x, y_{<j}) \\ &\quad - w \sum_{k=1}^{|x|} \log p_\theta(x_k \mid x_{<k}) \end{aligned}
(3)

With w=0w = 0, the gradient comes only from the output tokens: the model is trained to answer, and never to write instructions or inputs, which are machine-generated text with their own errors. The API default was w=0.01w = 0.01. The paper says only that 0 "works better in our case". OpenAI did not document which weights the API updates or which optimizer it uses, and the paper notes this.

Figure 4 · one training example and which tokens carry loss
template
A real generated instance under each of the 7 templates. Prompt tokens are grey and scaled by the slider; completion tokens (teal) always have weight 1. Words stand in for tokens here; GPT-3's tokenizer splits text into different pieces. The share readout assumes every token contributes the same loss.

Under template 1 the example has 29 prompt pieces and 4 completion pieces. At the API default of 0.01 the prompt would carry 0.29 / 4.29 = 6.8% of the loss (with equal per-token losses); at the paper's 0 it carries none. The released training file has 82,612 pairs: the 82,439 generated instances plus the 175 seed instances, minus duplicate pairs. Finetuning cost $338.

What the 52K dataset contains

statistic (Table 1)value
instructions52,445
classification / non-classification11,584 / 40,861
instances82,439
instances with empty input35,878
mean length: instruction / non-empty input / output15.9 / 12.7 / 18.9 words

At OpenAI's December 2022 price of $0.02 per 1,000 tokens, the $600 generation bill corresponds to about 30 million tokens. To check quality, an author took 200 random instructions with one random instance each and judged every field:

question (Table 2)yes
Does the instruction describe a valid task?92%
Is the input appropriate for the instruction?79%
Is the output a correct and acceptable response?58%
All fields are valid54%

Almost half the examples have an error somewhere, mostly in the output. Two from the released data: the instruction "Sort the given list in descending order." with input "List: 2, 1, 3, 4" has the output "[2, 1, 3, 4]", and "Explain the reasoning behind your answer to task 8." refers to a numbered task in the generation prompt that the finetuned model will never see. The authors argue that wrong examples still teach the format of a response, and the results below support training on the noisy data anyway.

On diversity, the paper parsed every instruction for a root verb and its direct object: 26,559 of the 52,445 instructions have that structure, and the top 20 verbs with their top 4 objects cover 14% of the set. We recomputed each instruction's highest ROUGE-L against the 175 seeds for 2,000 instructions drawn at random: the mean is 0.32, 47.5% score below 0.3, and none exceeds 0.7, which the filter guarantees.

Results on SuperNI

The first test is the SuperNI evaluation split: 119 tasks with 100 instances each, from task types held out of SuperNI's training split. Each model sees only the task definition and the input, with no solved examples (zero-shot), and decodes greedily. The score is ROUGE-L of the response against the reference, averaged and scaled to 0-100. For a one-word classification label this is close to exact match; for free-text answers it measures word overlap.

model (Table 3)paramsROUGE-L
T5-LM11B25.7
GPT3 (davinci)175B6.8
T011B33.1
GPT3 + T0 training175B37.9
GPT3 Self-Inst175B39.9
InstructGPT001 (text-davinci-001)175B40.8
Tk-Instruct (trained on SuperNI)11B46.0
GPT3 + SuperNI training175B49.5
GPT3 Self-Inst + SuperNI training175B51.6

The abstract's "33% absolute improvement" is 39.9 minus 6.8, so 33.1 ROUGE-L points. GPT3 + T0 training and GPT3 + SuperNI training are the paper's controls: the same davinci finetuned the same way on 50K instances sampled from the public human-written collections (PromptSource for T0, and SuperNI's training split). Self-Instruct data beats the T0 data by 2.0 points. The SuperNI-trained models score higher, which the authors attribute to their training tasks sharing SuperNI's style and format, and adding the Self-Instruct data to SuperNI training still adds 2.1 points.

InstructGPT001 is OpenAI's text-davinci-001 API engine. OpenAI's model index lists it as trained by supervised finetuning on human demonstrations and on highly rated model outputs, without the reinforcement-learning stage of the InstructGPT paper. The authors compare against 001 because it is the closest to their own supervised setup, and because newer engines were trained on recent user data and may have seen the SuperNI test set. The 0.9 point gap is to a model tuned on private human-written data.

Results on user-oriented tasks

SuperNI tasks are research datasets, skewed toward classification. For a second test, the authors wrote 252 new instructions aimed at uses of a deployed model (email writing, social media, productivity tools, entertainment, programming), each with one instance, in varied formats: bullet lists, tables, code, equations. Many cannot be scored automatically, so the instruction authors rated every model's response on four levels:

Two authors rated all responses independently, with model names hidden and the order shuffled. The reported numbers are from one of them. Agreement between the two is Cohen's κ = 0.58 on the four levels (0.75 on acceptable versus not), and Spearman correlation 0.81 when the levels are treated as ordered.

Figure 5 · human ratings of 252 responses per model
counts as acceptable:
GPT3 Self-Inst: A 112, B 49, C 66, D 25; acceptable 161/252 = 63.9%
Self-Inst trails InstructGPT 001 by 12 responses (4.8 points)
Counts from Figure 6 of the paper. Choose which ratings count as acceptable; the share above each bar and the readout below follow. Click a bar for its counts. Left of the dashed line: GPT3 and GPT3 finetuned on public or generated data. Right: OpenAI's engines.

Counting A and B as acceptable, GPT3 Self-Inst gets 161 of 252 responses (63.9%) and InstructGPT001 gets 173 (68.7%): the paper's "5% behind" is 12 responses, 4.8 points. GPT3 finetuned on SuperNI data gets 104 (41.3%), so on these tasks the generated data is far more useful than the human-written research data, the reverse of the SuperNI result. Combining SuperNI training with Self-Instruct data, the best model on SuperNI, gets 137 here, 24 fewer than Self-Instruct data alone. Plain GPT3 gets 1 acceptable response, and text-davinci-003, trained with reinforcement learning from human feedback, gets 222.

More data and better outputs

Two follow-up experiments use the same 252 instructions and count only A ratings. The first finetunes GPT3 on random subsets of the generated instructions, from the 175 seeds alone up to all 52,445. The second keeps every generated instruction and input but replaces each output with one written by InstructGPT003, which makes it distillation from a stronger model, as the paper says.

Figure 6 · rating-A share against amount and quality of data
52,445 (all): 112 of 252 rated A = 44.4% (interval ±6.1 points)
Values from Figure 7 of the paper; click a point or a button for its count out of 252. The optional whiskers are our addition: 95% binomial intervals for a score measured on 252 instructions, which show how far the same model could move on a different set of 252 similar instructions.

From 175 to about 16K instructions the A share rises from 31.0% to 43.7%, and the full 52,445 add 0.7 points, which is 2 more responses out of 252. The paper describes this as a plateau after 16K. With 252 test items, the 95% interval on each point is about ±6 points, so the curve can show the rise from 175 to 16K but cannot distinguish 16K from 52K. On SuperNI, the authors report, the gain levels off after a few hundred instructions, which fits the generated tasks being unlike SuperNI's. Rewriting the outputs with InstructGPT003 lifts the A share from 44.4% to 54.4%, 25 more responses (10.0 points), against 0.7 points from adding the instructions beyond 16K.

Limits and what came after

The paper lists three limitations. The method depends on what the pretrained model already knows, so gains likely concentrate on tasks that are frequent in pretraining text and may be small for rare ones. It may need a large model; only the 175B davinci was tested, and the authors point out that instruction tuning with human data also helps large models more. And an iterative loop can amplify the model's biases, including social stereotypes and the label imbalance from Step 3, which output-first reduced but did not remove.

Two more limits come from the evaluation. The seed tasks, the 252 test instructions and the ratings all come from the authors; Appendix A.1 puts the mean ROUGE-L between each seed and its closest test instruction at 0.34, and one seed, "Answer the following question.", appears verbatim in the test set. The reported ratings are also one rater's, and agreement with the second rater is moderate on the four levels (κ = 0.58) and higher on acceptable versus not (κ = 0.75).

The paper appeared on arXiv in December 2022, the same week as Unnatural Instructions (Honovich et al.), which generated instruction data with InstructGPT002 from SuperNI seed tasks. Stanford's Alpaca (March 2023) reused the Self-Instruct code with text-davinci-003 as the generator, 20 instructions per request, one instance per instruction and no classification split, for under $500, and finetuned LLaMA 7B on the 52K result. Generating instruction data with a strong model became common: LLaVA used text-only GPT-4 to write its image conversations, and Constitutional AI (also December 2022) replaced human harm labels with model-written critiques and preferences.

Provenance Verified against primary literatureHow we verify
Wang et al. (2022), Sections 2-4 and Appendix AMethod and numbers as taught here: 175 seed tasks, 8 in-context instructions (6 seed, 2 generated), the classification prompt with 12 + 19 seed examples, input-first and output-first instance generation, the 0.7 ROUGE-L threshold, Table 1 statistics, the Table 2 quality audit, Table 3 SuperNI scores, the Table 4 decoding settings, and the $600 and $338 costs.
yizhongw/self-instruct, self_instruct/bootstrap_instructions.py lines 19-70 and 160-215The prompt is a numbered list ("Come up with a series of tasks:" then "1." to "8." and an open "9."), not the "Task 1:" form printed in Table 5. Two machine instructions are sampled only once two exist, so the first requests use 8 seeds. A response cut off by the length limit is discarded whole. A candidate is rejected when its ROUGE-L F-measure (rouge_score, no stemmer) against any pool instruction exceeds 0.7. Keyword list: image(s), graph(s), picture(s), file(s), map(s), draw, plot, go to.
generate_instances.py lines 133-155; scripts/generate_instances.shInstances are generated once per instruction after the instruction loop finishes, greedily (temperature 0, top_p 0), presence penalty 1.5, at most 5 instances (stop at "Example 6" or "Task:"). Max tokens is 300 when the batch holds a classification task and 350 otherwise.
prepare_for_finetuning.py lines 54-91 and 269-303; scripts/finetune_gpt3.shSeven prompt templates for instances with input and six without, chosen at random per instance. Half the time a single "Field:" name is stripped from the input. Seed instances are included. The released training file has 82,612 lines, and 23.4% of prompts start with "Task:", which matches the 2-of-7 and 1-of-6 template shares. Fine-tuning: davinci, 2 epochs, prompt_loss_weight 0.
Huerta-Enochian & Ko (2024), arXiv:2401.13586The legacy OpenAI fine-tunes API weighted prompt tokens by prompt_loss_weight with a default of 0.01; the parameter was removed in January 2024. Equation (3) on this page is a schematic of that weighting; OpenAI never documented the exact normalization, and the paper notes it could not see which parameters the API updates.
OpenAI model index for researchers (2022), as quoted by secondary sourcestext-davinci-001, which the paper calls InstructGPT001, is listed as FeedME: supervised finetuning on human demonstrations and on model samples rated highly by labelers. text-davinci-003 is listed as trained with PPO. The PPO-ptx model of the InstructGPT paper is neither engine. Could not load the original page, which needs JavaScript.
Figures 6 and 7 of the paper, rereadFigure 6 bar counts (each column sums to 252) are printed on the bars. Figure 7 prints its percentages; each is a whole number of responses out of 252. The x positions of its two middle points (about 1K and 16K) are read from the log axis. The 95% intervals in this page's Figure 6 are our addition, not the paper's.
Our recount of the released dataall_instances_82K.jsonl: 82,439 instances, 52,445 unique instructions, 35,878 empty inputs, mean lengths 15.9 / 12.7 / 18.9 words (matches Table 1). 29,964 instructions keep one instance. Max ROUGE-L against the 175 seeds, on 2,000 instructions sampled at random: mean 0.32, median 0.30, none above 0.7.
Taori et al. (2023), stanford_alpaca READMEAlpaca reused the pipeline with text-davinci-003 as the generator, 20 instructions per request, no classification split, one instance per instruction, at under $500, and finetuned LLaMA 7B on the 52K result.
correctionThree checkable slips. (1) Section 2.2 says instances with the same input but different outputs are filtered out. The released code does more: filter_duplicate_instances in prepare_for_finetuning.py (lines 110-125) returns an empty list for the whole task when any two of its instances conflict, so every instance of that task is dropped and the instruction never reaches the training file. This page teaches the code's behavior. (2) Section 4.4 reports inter-rater agreement κ = 0.57 on the 4-level ratings, and Appendix B.2 reports κ = 0.58 for the same ratings. (3) Appendix A.1 counts 25 classification and 150 non-classification seed tasks, and its footnote 11 points to human_eval/user_oriented_instructions.jsonl, which is the 252-instruction test set. The seed tasks are in data/seed_tasks.jsonl, where 26 of the 175 are marked as classification. Separately, the headline "33% absolute improvement" is 33.1 ROUGE-L points (6.8 to 39.9), and "only 5% behind InstructGPT001" is 12 of 252 responses, 4.8 points, counting ratings A and B as acceptable.

Questions you might still have

?

If GPT3 wrote the data, how can training on it teach GPT3 anything new?
The finetuning data mostly changes the format of the model's behavior. The generated outputs come from prompts that contain other solved tasks, so GPT3 writes them in its few-shot mode, where it performs better. Finetuning then trains the model to produce that kind of output from an instruction alone, with no examples in the prompt, and to end the answer with an end-of-text token, which fixes the untuned model's habit of rambling on. The filters add selection on top: only instructions that pass the length, keyword and ROUGE-L checks, and instances that pass the duplicate checks, are kept. The paper does not run an ablation that separates these effects, so this is an interpretation of its results, not a measured breakdown.

?

Is Self-Instruct a form of distillation?
Section 5 of the paper calls it distillation from a model into itself: the teacher and the student are the same GPT3 davinci, and what is transferred is a set of tasks with examples, not output probabilities. Classic knowledge distillation (see the Knowledge Distillation explainer on this site) trains a smaller student on a larger teacher's probabilities. The one experiment in the paper that uses a stronger teacher, regenerating outputs with InstructGPT003, is ordinary distillation and is labeled as such.

?

How is Alpaca different from Self-Instruct?
Stanford's Alpaca (March 2023) ran a modified version of this pipeline with text-davinci-003, an RLHF-tuned model, as the generator instead of the untuned davinci, asked for 20 instructions per request, dropped the classification and non-classification split, generated one instance per instruction, and finetuned LLaMA 7B on the 52K result. Because the generator was already instruction-tuned, Alpaca distills a stronger model into a weaker one, which Self-Instruct avoided on purpose.

?

Why a ROUGE-L threshold of 0.7?
The paper gives no ablation of the threshold. ROUGE-L F above 0.7 means the longest common word subsequence covers most of both instructions, which catches a copied in-context example with one or two words changed. It does not catch a paraphrase: "Is the following sentence grammatically correct?" scores 0.26 against the seed "Tell me if the sentence is grammatical. If it is, output yes. Otherwise, correct the grammar errors." and would be kept. Raising the threshold admits more near-copies; lowering it rejects more distinct tasks that happen to share wording, as Figure 3 shows.

?

Does this work for small models?
The paper only runs it on the 175B davinci model and lists this as a limitation: the method relies on what the pretrained model already knows, so it might work best for large models. Later projects such as Alpaca applied data generated by a large model to smaller ones, which is a different setup because the generator and the trained model differ.

?

Can the experiment be rerun today?
Not exactly. The generation and finetuning both ran through OpenAI's API on the original davinci engine and the legacy fine-tunes endpoint, and OpenAI retired both in January 2024. The released code, the 175 seed tasks, the 82K generated instances, the finetuning file, the 252 evaluation instructions and every model's predictions on them are public, so the pipeline can be run against an open base model, and the published data can be used to finetune one.

Footnotes & further reading

  1. The paper: Wang, Kordi, Mishra, Liu, Smith, Khashabi & Hajishirzi, Self-Instruct: Aligning Language Models with Self-Generated Instructions (ACL 2023; arXiv December 2022). Code and data: github.com/yizhongw/self-instruct.
  2. The base model: Brown et al., Language Models are Few-Shot Learners (2020); see the GPT-3 explainer.
  3. The benchmark and Tk-Instruct: Wang et al., Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (EMNLP 2022).
  4. ROUGE: Lin, ROUGE: A Package for Automatic Evaluation of Summaries (2004). The code uses Google's rouge_score package.
  5. Instruction tuning: Wei et al., Finetuned Language Models Are Zero-Shot Learners (2021), see the FLAN explainer; Sanh et al., Multitask Prompted Training Enables Zero-Shot Task Generalization (T0, 2021).
  6. InstructGPT: Ouyang et al., Training language models to follow instructions with human feedback (2022); see the InstructGPT explainer.
  7. The prompt loss weight: Huerta-Enochian & Ko, Instruction Fine-Tuning: Does Prompt Loss Matter? (2024).
  8. Concurrent work: Honovich et al., Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022).
  9. Alpaca: Taori et al., Stanford Alpaca: An Instruction-following LLaMA model (2023).
  10. Distillation, for comparison: Hinton, Vinyals & Dean, Distilling the Knowledge in a Neural Network (2015); see the Knowledge Distillation explainer.