ReAct: Synergizing Reasoning and Acting in Language Models

A language model writes a thought, calls a tool, reads the result, and repeats until it can answer.

ReAct prompts a frozen language model with a few hand-written examples in which reasoning steps alternate with actions such as a Wikipedia search. The real search results are pasted back into the prompt, so later reasoning works from retrieved text instead of the model's memory.

Princeton and Google Brain, October 2022. All main results come from PaLM-540B with no training, on four benchmarks: two question-answering tasks backed by a three-command Wikipedia API, a text adventure set in a house, and a shopping website. The Thought, Action, Observation format it introduced is the one many later agent frameworks adopted.

The page follows the paper's order:

1. the two kinds of failure ReAct was built against, on one question from the paper;
2. the formal change (thoughts as actions that only edit the context) and the loop that runs it;
3. results on question answering, and where ReAct and chain-of-thought each fail;
4. the fallback rules that combine the two, and finetuning smaller models on ReAct episodes;
5. the decision-making tasks, where thoughts are sparse, and what later work found about why ReAct works.

Reasoning without looking, acting without thinking

By 2022 there were two separate ways to use a large language model on a multi-step task. Chain-of-thought prompting puts a few worked examples with written-out reasoning in the prompt, and the model then writes its own reasoning before the answer. All the facts in that reasoning come from the model's weights. If it misremembers one in step two, steps three and four build on the wrong fact, and nothing in the process checks it. The paper calls these failures fact hallucination and error propagation.

The other line used language models as policies in interactive environments: the model reads the current state as text and outputs the next action, such as a robot command or a browser click. Those systems acted but wrote down no reasoning between actions, so nothing in the text tracked the goal, the progress so far, or why the last action failed. (Inner Monologue, covered below, had the model restate facts about the scene, a limited exception.)

Figure 1 of the paper runs four prompting styles on one HotpotQA question: "Aside from the Apple Remote, what other device can control the program Apple Remote was originally designed to interact with?"

• Standard (question in, answer out) answers "iPod".
• Chain-of-thought reasons that the Apple Remote was designed for the Apple TV, which can be controlled by the iPhone, iPad and iPod Touch, and answers with those three. The first step is wrong: the remote was designed for a program called Front Row.
• Act only searches "Apple Remote" and gets back the sentence saying it was designed to control the Front Row media center program. It then searches "Front Row" (no page), then "Front Row (software)", gets the page, and finishes with the answer "yes". The facts were in its context; no step combined them into an answer.
• ReAct writes "I need to search Apple Remote and find the program it was originally designed to interact with", searches, writes that it now needs to search Front Row, gets a not-found result, writes "Front Row is not found. I need to search Front Row (software)", reads that page, and writes "Front Row (software) is controlled by an Apple Remote or the keyboard function keys. So the answer is keyboard function keys." It answers "keyboard function keys".

The ReAct trace shows the two directions the title's "synergizing" refers to. Reasoning steers the actions: a thought decides what to search for, and another reformulates the query after a failed search. Actions feed the reasoning: the fact about Front Row comes from the page, not from memory. The paper's names for the two directions are "reason to act" and "act to reason".

How ReAct works

The paper starts from the usual agent setup. At step $t$ the agent receives an observation $o_t$ from the environment and picks an action $a_t$ according to a policy $\pi$, conditioned on everything seen so far:

$c_t$ is the context. For a language-model agent it is literally the text in the prompt, and the policy is the model's distribution over continuations of that text. The paper's argument for why this is hard: the mapping from $c_t$ to the right action can require a lot of computation that is never written down. In the Act-only run above, the right final action needed the question and three observations combined, and the model produced "yes".

ReAct enlarges the set of allowed actions with free-form language:

$\mathcal{A}$ holds the task's real actions (search a page, open a drawer) and $\mathcal{L}$ is the space of language. An action $\hat{a}_t \in \mathcal{L}$ is a thought. The environment never sees it and returns no observation for it; its only effect is to extend the context:

A thought therefore works like scratch space the model writes for itself. The paper lists the kinds it observed: decomposing the goal into a plan, injecting commonsense ("a pepper shaker is more likely to appear in cabinets or on countertops"), pulling the relevant part out of an observation, tracking progress, and handling an exception such as a failed search.

The obvious alternative would be to train a policy that outputs thoughts and actions. The paper notes that $\mathcal{L}$ is unbounded, so learning a policy over it from scratch needs strong language priors, and takes those priors from a frozen pretrained model shown a handful of examples. Each example is a complete episode that a person wrote by typing thoughts next to the actions they took. Nothing is learned in the weights. Written out at the level of tokens, the policy is the model's next-token distribution under a fixed prompt prefix $P$ (instruction plus examples), with the action string $a_t = (w_1, \dots, w_K)$ decoded greedily:

The decoder does not distinguish thoughts from actions; the harness does. On the two question-answering tasks the examples alternate strictly (thought, action, observation, thought, ...), so every model call writes one thought and one action. On the decision-making tasks, which can run for dozens of actions, the examples include thoughts only at a few points and the model decides for itself when to write one.

The whole HotpotQA agent fits in a dozen lines. Condensed from the authors' released code, which runs GPT-3 because PaLM was not public:

# Condensed from hotpotqa.ipynb in github.com/ysymyth/ReAct (GPT-3 release)
prompt = INSTRUCTION + SIX_EXAMPLES + question + "\n"   # 1,016-word prefix
for i in range(1, 8):                                   # at most 7 steps
    out = llm(prompt + f"Thought {i}:",
              stop=[f"\nObservation {i}:"])             # greedy decoding
    thought, action = out.strip().split(f"\nAction {i}: ")
    obs, done = env.step(action)        # search[...], lookup[...], finish[...]
    prompt += (f"Thought {i}: {thought}\nAction {i}: {action}\n"
               f"Observation {i}: {obs}\n")
    if done:
        break
if not done:
    env.step("finish[]")                # out of steps: an empty answer
# The language model's weights never change. No step computes a gradient.

The stop string "\nObservation i:" ends each model call right where an observation would start. Without it the model would keep following its examples, which put an observation after every action, and write one of its own, the kind of invented fact ReAct is meant to remove. The prompt is also the only state: there is no memory object, planner, or tool registry beyond string parsing of search[...], lookup[...] and finish[...].

Figure 1 replays one episode phase by phase. The text is the first of the paper's six HotpotQA example episodes, written by the authors; in a real run the teal lines come from the model and the amber lines from the API. Watch three things: the model is idle while the API runs, every model call ends at the dashed stop marker, and the context grows by the episode on top of the fixed 1,016-word prefix.

In step 3 the search lands on a disambiguation page for "High Plains" ("refers to one of two distinct land regions"), and Thought 4 reformulates the query to "High Plains (United States)". The Act version of this example makes the same reformulation with no line saying why, so an Act prompt shows the model the pattern without the reason. At the end the episode has added about 230 words to the prefix, and each of the five model calls re-read everything above it.

The Wikipedia actions and the baselines

HotpotQA asks multi-hop questions whose answers need facts from two or more Wikipedia pages. FEVER gives a claim to label SUPPORTS, REFUTES or NOT ENOUGH INFO. The paper runs both in a question-only setup: the model sees the question or claim and none of the supporting paragraphs the datasets provide, so it must use its own knowledge or look things up. The authors gave it three actions:

• search[entity] returns the first 5 sentences of the Wikipedia page with that title, or, if there is none, up to 5 similar titles from Wikipedia's search.
• lookup[string] returns the next sentence on the current page that contains the string, like Ctrl+F. Calling it again returns the next match.
• finish[answer] ends the episode with that answer, graded by exact match after lowercasing and stripping punctuation and articles.

Anything else the model writes comes back as the observation "Invalid action: ...", which it sees on its next call.

The paper calls this API "significantly weaker than state-of-the-art lexical or neural retrievers", on purpose. Retrieval-augmented generation embeds the question and fetches the nearest passages; here the model has to name the exact page it wants and say in writing why, which is what the experiment is about.

The baselines are built by deleting lines from the same hand-written episodes (six for HotpotQA, three for FEVER; the paper found more examples did not help):

• Standard keeps only questions and answers.
• CoT keeps the thoughts and drops actions and observations, giving one block of reasoning per question.
• CoT-SC (self-consistency) samples 21 chain-of-thought answers at temperature 0.7 and returns the most common final answer.
• Act keeps actions and observations and drops the thoughts.

In the released prompt file the six HotpotQA examples are 938 words in ReAct format, 515 as Act, 308 as CoT and 114 as Standard. A ReAct prompt is about eight times longer than a Standard prompt before the model writes anything.

Results on HotpotQA and FEVER

Table 1 of the paper, PaLM-540B with prompting only:

ReAct beats Act on both tasks (27.4 against 25.7, and 60.9 against 58.9), the comparison that isolates the thoughts. Against chain-of-thought the picture splits. On FEVER ReAct wins, 60.9 to 56.3; the paper points out that a SUPPORTS claim and a REFUTES claim can differ by one detail, such as a year, so looking it up matters. On HotpotQA ReAct loses to CoT (27.4 to 29.4) and to plain Standard prompting (28.7). Every prompting number is far below the supervised systems trained on each dataset, 67.5 and 89.5.

To see why, the authors sampled 50 correct and 50 incorrect HotpotQA trajectories from each of ReAct and CoT (200 in total) and labeled what happened in each by hand. Figure 2 shows their Table 2.

Hallucination is 56% of chain-of-thought's failures and 0% of ReAct's. It also shows up among the correct answers: 14% of CoT's right answers were reached through a hallucinated fact or trace, against 6% for ReAct.

ReAct pays for that grounding in two ways. Reasoning errors make up 47% of its failures against 16% for CoT. Forcing every step into the thought, action, observation pattern makes it harder to write a reasoning step that does not fit it, and one error pattern is specific to ReAct: the model repeats its previous thought and action and never gets out of the loop. The paper suspects greedy decoding and suggests beam search might help, without testing it. The second cost is search: 23% of ReAct's failures came from searches that returned nothing or nothing useful, after which the model had a hard time reformulating.

The remaining failures, 29% for ReAct and 28% for CoT, are label ambiguity: the prediction was right but did not match the label string, so exact match scored it wrong.

Falling back between ReAct and CoT-SC

Table 2 suggests that CoT gets the structure of the reasoning right more often and ReAct gets the facts right more often. The paper combines them with two fallback rules, one for each direction.

ReAct → CoT-SC. Run ReAct. If it has not called finish within 7 steps on HotpotQA or 5 on FEVER, answer with CoT-SC instead. The limits come from the data: among ReAct trajectories with correct answers, only 0.84% on HotpotQA used 7 steps and 1.33% on FEVER used 5.

CoT-SC → ReAct. Sample $n$ chain-of-thought answers $y_1, \dots, y_n$. If the most common answer has fewer than $n/2$ votes, treat the model's internal knowledge as unsure and run ReAct instead:

With $n = 21$, an answer that gets 11 or more votes is returned as is; if the top answer gets 10 or fewer, say 8 votes for one answer, 6 for another and 7 spread over others, the question goes to ReAct. Self-consistency agreement acts as a confidence signal, and only the low-confidence questions pay for a ReAct run.

Figure 3 is the paper's Figure 2, digitized: accuracy against the number of chain-of-thought samples. Drag $n$ and compare each combination with the dotted amber line, CoT-SC's accuracy at all 21 samples.

The paper reports that both combinations reach 21-sample CoT-SC accuracy with 3 to 5 samples; on the FEVER plot CoT-SC → ReAct gets there at about 7. On HotpotQA at $n = 1$, ReAct → CoT-SC already scores 32.2 against ReAct's 27.4: the episodes that ran out of steps get one chain-of-thought sample, and that alone is worth almost 5 points. The best rule differs by task. On HotpotQA it is ReAct → CoT-SC (35.1 at 21 samples); on FEVER, where ReAct alone already matches 21-sample CoT-SC, handing the uncertain questions to ReAct gives the best result, 64.6.

Finetuning smaller models on ReAct trajectories

Prompting only works when the model is large enough to pick up the format from a few examples. To test smaller models the authors used a bootstrapping scheme in the style of STaR: run ReAct prompting with PaLM-540B on training questions, keep 3,000 trajectories that reached the correct answer, and finetune PaLM-8B and PaLM-62B to produce a whole trajectory (thoughts, actions and observations) given the question. Each baseline got the same treatment with 3,000 correct trajectories in its own format. Training used batch size 64; ReAct and Act ran for 4,000 steps, Standard and CoT for 2,000 steps on 8B and 1,000 on 62B, because those two got worse soon after finetuning started. The paper does not say whether the observation tokens were masked out of the loss; at test time they come from the API, as in the prompted version.

Prompted, ReAct is the worst of the four methods at 8B (about 5) and at 62B (about 17), because a small model cannot learn both reasoning and acting from six examples. Finetuned, it is the best at both sizes: about 25 at 8B, higher than every 62B prompting method, and about 33 at 62B, higher than every 540B prompting method in this figure (though below the 35.1 of the 540B fallback combination). The authors' explanation is that finetuning on Standard or CoT trajectories teaches the model to memorize facts, some of them hallucinated, while finetuning on Act or ReAct trajectories teaches it how to find facts on Wikipedia, which transfers to new questions.

Sparse thoughts in ALFWorld and WebShop

The second half of the paper moves to tasks with long action sequences and a reward only at the end. ALFWorld is a text game aligned with the ALFRED household-robot benchmark. The agent gets a goal such as "put some spraybottle on toilet" and issues commands like go to cabinet 2, open cabinet 2 and take spraybottle 2 from cabinet 2. There are six task types; a game can have more than 50 locations and take an expert more than 50 steps. Evaluation uses 134 unseen games.

Here thoughts are sparse. The model writes one as a command starting with think:, and the harness answers "OK." without changing the game state. An example episode from the authors' ALFWorld prompt:

Your task is to: put some spraybottle on toilet.
> think: To solve the task, I need to find and take a sparybottle, then put
  it on toilet.
OK.
> think: First I need to find a spraybottle. A spraybottle is more likely to
  appear in cabinet (1-4), countertop (1), toilet (1), sinkbasin (1-2),
  garbagecan (1). I can check one by one, starting with cabinet 1.
OK.
> go to cabinet 1
On the cabinet 1, you see a cloth 1, a soapbar 1, a soapbottle 1.
> go to cabinet 2
The cabinet 2 is closed.
> open cabinet 2
You open the cabinet 2. The cabinet 2 is open. In it, you see a candle 1,
and a spraybottle 2.
> think: Now I find a spraybottle (2). Next, I need to take it.
> take spraybottle 2 from cabinet 2
You pick up the spraybottle 2 from the cabinet 2.
> think: Now I take a spraybottle (2). Next, I need to put it in/on toilet 1.
OK.
> go to toilet 1
On the toilet 1, you see a soapbottle 2.
> put spraybottle 2 in/on toilet 1
You put the spraybottle 2 in/on the toilet 1.

The second thought uses commonsense the game never provides: where a spray bottle is likely to be. The later thoughts track which subgoal is done and which comes next. (The typo "sparybottle" is in the original prompt.) For each task type the authors annotated three such episodes; a prompt contains two of them, and the six ordered pairs give six prompt sets. Act prompts are the same episodes with the thoughts deleted. The baseline, BUTLER, is an imitation-learning agent trained on 10⁵ expert trajectories per task type. In the released code an episode is capped at 49 model actions, and thoughts count against that cap.

Figure 5 is Table 3 of the paper, success rates per task type. Toggle between the best of the six prompt sets and their average.

ReAct's best prompt set solves 71% of games, against 45% for Act's best and 37% for BUTLER's best. Its worst prompt set scores 48%, still above both. Across the six controlled pairs (the same examples with and without thoughts) ReAct's gain over Act ranges from 33% to 90% relative and averages 62%. The paper reports that without thoughts, Act fails to break goals into subgoals or loses track of where it is; Figure 1 of the paper shows Act trying "take peppershaker 1 from sinkbasin 1" twice and getting "Nothing happens." both times, on a sink basin that holds no pepper shaker.

The ReAct-IM rows test which kind of thought matters. Inner Monologue (Huang et al., 2022) fed a robot's language model with descriptions of the scene and of whether the last action succeeded. ReAct-IM re-annotates the same episodes with thoughts of that kind, which only restate what the agent already knows: the goal broken into subgoals and the current subgoal. They never say when a subgoal is finished, what comes next, or where an object is likely to be. Its best prompt set scores 53% against ReAct's 71%, and it trails on five of the six task types (Clean is the exception, 68 against 58).

Because the policy is driven by its own written plan, a person can correct it by editing that plan. In the paper's Figure 5, a ReAct episode on "put two keychain in safe" fails after a hallucinated thought sends the agent back to the drawer where it found the first keychain. A human deletes the hallucinated sentence from that thought and adds hints to a later one, two edits in total, and the episode succeeds. With Act or an RL policy there is no equivalent text to edit.

WebShop is a simulated shopping site with 1.18 million real Amazon products and 12,000 human-written instructions such as "I am looking for a nightstand with drawers. It should have a nickel finish, and priced lower than $140". The agent searches, clicks products and options, and buys. Score is the fraction of the requested attributes the purchased product has, averaged over 500 test instructions; success rate is the fraction of episodes where it has all of them. ReAct uses a single example episode.

One-shot Act already matches the trained IL and IL+RL agents. Adding thoughts lifts the success rate to 40.0, 10.9 points over the best trained agent; the abstract rounds this to 10%. Like the abstract's ALFWorld gain (34 points: 71% against BUTLER's 37%), it is in percentage points, not relative. The paper's example of a useful thought: "For 'space-saving ottoman bench for living room', the item has options '39x18x18inch' and 'blue' and seems good to buy." Human experts still lead by 19.6 points of success rate; the paper notes they explore more products and rewrite their queries more often.

What the paper does not show

Prompt-set selection. The 71% ALFWorld result and the 34-point gain in the abstract use the best of six prompt sets, picked by their score on the same 134 test games. The average over the six is 57%. BUTLER's 37% is likewise its best of eight.

Prompting alone does not beat chain-of-thought on HotpotQA. The prompting gains on question answering come from the fallback combinations, which need up to 21 chain-of-thought samples per question as well as ReAct runs.

Why interleaving helps is not established. The comparisons with Act and ReAct-IM show that the right kind of thought helps on ALFWorld. In 2024, Verma, Bhambri and Kambhampati re-ran ALFWorld with GPT-3.5, GPT-4 and Claude-Opus and found that putting all the reasoning in a single thought at the start did as well or better for the GPT models, that placeholder text in place of task-specific thoughts gave comparable results, and that using example episodes from a different task type dropped GPT-3.5-Instruct from 52% to 9% success. Their reading is that success depends mostly on how closely the examples match the test task. The original paper does not run those ablations.

Context length. The conclusion notes that harder tasks with larger action spaces need more demonstrations than fit in the input length of in-context learning. Every model call re-reads the prefix and the whole episode so far.

Reproducibility. PaLM was not publicly available. The authors released prompts and GPT-3 code, and Table 5 reports text-davinci-002 with ReAct at 30.8 HotpotQA exact match (500 validation questions) and 78.4% ALFWorld success, both above PaLM-540B. The paper suggests the gap comes from GPT-3's instruction tuning, without testing it.

Safety. The ethics statement notes that connecting a language model to real actions can mean looking up private information or taking harmful actions. The experiments limit actions to reading Wikipedia and to the WebShop simulator, where nothing is actually bought.