Why it matters

ToT beats CoT on tasks with multiple viable approaches. Puzzles, planning problems, creative writing. Understanding when to apply it opens up better solutions to hard problems.

Advertisement

The architecture

State: current reasoning trace. Actions: generate K possible next thoughts. Evaluate each: model rates them or heuristic ranks them. Expand: continue with top-N.

Search: BFS or DFS through the tree. Backtrack from dead ends.

Tree of Thoughts algorithmGenerate K next thoughtsat each stateEvaluaterate promiseExpand top-Nsearch treeCost is K× single-path but produces higher quality on planning and exploration tasks
ToT search structure.
Advertisement

How it works end to end

Evaluation: LLM-as-judge (ask model to rate thoughts), heuristic score, or task-specific metric.

Search strategy: BFS explores broad; DFS explores deep. Depth-first works better when solutions exist along one path.

Cost: multiplies inference cost by branching factor times depth. Expensive but produces breakthrough quality on hard tasks.