Build in Public

Day 2 of Ask TanStack Query: I went and got the books and the quiz

Today I downloaded the official TanStack Query docs and a pile of real questions. The docs are the library. The questions are the quiz.

By Brian Shimkus Oct 3, 2026 5 min read

The short version

Yesterday a current AI model answered five questions about TanStack Query and got two of them wrong. The answers looked like real code. Nothing in them said they were out of date. That is why this project will look answers up in the current instructions instead of trusting the model's memory.

Today I downloaded those instructions, and I downloaded a pile of real questions to grade the project with later.

The books: 270 pages of official docs

The instructions are the official TanStack Query docs. They live on GitHub as ordinary text files, 270 of them. A small program reads each file, keeps the title, and saves the page. That pile is the library.

It covers the React docs plus the shared reference pages, such as QueryClient and QueryCache, that React users read too. Each saved page keeps its title, its text, and the web address of the page on the TanStack site. That address matters later, because every answer the assistant gives will point back to a real page.

The quiz: 962 answered questions

The questions are from GitHub Discussions on the same project. I kept only the ones where a maintainer had marked an answer as correct: 962 of them. The question is the prompt. The marked answer is the answer key.

Downloading them meant asking the GitHub API for the project's Q&A category, one page at a time, keeping each question, its link, and the marked answer. Using a real person's answer as the key is better than inventing questions myself, because these are the things developers actually got stuck on.

What I noticed when I read a handful

I will not remember the details, and I do not need to. Two things showed up over and over.

1. A lot of the marked answers are old

They still describe an earlier version of the tool. "Marked correct" means a maintainer said it was right at the time. It does not mean it is still right.

2. A lot of them point at a docs page

That link is the useful part. Later, the quiz can check whether the assistant found the same page.

What I set up on Day 2

Day 2 was five small commits.

Three empty packages. app, ingest, and evals, so each job has a home before it has code.
The docs loader. ingest/load_docs.py reads the Markdown files and saves one JSON file. Some pages borrow their text from another file, so the loader follows that reference and skips any page that ends up empty.
A web library. I added httpx so the program can talk to GitHub.
The question downloader. ingest/fetch_discussions.py finds the Q&A category, asks for answered discussions only, and saves them to a second JSON file.
The journal. I wrote down what I did, what broke, and what I learned.

The downloaded data stays out of the repository. The scripts re-create it, which keeps the repo small and means anyone can rebuild the same library and quiz.

Two raw materials, and they do not fully agree

So the project has its two raw materials now: a current manual, and a stack of real questions that do not all agree with it.

That mismatch is useful. It means the quiz cannot treat every marked answer as the final word. It also means the assistant has to prefer the current docs, which is the whole point of the project.

Frequently asked questions

Where do the TanStack Query docs come from?

From the official TanStack Query repository on GitHub, where the docs are plain Markdown files. I read the React docs and the shared reference pages: 270 pages in total.

Where do the test questions come from?

From the project's GitHub Discussions Q&A. I kept only the discussions where a maintainer marked an answer as correct: 962 of them.

Does a marked answer mean it is still correct?

No. It means a maintainer said it was right at the time. Many of these answers describe an earlier version of the library.

Why keep only the discussions with a marked answer?

Because they come with an answer key. The question is the prompt, and the answer a maintainer marked as correct is what the quiz compares against.

Next up

Next, the manual gets cut into pieces the program can search. I will write about what I learn as I go, including the mistakes.

Follow on GitHub Read the Case Study