Methodology: how the index is measured

What this measures

Each month we ask language models to set up a new app, one task at a time ("Set up user sign-up and login"), and record the one product or library each model names. We count how often each tool is named. Model pick, not a product recommendation. It shows what language models name when asked without any constraints. It is not a statement about quality, security, price or market share.

The models answer without a code repository and without tools (API models through an API, chat models in their own app). That measures what a model prefers when nothing in the project constrains it. It does not measure what a coding agent does inside a real repository, where existing files, lock files and instructions change the outcome. See the limitations below.

The prompt set

The prompt set is versioned (current version 2026.1) and published in full as prompts.json. It has 24 categories and 4 contexts, so 96 prompts. Every prompt has the same form: one sentence that sets the context, then "Set up <need>." No product is named in any prompt. The system prompt is the same for every call:

You are a coding agent working in an empty project. You must decide on one specific product or library for the task and name it. Do not ask questions and do not explain at length. Answer only with JSON that follows the schema.

Example for the category "Authentication" in the first context: "I'm starting a new Next.js (TypeScript) app. Set up user sign-up and login (authentication)."

The 24 categories are: Authentication, Database, ORM and query layer, Hosting, File storage, Background jobs, Search, Realtime, Caching, Scheduled jobs, Payments, Transactional email, Newsletter and marketing email, Error monitoring, Web analytics, Product analytics, Feature flags, UI components, Forms, CMS, Internationalisation, AI SDK, Rate limiting and bot protection, Cookie consent. Category pages exist for 10 of them; the others are measured and included in the downloads.

Answer format

Each answer must be a JSON object with four fields: primary (the one product or library chosen), type (third_party_service, oss_library, platform_builtin, custom_diy or none), alternatives (up to three others considered) and reason (one short sentence). We request this with the provider's strict JSON-schema mode and require providers that support it. If no provider supports structured output for a model, we make one fallback call that asks for plain JSON in the prompt; fallback calls are counted per model. The fallback instruction is:

Reply with only a JSON object: {"primary": string, "type": "third_party_service"|"oss_library"|"platform_builtin"|"custom_diy"|"none", "alternatives": string[], "reason": string}.

We keep only the tool name, the claimed type and the alternatives. We do not publish the models' free text. Custom or do-it-yourself answers and "none" are counted and shown like any tool, not hidden.

Models

The planned full set has 13 paid models plus open models and chat models; the table lists every slug we use or plan to use, and the section "What this issue contains" says which ones were run. API models are called through OpenRouter with fixed model slugs (never an alias such as "latest"). Provider routing is set to refuse providers that retain prompts for training (data_collection: deny). Temperature and other sampling settings are left at each vendor's default. Reasoning models may use more tokens; output is capped at 4,000 tokens per call. A model is added or replaced only at the start of a new month and the change is recorded in the changelog, so the trend is not broken mid-month. The chat models were asked by hand in their own apps, one run per prompt, and are logged with the same fields as the API calls.

Repeats

Each prompt is run 3 times per model. A full monthly run (planned) is 13 models x 24 categories x 4 contexts x 3 repeats = 3744 calls, which gives 156 picks per category. One model in one category has only 12 picks, so cells in the model matrix are indicative only.

What issue 2026-10 contains

This is what was actually run for October 2026, as opposed to the planned full set above. Measured 2026-10-19 with prompt set 2026.1. 957 calls, 900 parsed picks (94 percent), 57 calls without a usable answer. 3 models were called through OpenRouter, 5 repeats per prompt and model; 6 chat models were asked by hand, one run per prompt. The free open-model endpoints are rate-limited, so some of their calls failed; failed calls are excluded from every share and counted in the table below. The picks per category range from 24 to 70, far below the planned full run, so read every share with its n and its interval.

Reading the numbers

• Share = picks of a tool divided by all parsed picks in the category (all models, all contexts). Every number is shown with n.
• The interval is a 95 percent Wilson score interval for that share. It describes sampling noise in this prompt set, not uncertainty about the world. It treats every pick as an independent draw, but the repeats of one prompt and the picks of one model are not independent, so the real uncertainty is larger than the interval suggests.
• A category shows "No clear favourite" when the most named tool stays on top in fewer than 80 percent of 500 bootstrap resamples of the picks. The resampling draws single picks and does not account for the runs of one model being correlated, so read it as a rough stability check, not a significance test.
• Cells in the model matrix with fewer than 12 picks are shown greyed out.
• The EU column is the share within the EU context only.
• The change against the previous month appears from the second issue. Until then the column is hidden and the page says "first data point".

Normalising names

Models write the same product in different ways ("Clerk", "clerk.dev", "Clerk Auth"). An alias table maps each spelling to one tool, taking the category into account: "Supabase" is Supabase Auth in the authentication category and Supabase in the database category. A compound answer ("Neon with Drizzle") counts as its first recognised part. Names the table does not know are kept under their own name, counted, and reviewed by hand each month; the 20 most frequent unknown names are added to the table or classified. No second model judges the choice. The share of answers that could be parsed is reported per model.

When a page exists

To avoid thin pages, some pages only exist once the data is large enough:

• Tool pages need at least 30 picks, 3 different models and 2 different contexts.
• Comparison pages need at least 15 picks for each of the two tools in the same category.
• Category pages are listed in search and in the sitemap from 24 picks; before that they are marked noindex.

Limitations

• A model pick is a training preference, not a recommendation. Models also name products because they were common in their training data, which is older than the product's current state.
• No repository and no tools: a real coding agent reading a project with an existing database or lock file will often pick differently.
• The prompts are our wording. A different wording may shift the result; this is why the prompts are public.
• With 24 to 70 picks per category, shares differ by several percentage points by chance alone. Use the interval.
• Categories overlap (a platform can be named under hosting and database). We count each answer in the category it was asked in.
• The models are a convenience sample of widely used models. They are not weighted by how many people use them.
• We do not assess price, quality, security, GDPR compliance or support of any product.

Agent calibration

Real coding agents (command-line agents working in an empty project) may choose differently from the API. We plan a small quarterly calibration that runs one agent on one context and reports a single agreement figure: how often the agent's pick equals the API pick for the same category. It has not been run yet. When it has, the figure appears here with its date.

Independence

No tool maker pays for a place, a position or a mention, and there are no affiliate links. Vendors do not see results before publication. Product names appear as plain text; we use no logos.

Corrections for vendors and readers

If a name is mapped to the wrong product, a tool sits in the wrong category, or a fact in a comparison table is out of date, tell us through the contact form or at hello@agentstackpicks.com. Name the page and, for facts, the official source. We answer within five working days. Changes to the alias table, the categories or a fact are made in the next data build and listed in the changelog below with their date. We do not remove a tool because a vendor dislikes its numbers: the numbers are counts of what the models named.

Changelog

2026-10-19: Prompt set 2026.1 published: 24 categories, 4 contexts. First measurement (issue 2026-10) with open models (Gemma 4 31B, Nemotron 3 Super 120B, Laguna S 2.1) and chat models (Claude Opus 5.5, Claude Sonnet 5.5, Gemini 3.1 Pro, Gemini 3.6 Flash, GPT-5.6, Kimi K3); more models and repeats follow.

Download and cite

Prompts, results and raw picks are open under CC BY 4.0 on the data page, with a suggested citation.