The row benchmark and the regression gate

Counts catch an update that does too much; they cannot tell you how long a thousand rows take to paint, or how many bytes they keep alive. For those the repository has a real-browser benchmark in tools/bench. It builds three apps with identical markup (ShadowJS, Solid 1.9 and React 19), opens each in a fresh page of a full headless Chromium, and measures eight operations: create 1,000 rows, create 10,000 rows, update every 10th row, select a row, swap two rows, remove a row, clear, and append 1,000 rows. Each operation runs 3 warm-up iterations (discarded) and 15 measured iterations in each of 2 rounds, with the framework order rotated between rounds, so every cell has 30 samples.

npm --prefix tools/bench install   # once: the benchmark has its own install, outside the root workspaces
npx --prefix tools/bench playwright-core install chromium   # once: the full Chromium build
npm run bench                      # builds the ShadowJS packages, then runs the benchmark

The raw result (every sample, memory samples, bundle files, machine and browser facts) is written to tools/bench/results/bench-<stamp>.json, and the Markdown tables are printed. node report.ts <file> in tools/bench prints the same tables from a saved file. The run waits until no vitest or Playwright browser process is running, because a busy machine ruins timings.

How to read the numbers#

• Paint is the time from the click to the second animation frame. It is quantized to frames, and two empty frames take about 32 ms on the reference machine (the frame floor), so any operation near 32 ms is below what this measure can resolve. Do not read a difference between two frameworks inside the floor.
• Click is the time inside the synchronous click handler only. It shows work the framework does before it returns control. React defers rendering out of the handler, so its click column is not comparable with the other two; compare it only between runs of the same app.
• Heap per 1,000 rows is the JavaScript heap with 1,000 rows minus the heap with none, both after a full garbage collection, over 5 fresh pages.
• Bundle is the gzip level 9 size of the production JavaScript and CSS of each app.
• Medians and 90th percentiles use linear interpolation. Numbers from different machines are not comparable; only compare runs made on the same machine, with the same options.

One recorded run (Apple M5 Pro, Chromium 153, 2026-10-10) found the 1,000-row operations, update, select, swap, remove and clear all within the floor for the three frameworks. Create 10,000 rows took 247 ms median for ShadowJS, 217 ms for Solid and 338 ms for React. After a later allocation pass the heap delta for 1,000 rows fell from 3.79 MiB to 2.49 MiB (Solid: 2.11 MiB) and the retained bytes per bench row from 3,515 to 2,164 (Solid: 1,798). The ShadowJS production JavaScript measured 11.8 KiB gzip, against 7.5 KiB for Solid and 67.2 KiB for React. These are one machine and one run each; run the benchmark yourself before you quote a figure.

The regression gate#

tools/bench/compare.ts compares a new result with a stored baseline, tools/bench/baseline.json:

cd tools/bench
node compare.ts baseline.json results/bench-<stamp>.json

For the ShadowJS app it compares, per operation, the paint time and the synchronous click time, plus the heap per 1,000 rows and the gzip size of the bundle (JavaScript plus CSS), and prints a Markdown table. For the sampled metrics it computes the 95% bootstrap interval of the difference in medians (4,000 resamples from a seeded generator, so the same two files always give the same table). A metric fails only when both hold:

1. the whole interval is above zero (the new run is worse beyond the noise), and
2. the median grew by more than the minimum relative change, 5% by default (--threshold=0.05).

The bundle size is exact, so it is judged by the relative change alone (--bundle-threshold, which defaults to --threshold). Improvements are reported but never fail. The exit code is 0 for no regression, 1 for at least one regression, and 2 for bad arguments or a file that is not a usable result. --output=file.md also writes the table to a file. The table ends with notes when the baseline came from a different machine, different run options or a different browser, because such a comparison means little.

Recording the baseline. The repository does not ship a baseline, because a baseline is a measurement from one machine class and only a real run can produce it. To record one:

1. Check out the commit you want to protect, with a clean working tree.
2. On the machine that will run the gate, run npm run bench. For the nightly workflow (.github/workflows/bench.yml), download the bench-result-<run id> artifact of a workflow run instead, so the baseline comes from the same runner class.
3. Validate and copy it: node tools/bench/update-baseline.ts <result.json>. The script checks the shape of the file, refuses a result recorded with uncommitted changes unless you pass --allow-dirty, and copies it byte for byte to tools/bench/baseline.json.
4. Commit baseline.json in a reviewed change that says why the numbers moved.

The nightly workflow runs the benchmark, compares with baseline.json when the file exists, uploads the result JSON as an artifact, and writes the table to the job summary. It fails the job when the comparison reports a regression. Without a baseline it only records the run.

Measure retained memory#

Use the browser’s Memory panel to compare the same page before mounting a list, after mounting it and after unmounting it. Trigger garbage collection before each snapshot. After unmounting, component owners, event handlers and row data should no longer be retained by the application. Inspect retaining paths when they are.

Size gates#

Bundle size is gated twice. npm run gate:size builds the scenarios in tools/gates and fails when one grows past its entry in tools/gates/size-budget.json; budgets only ratchet down. npm run gate:treeshake measures every public export and fails when an export grows by more than 3% or 40 bytes, or retains code outside its own module's imports. Bundle size explains both.

A submission for the public benchmark#

benchmarks/js-framework-benchmark holds a keyed ShadowJS implementation of the standard rows app, laid out for the community js-framework-benchmark. It reuses the component logic of the app measured here (keyed For, useStore, useSelector) and has its own README on how to submit it. That project measures with its own harness, so its numbers will not match the ones from tools/bench.