Internal API benchmark — runbook

This page explains how to run the internal API benchmark script and produce a report. For what the script measures and why, read Internal API benchmark — overview.

The script lives at jodapp-web/scripts/internal-benchmark/jod-latency-benchmark.sh. The command examples below assume you run them from the root of the jodapp-web checkout, so the script path is scripts/internal-benchmark/jod-latency-benchmark.sh.

A server-side run measures the Rails API over both hosts: the private host the server render calls, and the public host the browser calls. Measuring both from one machine in one run is the only honest comparison of the two paths, because nothing but the host differs. The server-side report opens its Results section with that comparison. A run with API_INTERNAL_JODAPP_URL set on the instance records the page timings over the private path.

Run it with a session cookie, or the answer is wrong

The script measures the Rails API with two endpoints. They are not interchangeable.

Endpoint	What it does	What it tells you
/up	The health route. Touches no database.	The network floor. The smallest cost the path can have. No page ever calls it.
GET /identities/user_sessions/current	Reads the session row named by the signed cookie, then asks two services whether the person may enter the talent area and the employer area.	One real signed-in fetch. What a page actually pays. It is also the endpoint the coming session middleware calls.

The second one needs JOD_SESSION_COOKIE. Without it, the script skips that endpoint and says so loudly. The report can then only show network time. It cannot answer the loader() versus clientLoader() question.

So supply the cookie on all three runs. See "Signed-in pages" below.

Check that your environment serves the endpoint

Environments do not always run the same build. The script checks before it starts, with one request and no cookie. On a 404 it skips the real-fetch measurement instead of recording 404 timings.

Answer	Meaning
401	The route is there. That run can measure a real fetch.
404	That Rails API does not serve it yet. That run measures the network floor only.

You do not need to check by hand. Run the command and read the first few lines of output. The script prints one of these:

  session endpoint answers: http://api.internal.jodapp.dev/identities/user_sessions/current -> 401 (deployed)

  session endpoint MISSING on qa: http://api.internal.jodapp.dev/identities/user_sessions/current -> 404
  THIS RUN CANNOT SAY WHAT A DATA FETCH COSTS

If you see the second one, that environment has not picked up the deploy yet. Wait for it and run again.

A heavier endpoint to compare against

The session endpoint is small on purpose. For a heavier read, run again with --session-endpoint /identities/users/current. That one loads the user with about ten associated tables. It shows what a fatter fetch costs on the same two paths.

Run these three commands

Run all three. One alone cannot answer the question.

#	What it measures	Where you run it
1	QA web server to QA Rails API, and QA page render time	on jodapp.qa.web
2	prod web server to prod Rails API, and prod page render time	on jodapp.prod.web
3	browser to prod Rails API, and prod page arrival time	on your laptop

Get the session cookie first. "Signed-in pages" below shows how. All three runs need it.

1. QA, server side

Copy the script over, then sign in and run it there:

scp -i ~/.ssh/jodapp.qa.deploy scripts/internal-benchmark/jod-latency-benchmark.sh ubuntu@10.1.140.73:/tmp/

ssh -i ~/.ssh/jodapp.qa.deploy ubuntu@10.1.140.73

On the instance:

read -r -s JOD_SESSION_COOKIE && export JOD_SESSION_COOKIE

bash /tmp/jod-latency-benchmark.sh --mode server --env qa

Leave --out unset. The script writes its own timestamped filename under /tmp/results/. A fixed name, such as --out /tmp/qa-server.md, overwrites the previous run's report every time you run it. That mistake once cost the 2026-09 analysis several runs of raw data: Internal API benchmark — 2026-09 analysis, "Some numbers on this page have no surviving report."

Back on your laptop, list the reports on the instance and note the newest one:

ssh -i ~/.ssh/jodapp.qa.deploy ubuntu@10.1.140.73 'ls -t /tmp/results/'

Copy that file down, then move it into the docs repo. The report belongs in benchmark-results/, next to the reports already there, named <year>-<month>-benchmark-qa-server.md. The commands below assume the docs repo is checked out as a sibling of jodapp-web, which is the usual layout:

scp -i ~/.ssh/jodapp.qa.deploy 'ubuntu@10.1.140.73:/tmp/results/<the file the ls command listed>' ./qa-server.md

mv qa-server.md ../docs/docs/50-59-frontend/41-board-web/internal-api-benchmark/benchmark-results/<year>-<month>-benchmark-qa-server.md

ssh -i ~/.ssh/jodapp.qa.deploy ubuntu@10.1.140.73 'rm -rf /tmp/jod-latency-benchmark.sh /tmp/results'

Sends 276 requests once QA serves the session endpoint, or 184 while it still answers 404. Without a session cookie, it sends 161. It usually finishes in one to two minutes.

2. prod, server side

The same steps, with the prod key, the prod IP and --env prod:

scp -i ~/.ssh/jodapp.prod.deploy scripts/internal-benchmark/jod-latency-benchmark.sh ubuntu@10.0.143.187:/tmp/

ssh -i ~/.ssh/jodapp.prod.deploy ubuntu@10.0.143.187

On the instance:

read -r -s JOD_SESSION_COOKIE && export JOD_SESSION_COOKIE

bash /tmp/jod-latency-benchmark.sh --mode server --env prod

Leave --out unset here too, for the same reason.

Back on your laptop, list the reports and note the newest one:

ssh -i ~/.ssh/jodapp.prod.deploy ubuntu@10.0.143.187 'ls -t /tmp/results/'

Copy it down, then move it into the docs repo the same way. Name it <year>-<month>-benchmark-prod-server.md — the 2026-09 run in this folder used exactly that pattern, 2026-09-benchmark-prod-server.md:

scp -i ~/.ssh/jodapp.prod.deploy 'ubuntu@10.0.143.187:/tmp/results/<the file the ls command listed>' ./prod-server.md

mv prod-server.md ../docs/docs/50-59-frontend/41-board-web/internal-api-benchmark/benchmark-results/<year>-<month>-benchmark-prod-server.md

ssh -i ~/.ssh/jodapp.prod.deploy ubuntu@10.0.143.187 'rm -rf /tmp/jod-latency-benchmark.sh /tmp/results'

Sends 276 requests, or 161 without a session cookie. It usually finishes in one to two minutes.

3. prod, client side

This one runs on your laptop, so it needs no copying:

scripts/internal-benchmark/jod-latency-benchmark.sh --mode client --env prod --network "describe your city and connection here"

Change the --network text to describe where you actually are. The report prints it.

This matters. Hotel wifi in Berlin and home fibre in Singapore give very different numbers. A reader has to know which one produced the report.

Leave --out unset here too. The script writes its own timestamped filename under scripts/internal-benchmark/results/, so running this more than once never overwrites an earlier run.

Sends 253 requests. It usually takes three to eight minutes, because every request crosses the internet. List the reports and note the newest one, then move it into the docs repo, named <year>-<month>-benchmark-prod-client.md:

ls -t scripts/internal-benchmark/results/

mv 'scripts/internal-benchmark/results/<the file the ls command listed>' ../docs/docs/50-59-frontend/41-board-web/internal-api-benchmark/benchmark-results/<year>-<month>-benchmark-prod-client.md

If you run this more than once in one sitting, as the 2026-09 runs did, add a short time suffix so each report keeps its own file instead of one overwriting the sidebar entry of another — for example 2026-09-benchmark-prod-client-1446.md.

A quicker server-side run, without the cookie

This one-line form sends the script over stdin and brings the report back. It cannot carry the cookie, because stdin already carries the script. So it measures network time only.

ssh -i ~/.ssh/jodapp.qa.deploy ubuntu@10.1.140.73 'bash -s -- --mode server --env qa --out -' < scripts/internal-benchmark/jod-latency-benchmark.sh > qa-server.md

Use it for a quick re-check after a deploy. Do not use it for the loader() versus clientLoader() decision.

Before you run 1 and 2: check your ssh setup

The two server-side commands assume three things.

Assumption	If it is wrong
You reach the private IPs 10.1.140.73 and 10.0.143.187 from your laptop	You need the VPN or a bastion first. The private IPs do not route from the open internet.
The ssh user is ubuntu	Change ubuntu@ to your user.
The keys are ~/.ssh/jodapp.qa.deploy and ~/.ssh/jodapp.prod.deploy	Change the -i path. These are the keys in jodapp-web/deploy/deploy.qa.yml and deploy/deploy.yml.

If you already have an ssh alias, use it and drop the -i flag:

ssh jodapp.qa.web 'bash -s -- --mode server --env qa --out -' < scripts/internal-benchmark/jod-latency-benchmark.sh > qa-server.md

Signed-in pages

Without a session cookie the script skips two things and says so in the report:

• the real signed-in API call, which is the number the whole decision rests on
• /employers/dashboard, the real employer journey

Do not skip this section. A run without a cookie measures network time only.

Step 1: copy the cookie out of Chrome

1. Sign in to https://jodapp.com as an employer.
2. Open DevTools. Press F12, or Command-Option-I.
3. Go to the Network tab.
4. Reload the page.
5. Click the first request in the list. It is the HTML document, named dashboard or similar.
6. Open Headers, then Request Headers.
7. Find the line that starts with cookie:.
8. Copy the whole value after cookie:. It looks like jodapp_session_id=...; CSRF_TOKEN=abc....

Use the Network tab, not the DevTools console. The jodapp_session_id cookie is HttpOnly, so document.cookie in the console does not show it.

Step 2: put it in a variable without typing it on the command line

read -r -s JOD_SESSION_COOKIE && export JOD_SESSION_COOKIE

Press Enter, paste the cookie, then press Enter again. Nothing appears on screen. That is correct.

Why this way and not export JOD_SESSION_COOKIE=eyJ...:

• the value never enters your shell history
• the value never appears on screen, so it stays out of a screen recording

Step 3: run the three commands above

On a server run, do the read -r -s step again on the instance. The variable does not travel over ssh.

The cookie value never reaches the terminal output or the report. The script also searches the finished report for the cookie value. If it finds the value, it saves nothing and stops with an error.

What you cannot measure, and why

You want	Can you?	Why
QA pages from your laptop	No	jodapp.dev sits behind Cloudflare Access. Every request from outside gets a 302 to jodapp.cloudflareaccess.com. You would measure the redirect. Use run 1 instead.
QA public API from your laptop	Yes	api.jodapp.dev is public. Run --mode client --env qa --only api.
prod pages from your laptop	Yes	jodapp.com is not behind Access.
A real signed-in fetch on QA	Only once QA serves the endpoint	The script checks and tells you. While QA answers 404, that run measures the network floor only.
The private API from your laptop	No	api.internal.jodapp.com only resolves inside the VPC. Use run 2 instead.

The script refuses the combinations that cannot work, and explains why. Before it starts, it also sends one request to check for the Access redirect. So it never records a redirect as if it were a real page.

Useful flags

See Internal API benchmark — overview for the full flag reference. The most useful one while you are setting up a run:

Flag	What it does
--dry-run	Prints the plan and the exact curl commands. Sends no requests. Run this first if you want to see what will happen.

Run it again to measure the session middleware

The session middleware on the web server calls the Rails API once on every navigation that carries a session cookie.

The 2026-09 runs were recorded before that middleware shipped, so they are the "before" picture. See Internal API benchmark — 2026-09 analysis.

To measure what the middleware costs:

1. Check which endpoint the middleware calls.
2. If it is not /identities/user_sessions/current, add --session-endpoint /the/new/path.
3. Run the same three commands.
4. Compare /privacy and /employers/dashboard between the old report and the new one. The difference is what the middleware costs.

Moving a report into the docs site

Each report starts with Docusaurus front matter. It uses only Markdown that MDX 3 accepts. To move one in:

1. Copy the file into docs/50-59-frontend/41-board-web/internal-api-benchmark/benchmark-results/.
2. Name it <year>-<month>-benchmark-<env>-<mode>.md, for example 2026-09-benchmark-prod-server.md.
3. Set title and sidebar_label in the front matter so the page reads well in the sidebar. Leave the rest of the report as it is — it is the raw record.
4. Run npm run build to confirm it compiles.