DataMonarch/locust-load-test

Load testing with locust and uv

This skill takes a project from "how much can it take?" to a report with numbers the team can act on. Do the steps in order, because each one feeds the next. The user makes two decisions, the test level and the traffic scenario, and they make them only after they have seen what you found in the code.

$SKILL is this skill's folder, the directory you read this SKILL.md from. Set it once in the shell (SKILL=/path/to/locust-load-test). The load test itself lives in the target repository, in loadtest/ or wherever the user chooses.

1 Discover  -> loadtest/endpoints.json   every endpoint, with its callers and cost hints
2 Classify  -> loadtest/PLAN.md          core / support / peripheral table; show it to the user
3 Ask       -> tiers + scenario + environment + constraints
4 Build     -> loadtest/ (a uv project)  harness.py, payloads/, run scripts
5 Run       -> loadtest/results/<run>/   smoke first, then the chosen tiers, riskiest last
6 Collect   -> results/<run>/metrics, profiles   server-side view, when reachable
7 Report    -> loadtest/report.html + findings in plain words

1. Discover the endpoints

uv run $SKILL/scripts/discover_endpoints.py . -o loadtest/endpoints.json --markdown loadtest/endpoints.md

The scanner reads the code and writes nothing else. It finds route declarations across the common stacks: Python, Node, JVM, Go, Ruby, .NET and PHP frameworks, plus OpenAPI files, GraphQL schemas, gRPC protos, websockets and SSE. It resolves router prefixes and mounts where it can. It also counts the client call sites (fetch, axios, httpx, requests, generated clients, tests, and the links, forms and htmx attributes of server-rendered templates) for each endpoint, so you can see what the product's own UI and services use. Each entry has the method, the path, file:line, the handler, auth hints, body hints, a streaming hint, cost hints, the calls the handler makes, and who calls it.

A framework's own route list beats any static scan, because the framework has already resolved every prefix, mount and generated route. When the app can produce one, use it as the source of truth. That means an OpenAPI document from a running dev server (/openapi.json), flask routes, bin/rails routes, php artisan route:list --json, or Spring's /actuator/mappings:

uv run $SKILL/scripts/discover_endpoints.py . --route-dump http://localhost:8000/openapi.json -o loadtest/endpoints.json

Then read the output critically before trusting it:

• unmatched_client_calls are either routes the scan missed or calls to other services.
• prefix_resolved: false and notes mark the guesses. shared_handler: true means one function serves several methods and the scan could not tell which method its cost hints belong to; read that handler.
• existing_load_tests lists locustfiles or k6 scripts already in the repo. Reuse their payloads.
• Spot-check five entries against the source files.

Read references/discovery.md when the inventory looks thin, when a stack is not covered, or when you need the route-dump command for a framework.

2. Classify: core, support, peripheral

Sort the endpoints by judgement. Do not sort them by keyword: the same /search is core in a shop and support in an admin console.

Tier	What it is	Typical examples
core	What users come for, and usually where the heavy work happens: compute, model calls, data processing, the main write path	checkout in a shop; convert or extract in a document service; completions in a chat API
support	What makes the core usable	login, sessions, settings, lists and search over the user's own data, uploads that feed core, admin CRUD
peripheral	What the product does not exist for	health, metrics, docs, static files, debug routes, webhooks from services it does not own

Evidence to weigh:

• what the README and docs say the product is for;
• what the UI calls most (client_calls) and what the tests exercise (test_calls);
• what the handler costs: follow calls into the service layer, since cost_hints only sees the handler body. Model calls, DB writes, external calls, CPU work (password hashing, compression, images: cpu), file processing and queue publishes are what make an endpoint expensive;
• naming;
• what the team's own monitoring or logs say is busiest, if you can see them.

Write loadtest/PLAN.md with this table, one row per endpoint you intend to load, and a short list of the endpoints you left out and why:

| endpoint | tier | why | cost hints | payload source |
|---|---|---|---|---|
| POST /api/orders/{id}/checkout | core | the purchase path; UI calls it from 3 places | db writes, payment API call | tests/fixtures/order.json |

Show this table to the user before asking the questions, and invite corrections. A wrong tier makes every later number answer the wrong question, and the user knows their product better than any scan.

3. Ask the user

Use the AskUserQuestion tool. If it is not available, ask the same questions as a plain numbered list and wait for the answers. Ask only after the user has seen the plan table.

• Q1 Core level. Tier 1, 2 or 3.
• Q2 Support level. Tier 1, 2 or 3, or leave support out.
• Q3 User-load scenario. Two to four options derived from this project (see below), with the one you recommend first, marked "(Recommended)", and one line on why it fits.
• Q4 Target environment. Which deployment to load (staging, a local stack, a load-test copy). Never load a production system unless the user says yes to that explicitly. Load spills over onto shared databases, tenants and bills.

Tier	Runs	Answers	Costs
1 Smoke / baseline	low steady load (1 to a few users) for a few minutes	does every path work, and what is the latency with no contention?	minutes; negligible risk
2 Expected load	step ramp to the expected peak, hold it, then a short soak at that level	does it meet the latency target at its busiest, with what headroom?	about an hour; the target runs at its real peak
3 Stress / capacity	step ramp until a stop rule fires (the knee and the breaking point), plus a spike	where does it break, how, and does it recover?	one to two hours; the target will saturate and may fall over, so it needs a window and someone watching

Derive Q3's options from what the project says about its traffic: README and docs (who uses it, seat counts, tenants), config (rate limits, worker and replica counts, autoscaler bounds, queue sizes, connection pools), cron and batch jobs, and the callers' own timeouts and retries. Common shapes are a business-hours step ramp, a morning spike, a steady soak, an integration burst from another service, a batch import, and mixed interactive plus background traffic. Every option states its numbers: users, spawn rate, think time, duration and request mix. When the project says nothing about its traffic, say so, name the assumption each number rests on, and ask for the expected peak in Q3. references/tiers-and-scenarios.md shows how to turn config into those numbers. Read it before you ask.

After the answers, settle two things in plain text:

• Credentials. They come from a file or an environment variable that the harness reads. They are never pasted into the chat.
• Constraints. List the ones you found and ask for the rest: rate limits, licence or activation windows, what a restart costs, shared tenants, and the cost per request (model tokens, paid APIs). A restart inside a licence window can lock a service out for a day, and a few parallel heavy jobs with one huge input can take a target down. Ask before the load, not after.
• Data state. How many rows the lists and searches hold in real use, whether the data may be reset between runs, and whose rows a test user may edit or delete. An empty table makes a list page free, and rows that a run adds make each later step read more than the one before.

4. Build the harness

mkdir -p loadtest && cd loadtest
cp $SKILL/templates/pyproject.toml . && uv sync   # its own .venv and uv.lock; locust pinned, Python 3.11+
cp $SKILL/templates/{harness.py,run_local.sh,run_plan.sh,collect.sh,seed.py,loadtest.env.example} .
cp $SKILL/templates/run_docker.sh .      # when the generator runs on the target's Docker network
cp -r $SKILL/templates/{run_k8s.sh,k8s} .  # when it runs inside the target's cluster
git -C .. status --short                 # only loadtest/ may appear

Do not run uv init here. In a repository that is itself a uv project, it adds loadtest to the app's [tool.uv.workspace], installs locust into the app's .venv and writes an app-level uv.lock. If that has happened, undo it: git checkout -- pyproject.toml in the app, delete the new uv.lock, and run uv sync there. Then fill the template in:

• REQUESTS. Fill the table in harness.py from PLAN.md: tier, name, weight, mix, body and expected status. Take payloads from the project's tests, fixtures and schemas, and put them in loadtest/payloads/. Never use production data. A server-rendered app answers 200 for its login page and its error pages too, so check what only the right page has: expect_text, and expect_location where a redirect is the answer. A row that is redirected without saying so fails.
• REQ_TIMEOUT. Set it to the real caller's timeout: the gateway, the frontend's abort, the client SDK, or the proxy's read timeout. The harness holds every request to it as a deadline for the whole answer, so a later answer fails as deadline exceeded instead of counting as a success that the product would already have reported as a failure.
• MIX. Give each traffic mix with a different cost its own tag, and run each mix as its own scenario (short chat-sized inputs, document-sized inputs, an option that costs extra). Their capacities can differ several times over, and a blended run hides the expensive one.
• Auth. Set AUTH so each simulated user logs in once, not once per request. A form login with a session cookie also needs LOGIN_EXPECT_LOCATION or LOGIN_EXPECT_TEXT, since a wrong password answers 200 too (see loadtest.env.example). Sign-ins get stage login and stay out of the step numbers and stop rules: each lands at the start of the step that added its user, and a slow password hash would end the run on a cost the steady load never pays. To load login itself, make it a row.
• Data. Fill seed.py for the app and run python3 seed.py up --users N --rows M: through the app's API, each test account gets its own rows, which SEED_MANIFEST hands to its user; the row keys capture and consume track what a user creates and deletes. Pair creates with deletes, or reseed between runs.
• Journeys. Multi-step flows (upload then poll, create then read) are a method on a user class; the template has an example.

references/harness.md covers every field, the shapes, the auth modes, streaming, non-HTTP protocols and the three placements. Read it while building.

5. Run safely

1. Smoke first. Run RUN_NAME=smoke SHAPE=smoke ./run_local.sh: one user per user class sends every row once, in table order (put the row that creates an id before the rows that use it), then idles. The log ends with NOTE smoke: every row was sent, or names the rows that were not. Check zero failures, plausible latency and correct bodies. It proves the wiring before anything costs load.
2. Check that the machines are quiet. Before a run that measures capacity, run uptime and vmstat 1 5 on the generator and on the target's host. Other jobs on the same cores make every number a measure of the contention. The harness samples its own host before each run into run.json (host_check and a note); put the target host's numbers in NOTES when it is another machine. The report puts a banner on a run whose host or generator CPU reached 90 % during its steps and caveats its knee. If a machine is busy, wait, pin to idle cores (taskset), or move, and say which.
3. Then the chosen tiers, safest first. Run expected load, then soak, then stress, then spike. Risky inputs (the largest payloads, the extra-cost options) go last, so if they hurt the target, nothing before them is lost. run_plan.sh runs the list in order and starts each run only when the target's health check passes.
4. Place the generator next to the real callers when the network path matters: the same cluster, VNet or compose network (run_k8s.sh, run_docker.sh). A laptop run measures the internet as much as the service. Every run records GENERATOR, and the generator's own CPU goes to metrics/client.csv; above about 90 % CPU, the numbers describe the generator.
5. Respect the stop rules. The harness ends a run when the failure ratio passes STOP_FAIL_RATIO, when p95 passes STOP_P95_MS, when more than half of the last 20 s failed, or when most sign-ins failed. Many services queue instead of shedding load: zero errors, and latency that only grows. So set the p95 ceiling to the caller's timeout or the SLO. Do not push past a stop rule "to see".
6. Watch health between runs and let queues drain (COOLDOWN in run_plan.sh).
7. Clean up everything you created: ./run_k8s.sh down (pod, ConfigMaps, Secret), stopped containers, test users and test data. List for the user anything that cannot be removed, such as soft-deleted rows or audit entries.

Read references/safety.md before the first run beyond smoke, and before any run against a shared system.

6. Collect the server side

Client numbers say what happened. Server numbers say why. Start collectors before locust, stop them after it, and write to the same run directory:

uv run $SKILL/scripts/collect_metrics.py proc --match 'gunicorn|uvicorn' --out results/ramp/metrics/proc.csv &  # on the target's host
uv run $SKILL/scripts/collect_metrics.py prom http://target:8000/metrics --keep http_,process_ --out results/ramp/metrics/server.csv &
uv run $SKILL/scripts/collect_metrics.py kubectl -n shop -l app=api --replicas --context CTX --out results/ramp/metrics/k8s.csv &
uv run $SKILL/scripts/collect_metrics.py azure --resource ID --metrics "Percentage CPU" --run-dir results/ramp   # after the run

collect.sh holds the live lines for one run. COLLECT_CMD=./collect.sh ./run_plan.sh starts it before each run with that run's directory and stops it after, so every run carries its own server metrics.

The sources are a Prometheus /metrics endpoint, docker stats, kubectl top, local /proc, and Azure, AWS or GCP monitors fetched for the run's window. Cloud monitors give one-minute points and sometimes lack a metric, such as memory on some VM types. Say what was unavailable rather than leaving a gap unexplained. Two checks matter more than any chart:

• Autoscaling. Did the autoscaler actually add ready replicas, or only ask for them? It can request pods that no node has room for. Use --replicas.
• Health checks. Does /health stay fast at saturation? It often does, and then it says nothing about capacity.

For CPU profiles as folded stacks, read references/profiling.md first. Its rules:

• Default to non-blocking. Use py-spy --nonblocking for every Python process. --native pauses the process on each sample. On CPU-bound inference it stalled the service in seconds, and loading its symbols exhausted memory. Use it only at --rate 20, on one replica, with memory headroom, and never during a step whose latency you will report.
• Check for perturbation. Afterwards, read the latency around the profiling window. A jump there means the profiler perturbed the run, and that step must not be quoted.
• Profile in parallel. Profile every role at the same held step, at the same time, for the same 40–60 s.
• py-spy without root. When yama ptrace_scope=1 blocks non-root py-spy, a sitecustomize.py that calls prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY) lets it attach by PID. Spawned interpreters inherit it. Workers forked without exec (Gunicorn, Celery prefork) need the call in their post-fork hook.
• Ray. Ray exports Prometheus metrics with ray start --metrics-export-port=N even when its dashboard is off. The ray_serve_* queue and latency series show where requests wait.
• Sealed targets. When only the app port answers (no shell, no agent, no dashboard), say so. Offer to reproduce locally: the same commit, config and model or data revision, pinned to the target's CPU count (taskset plus the framework's own CPU limit). Check parity with one known request first, and label every profile "reproduction" with the differences listed.

Convert other profilers' output with uv run $SKILL/scripts/fold_profile.py node|perf|pprof|meta ....

7. Report

uv run $SKILL/scripts/report.py loadtest/results/smoke loadtest/results/ramp-core loadtest/results/soak-core \
    -o loadtest/report.html --title "Shop API load test, staging"
uv run $SKILL/scripts/import_locust.py old/run old/run-steps.json loadtest/results/old-run   # runs with only locust CSVs

Open the report and look at every panel before writing anything. The report says which panels it could not draw and why. Then write the findings, each one a number with its context:

• Capacity: the req/s sustained with p95 within the SLO or the caller's timeout.
• The knee: the users and req/s where throughput stops rising, and the latency there.
• Past the knee: how it fails. Errors, or queueing with rising latency? Does it apply backpressure (429/503), or does it let callers time out?
• Health and scaling: whether health checks reflect saturation, and whether autoscaling actually added capacity.
• Soak stability: p95 drift, memory growth, error creep.
• Per mix: capacity for each traffic mix.
• Recommendations: concrete ones (limits, timeouts, workers, replicas, caching, backpressure), ranked by effect.

Write them to loadtest/FINDINGS.md (Markdown: headings, - and 1. lists, blank lines between paragraphs) and rebuild the report with --findings loadtest/FINDINGS.md, so they open the page above the charts they rest on.

Every number says where the generator ran and what it ran against: the target URL, the version, the replica count and the CPU. A profile from a reproduction says so. Read references/reporting.md for the definitions, the findings template, and how to word the caveats.

Reference files

• references/discovery.md: read when the scan looks incomplete, a stack is not covered, or you need a framework's route-dump command.
• references/tiers-and-scenarios.md: read before asking Q1–Q3. It has tier details, run lists per tier, and how to turn config into scenario numbers.
• references/harness.md: read while building. It covers REQUESTS fields, shapes, auth, streaming, journeys, non-HTTP protocols and placement.
• references/safety.md: read before the first run beyond smoke, and before touching any shared system.
• references/collectors.md: read when setting up server-side metrics.
• references/profiling.md: read before attaching any profiler.
• references/reporting.md: read when writing findings.
• references/data-contract.md: the run-directory format that every run writes and the report reads.