# Foundations: From "It Works" to "It Works for 500 Users

Imagine your functional tests all pass. The login works, the cart works, checkout returns a 200. Then the release goes out, a few hundred people try to check out at the same time, and the same checkout takes twenty seconds or fails outright.

Nothing in your test suite was wrong. It just answered a different question. Functional tests ask "does it work?" **Performance testing** asks "does it still work, fast enough and reliably enough, when many people use it at once?" It's a kind of **non-functional testing**, which means testing *how well* a system does its job (speed, stability, capacity) rather than *whether* it does it.

This is Part 1 of *Performance Testing from Zero to Production*. By the end you'll be able to plan a realistic test, write and run it with k6, read the results without being fooled by averages, and turn requirements into pass or fail rules.

One rule before anything else. **Only run load tests against systems you own, or that you have written permission to test.** A load test is a lot of traffic on purpose. Against someone else's system, it can look like an attack and can cause real harm. That includes public "demo" sites.

## Why functional testers should care about performance

A functional test checks one user doing one thing. Real systems serve many users doing many things at the same time, and that changes the behaviour in ways a single request never shows. Databases lock rows, connection pools run out, caches fill up, and queues grow.

None of that appears when you click through a flow on your own. So "it works" and "it works for 500 people at once" are two different claims, and only one of them is covered by your regression suite.

Skipping performance testing usually doesn't fail loudly in development. It fails in production, at the worst possible time, during a sale or a launch or the first busy Monday. Typical results are slow pages, timeouts, errors under load, and an expensive rush to add servers without knowing whether servers are even the problem.

The good news is that you already have most of the skills. Designing test cases, defining expected results and automating checks all carry over. What's new is the vocabulary and the way you read results.

## The core vocabulary

**Response time** is how long a user waits from sending a request until the full response arrives. **Latency** is often used to mean the same thing. Some people use it more narrowly, for the delay before the response *starts* to arrive or for network delay alone. There's no single agreed definition, so check what a number actually measures before comparing it with another. In k6, the main response-time metric is `http_req_duration`. It covers sending the request, waiting and receiving the response, but not looking up the server's address or opening the connection.

**Throughput** is how much work the system completes in a period of time. For APIs, it's usually measured in **requests per second** (RPS).

**Concurrency** is how many things happen at the same time. **Concurrent users** are users active in the system at the same moment, not the total number of users who visit in a day.

Load tools simulate users with **virtual users** (VUs). A VU is a small worker that runs your script again and again, like an automated tester running the same test case in a loop. Each full run of the script is one **iteration**. A VU isn't a real person, though. It only does what your script tells it to.

Real people pause between actions to read, scroll and decide. That pause is **think time**, and you add it to scripts on purpose. More on that later.

Two more terms describe failure. **Error rate** is the share of requests that fail, for example 2 out of 100, or 2%. A **timeout** is when the client gives up waiting for a response after a set time limit. Timeouts count as errors, and they're often the first sign of trouble.

Here's how these fit together.

![Timeline of three virtual users, each alternating a 0.5 second response time and 1.5 second think time. One iteration takes about 2 seconds, so 10 virtual users produce about 5 requests per second.](https://cdn.hashnode.com/uploads/covers/6ae39fea70d93e191f662f3a/3ead1ba2-1aea-44ff-a49e-8375c557395a.png)

For example, imagine each VU sends one request, waits 0.5 seconds for the response and then thinks for 1.5 seconds. That's one request every 2 seconds per VU. With 10 VUs, you get about 5 requests per second.

Now imagine the system slows down and responses take 1.5 seconds. Each loop now takes 3 seconds, so the same 10 VUs send only about 3.3 requests per second. In a test with a fixed number of VUs, throughput drops *because* the system got slower. This is a simplified model with round numbers, but the lesson is real. The number of VUs, the response time, the think time and the throughput are all linked, so you can't read one without the others. Part 2 goes deeper into this relationship and the traps it creates.

> ✅ **Best practices**
> - Report response time, throughput and error rate together. Any one of them alone can mislead you.
> - Say what a number measures. "Response time" from the client, from the server and from a monitoring tool can be three different things.

## Why averages lie

Run a test and you'll get thousands of response times, not one. They won't all be the same. Most will be close together, and a few will be much slower. How those values are spread out is called the **distribution**.

The **average**, or **mean**, adds up all values and divides by how many there are. It's familiar, but it hides the slow requests. A handful of very slow responses get blended into many fast ones, and the result can describe almost nobody's actual experience.

The **median** is the middle value when you sort all the response times. Half the requests took this long or less. It tells you what a typical request looks like.

A **percentile** tells you the value below which a given share of requests fall. **p95** is the response time that 95% of requests were at or below. So at most about 1 in 20 requests took longer. You'll also see **p50** (the median), **p90** and **p99**.

Here's a worked example. Imagine 100 requests. Ninety of them take 200 ms, and ten take 3,000 ms (3 seconds).

- The mean is 480 ms. It sounds acceptable.
- The median is 200 ms. That's the typical request.
- The p95 is 3,000 ms. One request in ten took 3 seconds.

The mean describes nobody. It's slower than what most users saw and much faster than what the unlucky few saw. (Percentiles have more than one calculation method, so tools can differ slightly at the edges. With real data volumes this rarely matters.)

![A latency distribution with most requests around 100 to 300 ms and a long tail to the right. The median is about 200 ms, the mean about 240 ms, p95 about 540 ms and p99 about 810 ms.](https://cdn.hashnode.com/uploads/covers/6ae39fea70d93e191f662f3a/7167863c-e4a3-41b5-bfa3-5314346163b2.png)

The slow end of the distribution is called the **tail**, and **tail latency** means response times at high percentiles like p95 and p99. Response times often have a long tail to the right, as in the picture. That's why the mean sits to the right of the median, pulled up by the slow requests.

Why care about 1% of requests? Because users make many requests. A single page can trigger dozens of API calls, and a session has many pages. If 1% of requests are slow, a large share of *users* will hit at least one of them. For example, a user who makes 100 requests has about a 63% chance of hitting at least one slow one, assuming slow requests happen at random. Google's SRE book makes a similar argument for using percentiles rather than averages when you define service targets ([Google SRE book](https://sre.google/sre-book/service-level-objectives/)).

An **outlier** is a single value far away from the rest. Don't delete outliers to make a report look tidy. Sometimes they're noise, but often they're the most important thing in the data.

> ⚠️ **Common mistakes**
> - Reporting only the average response time.
> - Judging a run by its maximum alone. The max is a single request, so it's very sensitive to one-off events. Don't ignore it either, because it can point to timeouts.
> - Comparing p99 values from very short tests. With few requests, p99 rests on a tiny number of samples. Part 2 covers an even subtler trap in how slow requests get measured.

## The test types and when to use each

You usually don't run one performance test. You run several, each with a different question and a different **load profile**, the shape of the load over time.

Most profiles have three phases. **Ramp-up** is the period when the tool gradually adds VUs. **Steady state** is the period when load stays at a stable level and you take your main measurements. **Ramp-down** gradually removes VUs at the end. A gradual ramp-up gives the system time to warm up and shows at what load problems begin. Jumping from 0 to full load at once is a special test of its own, as you'll see below.

![Six small charts of virtual users over time. Smoke is a tiny short plateau, load is a plateau at expected load, stress is a higher plateau, spike is a sudden peak, soak is a long plateau at expected load and breakpoint is a steady climb until the system breaks.](https://cdn.hashnode.com/uploads/covers/6ae39fea70d93e191f662f3a/78510270-125a-4219-88c7-8f02b00d244a.png)

| Test type | Question it answers | Load shape | What "pass" looks like |
|---|---|---|---|
| **Smoke test** | Does the script work and does the system respond at all? | Very few VUs for a short time | No errors, sensible response times |
| **Load test** (average load) | Does it meet its targets under expected load? | Ramp up to normal load, hold, ramp down | Targets met for response time and error rate |
| **Stress test** | How does it behave above normal load? | Ramp up beyond normal load, hold | Slows down gracefully, no data loss, recovers |
| **Spike test** | Does it survive a sudden surge? | Very fast jump to high load, then back | Survives, recovers within an agreed time |
| **Soak test** | Does it get worse over a long time? | Normal load held for hours | No steady growth in response time, memory or errors |
| **Breakpoint test** | Where are its limits? | Load keeps increasing until something fails | You learn the limit and how it fails |

A smoke test is the first thing you run. k6's own guidance suggests a handful of VUs for up to a few minutes ([k6 smoke testing](https://grafana.com/docs/k6/latest/testing-guides/test-types/smoke-testing/)).

A load test checks your normal, expected traffic. People also use "load testing" loosely for all of these test types. In k6's guides and in this series, it means this one. The k6 guide suggests a ramp-up of roughly 5 to 15% of the total test time ([k6 load testing](https://grafana.com/docs/k6/latest/testing-guides/test-types/load-testing/)). Treat that as a starting point, not a law.

A stress test pushes past normal load to watch **degradation**, how quality drops as load rises. Does it slow down gradually or fall over? It also checks **recovery**, whether the system returns to normal once the extra load goes away.

A spike test checks sudden surges, such as the start of a ticket sale.

A soak test, also called an **endurance test**, holds normal load for a long time. It finds problems that build up slowly, such as memory leaks, growing queues or logs filling a disk. How long? It depends. k6's guide lists durations from a few hours up to a few days ([k6 soak testing](https://grafana.com/docs/k6/latest/testing-guides/test-types/soak-testing/)). A useful way to decide is to run long enough to cover the slowest cycle you care about, for example a nightly batch job, a cache expiry period or a full business day.

A breakpoint test keeps increasing load until the system fails. It shows you the **breaking point** and how the system fails. That's not the same as **capacity**, the maximum load the system can handle while still meeting its targets. Capacity is usually lower than the breaking point, and to trust it you need to hold that load for a while, not just pass through it on the way up. The part of the system that limits capacity is the **bottleneck**. When a resource is fully used and work starts to queue, that's **saturation**. Run them only where breaking things is expected.

> ✅ **Best practices**
> - Always start with a smoke test. It catches broken scripts before you waste an hour on a big run.
> - Write down the question before choosing the test type. "Can we handle Black Friday?" and "Do we leak memory?" need different tests.

## Designing a realistic test

A load test is only as useful as it is realistic. Ten thousand VUs hitting one endpoint in a tight loop tells you very little about real users.

### User journeys and workload mix

Start with **user journeys**, the sequences of steps real users take, such as browse a product or search and buy. A journey is the performance-testing cousin of an end-to-end test case.

Then decide the **workload mix**, the share of traffic each journey gets. Your **workload model** is the whole picture of who does what, how often and at what rate.

![A bar chart of an example workload mix. Browse is 60%, search 25%, add to cart 10% and check out 5%. Checkout is the smallest share but often the most expensive.](https://cdn.hashnode.com/uploads/covers/6ae39fea70d93e191f662f3a/7148c78f-4041-4f92-be94-50d93bc5e98b.png)

These percentages are examples only. Take real ones from your analytics, access logs or product team. If you have no data, write down your assumption.

### Think time and pacing

As mentioned earlier, think time is the pause a real user takes between actions. Without it, each VU fires requests as fast as the system answers, which is far more aggressive than a person.

**Pacing** is controlling how often a VU *starts* an iteration, for example "begin one journey every 10 seconds", as long as the previous one finished in time. Think time is a pause *inside* the journey, and pacing controls the rhythm *between* journeys.

Typical think times depend on your users and pages, so measure them where you can. Adding a little randomness, for example 1 to 3 seconds instead of exactly 2, stops all VUs from moving in lockstep.

A related note. k6 can generate load in two ways. One fixes the number of VUs, as in this article. The other fixes the rate at which new iterations start, whatever the response times ([k6 open and closed models](https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/)). The choice changes what your results mean, and Part 2 covers it in depth.

### Test data and parameterisation

**Test data** is the data your script uses, such as users, products and search terms. If every VU logs in as the same user and views the same product, caches make everything look fast and the database barely works. That's a very optimistic test.

**Parameterisation** means feeding different data into each request or iteration instead of hard-coding one value. It's the same idea as data-driven functional tests. In k6, you can load a file of test users once and share it across VUs with `SharedArray` from `k6/data`.

### Dynamic values and sessions

Many systems return values that you must send back later, such as a login token, a session ID or an order number. Capturing a value from one response and reusing it in a later request is called **correlation**.

A **session** is the context that ties one user's requests together, such as who they are and what's in their cart. Some systems keep it on the server and send the client a session ID. Others keep it on the client, for example in a signed token. Either way, your script has to carry the right value from one request to the next.

```javascript
// Example only: placeholder URL and credentials
const login = http.post(`${BASE_URL}/login`, JSON.stringify({ user: 'test-user-01', pass: 'placeholder' }), {
  headers: { 'Content-Type': 'application/json' },
});
const token = login.json('token'); // capture the dynamic value
http.get(`${BASE_URL}/orders`, { headers: { Authorization: `Bearer ${token}` } });
```

If you hard-code a token you copied once, the script works until the token expires, and then every request fails. That's the classic sign of a missing correlation.

### The environment you test in

**Test environment parity** means how closely your test environment matches production in size, configuration, data volume and dependencies. A test on a half-size database with an empty cache predicts production poorly. Perfect parity is rare and expensive, so write down the differences and be careful about what you conclude from them.

Also tell people before you run a test in a shared environment. A load test in staging can block other teams, trigger alerts and, if third-party services are connected, send real traffic to systems you don't own.

> ⚠️ **Common mistakes**
> - Testing one endpoint in a loop and calling it a "realistic" test.
> - Reusing one test user for all VUs.
> - Forgetting that payment providers, email services or other third parties may be called during the test.

## Your first k6 script

**k6** is a free, open-source load-testing tool from Grafana Labs. You write tests in JavaScript and run them from the command line. I use it here because its scripts are short and readable, but every idea in this article applies to other tools too.

Install it with your package manager, for example `brew install k6` on macOS or `winget install k6 --source winget` on Windows. Other options, including Docker, are in the [k6 installation guide](https://grafana.com/docs/k6/latest/set-up/install-k6/).

Here's a small test of the "browse" journey.

```javascript
// Example only: placeholder URL. Run only against a system you own.
import http from 'k6/http';
import { check, sleep } from 'k6';

const BASE_URL = 'http://localhost:8080';

export const options = {
  stages: [
    { duration: '1m', target: 20 }, // ramp-up to 20 VUs
    { duration: '3m', target: 20 }, // steady state
    { duration: '1m', target: 0 },  // ramp-down
  ],
};

export default function () {
  const list = http.get(`${BASE_URL}/products`);
  check(list, { 'list: status is 200': (r) => r.status === 200 });
  sleep(Math.random() * 2 + 1); // think time: 1 to 3 seconds

  const id = Math.floor(Math.random() * 10) + 1; // pick a product
  const item = http.get(`${BASE_URL}/products/${id}`);
  check(item, { 'item: has a name': (r) => r.json('name') !== undefined });
  sleep(Math.random() * 2 + 1);
}
```

Let's walk through it.

The `import` lines load k6's HTTP client and two helpers, `check` and `sleep`.

`options` configures the run. Each entry in `stages` is a **stage**, a period in which k6 moves the number of VUs towards a `target` over a `duration`. These three stages give you ramp-up, steady state and ramp-down.

Behind the scenes, k6 runs every test as one or more **scenarios**. A scenario describes how a group of VUs runs your code, and each scenario uses an **executor**, the k6 component that decides how VUs and iterations are scheduled. When you use `stages`, k6 creates one scenario for you with the `ramping-vus` executor. You don't need to think about executors yet. Just know the word when you see it in the docs.

The `default` function is the journey. Each VU runs it again and again, and each run is one iteration.

`check` is k6's version of an assertion, with one important difference. A failed **check** is counted and reported, but it doesn't stop the test or make it fail. In the docs' words, failed checks "do not cause the test to abort or finish with a failed status" ([k6 checks](https://grafana.com/docs/k6/latest/using-k6/checks/)). You'll turn checks into a real pass or fail rule in a moment.

`sleep` pauses the VU, here for a random 1 to 3 seconds. That's the think time. The random product ID is a tiny example of parameterisation.

Run it with:

```bash
# Example only
k6 run first-test.js
```

If you've used other tools, or your team already has one, the ideas map closely:

| | k6 | JMeter | Gatling | Locust |
|---|---|---|---|---|
| Tests written in | JavaScript or TypeScript | Mostly a GUI, saved as `.jmx` files | Java, Kotlin, Scala, JavaScript or TypeScript | Python |
| Simulated user | Virtual user (VU) | Thread, in a thread group | Virtual user | User |
| Think time | `sleep()` | Timers | `pause()` | `wait_time`, e.g. `between(1, 3)` |
| Load shape | `stages` | Thread group ramp-up period | Injection profiles, e.g. `rampUsers` | `-u` and `-r` flags, or a `LoadTestShape` class |
| Response validation | `check()` | Assertions | Checks | `catch_response=True` and `response.failure()` |
| Failed validation fails the request? | No, it's counted separately | Yes | Yes | Yes |
| Test-wide pass or fail | `thresholds` | Nothing built in. Added with plugins or a CI step | Assertions | Exit code 1 on any failure by default, custom rules in code |
| Licence | AGPL-3.0 | Apache 2.0 | Apache 2.0 (Community Edition) | MIT |

The row to remember is validation. In JMeter, Gatling and Locust, a failed assertion or check marks the request as failed, so it shows up in the error rate. A failed k6 check doesn't, which is why the next sections use thresholds.

> 🧪 **Try it**
> Run the script against an application you run on your own machine. If you don't have one, run a local copy of a test service such as [go-httpbin](https://github.com/mccutchen/go-httpbin) in Docker and change the paths to its endpoints. First set the stages to 2 VUs for 30 seconds as a smoke test. Never point it at a public website.

## Reading the results

When the test finishes, k6 prints an end-of-test summary. Here's an example of what it can look like. The numbers are made up for illustration, and the output is shortened.

```text
# Example only: illustrative numbers, shortened output (k6 v2.x)

     scenarios: (100.00%) 1 scenario, 20 max VUs, 5m30s max duration (incl. graceful stop):
              * default: Up to 20 looping VUs for 5m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)

  █ TOTAL RESULTS

    checks_total.......: 2280    7.6/s
    checks_succeeded...: 99.61%  2271 out of 2280
    checks_failed......: 0.39%   9 out of 2280

    ✓ list: status is 200
    ✗ item: has a name
      ↳  99% — ✓ 1131 / ✗ 9

    HTTP
    http_req_duration..............: avg=120ms min=40ms med=95ms max=1.8s p(90)=210ms p(95)=340ms
      { expected_response:true }...: avg=115ms min=40ms med=95ms max=1.2s p(90)=200ms p(95)=320ms
    http_req_failed................: 0.39%  9 out of 2280
    http_reqs......................: 2280   7.6/s

    EXECUTION
    iteration_duration.............: avg=4.2s  min=2.1s med=4.2s max=7.9s p(90)=5.5s p(95)=5.8s
    iterations.....................: 1140   3.8/s
    vus............................: 1      min=1  max=20
    vus_max........................: 20     min=20 max=20

    NETWORK
    data_received..................: 1.1 MB 3.7 kB/s
    data_sent......................: 250 kB 830 B/s
```

These numbers are **client-side metrics**, measured by the load tool from the outside. They show how long HTTP requests took from the load generator's point of view. They don't include what a browser does after the response arrives, such as rendering the page or running JavaScript. Measuring that needs browser-level testing, which k6 also supports with its browser module. Here's what the main lines mean ([k6 metrics reference](https://grafana.com/docs/k6/latest/using-k6/metrics/reference/)):

| Line | What it means |
|---|---|
| `checks_*` | How many checks passed and failed. Here 9 "has a name" checks failed. |
| `http_req_duration` | Response time of each request. `med` is the median, `p(90)` and `p(95)` are percentiles. |
| `{ expected_response:true }` | The same, counting only successful responses. |
| `http_req_failed` | The share of HTTP requests k6 classed as failed. By default, that's any status outside 200–399 (you can change this with `http.setResponseCallback`). It won't catch a 200 response that contains a failed operation. That's what checks are for. |
| `http_reqs` | Total requests and the throughput (requests per second). |
| `iteration_duration` | How long one full journey took, including think time. |
| `iterations` | How many journeys completed. |
| `vus`, `vus_max` | How many VUs were active. |

What should you look at first? I'd suggest this order.

1. **Errors.** Look at `http_req_failed` and the failed checks. Response times from a test full of errors mean very little, because errors are often fast.
2. **Percentiles, not the average.** Here the median is 95 ms but the p95 is 340 ms and the max is 1.8 s. Most requests are quick, and there's a noticeable tail.
3. **Throughput.** Is it what you planned? If it's lower than expected, slow responses may be holding the VUs back, as in the earlier example.

Notice that k6 shows `p(90)` and `p(95)` but not `p(99)` by default. You can change this with the `summaryTrendStats` option, for example to add `p(99)`.

Treat one run as one data point, not a verdict. Results vary between runs, and the machine running k6 can itself become the bottleneck. Part 2 is about exactly that, making sure your numbers can be trusted.

> ⚠️ **Common mistakes**
> - Reading response times without checking the error rate first.
> - Including ramp-up and ramp-down in the conclusion without thinking about it. The summary covers the whole run, including the ramps.
> - Running the load tool on an overloaded laptop and blaming the server.

## Pass or fail: thresholds as quality gates

Checks tell you what happened. Thresholds tell you whether that's acceptable.

A **threshold** is a pass or fail rule on a metric for the whole test. If any threshold fails, k6 marks the test as failed and exits with a non-zero exit code (99 in current versions). That's what lets a CI pipeline stop on a performance problem.

Add a `thresholds` block to `options`:

```javascript
// Example only: placeholder thresholds. Agree real values with your team.
export const options = {
  thresholds: {
    http_req_failed: ['rate<0.01'],   // fewer than 1% of requests fail
    http_req_duration: ['p(95)<500'], // 95% of requests finish under 500 ms
    checks: ['rate>0.99'],            // more than 99% of checks pass
  },
};
```

In the real script, this block sits inside the same `options` object as `stages`. The syntax is a metric name plus an expression such as `p(95)<500` ([k6 thresholds](https://grafana.com/docs/k6/latest/using-k6/thresholds/)). The `checks` threshold is what lets failed checks fail the test, but only when too many of them fail. `rate>0.99` still allows up to 1% of checks to fail. In the earlier example, 9 failures out of 2,280 checks is a 99.6% pass rate, so that run would pass this threshold. Set the limit based on how much failure you can accept, and be stricter for critical operations such as checkout.

When a threshold fails, the summary says so at the top. Here's an example from a different, slower run.

```text
# Example only: illustrative numbers
  █ THRESHOLDS

    http_req_duration
    ✗ 'p(95)<500' p(95)=620ms

    http_req_failed
    ✓ 'rate<0.01' rate=0.20%
```

Where do the numbers come from? Ideally from agreed targets, not from a guess. An **SLA** (service level agreement) is a formal promise to customers, often with penalties if it's broken. An **SLO** (service level objective) is a target for how well a service should perform, usually set by the team that runs it, such as "95% of checkout requests complete in under 500 ms". Thresholds are a natural way to test against SLOs. If you don't have SLOs yet, start from a **baseline**, the measured performance of the current version, and fail the test when results get clearly worse. Part 2 covers deriving thresholds from SLOs and baselines properly.

A threshold used to block a release is a **quality gate**. Gates belong in CI, but not every test does. Full load, soak and stress tests take a long time and need a stable environment, so run them on a schedule or before major releases. A short smoke test with thresholds, a few VUs for a minute or two, fits into a CI pipeline and catches obvious problems early.

### Running a smoke test in GitHub Actions

Here's what that can look like in GitHub Actions. The workflow starts your application inside the CI runner, installs k6 with Grafana's official setup action and runs a smoke test.

```yaml
# Example only: placeholder paths. The app runs inside the CI runner.
name: performance-smoke

on: pull_request

jobs:
  k6-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - name: Start the app under test
        run: docker compose up -d --wait # placeholder: start your app your way
      - uses: grafana/setup-k6-action@v1
      - name: Run the k6 smoke test
        run: k6 run tests/smoke.js
```

Here `tests/smoke.js` is the same journey with a smaller load, for example `vus: 2` and `duration: '1m'` in `options`, plus the thresholds above. If a threshold fails, k6 exits with a non-zero code, the step fails and the pull request shows the job as failed. Grafana also offers [`run-k6-action`](https://github.com/grafana/run-k6-action), which wraps `k6 run` and can run several scripts at once.

Two cautions. First, the target. Testing an app started inside the runner is the safest option, as long as its own dependencies, such as databases, payment providers or email services, are test versions or mocks too. Pointing CI at a shared environment needs the same permission and warnings as any other load test. Second, the numbers. GitHub-hosted runners are shared virtual machines, so their speed varies from run to run. That's fine for catching obvious breakage, but keep the thresholds loose and don't treat CI results as precise benchmarks. Part 2 explains why.

> ✅ **Best practices**
> - Always set thresholds for error rate as well as response time.
> - Keep thresholds in the script, under version control, so changes are reviewed like code.
> - Use a small smoke test as the CI gate, and run the big tests on a schedule.

## Seeing inside the system

Imagine your test shows the p95 rising from 300 ms to 2 seconds as load grows. Why?

The load test can't tell you. It sees the system from outside, the same way a user does. It shows you symptoms, not causes. To find the cause, you need **server-side metrics**, measurements taken inside the system, such as CPU usage, database query times or queue lengths.

**Observability** is how well you can understand what's happening inside a system from the data it produces. That data usually comes in three forms ([OpenTelemetry observability primer](https://opentelemetry.io/docs/concepts/observability-primer/)).

- **Metrics** are numbers measured over time, such as requests per second or memory usage.
- **Logs** are timestamped records of events, such as "payment failed for order 123".
- **Traces** follow one request as it travels through several services, showing where the time went.

A **dashboard** shows these signals as charts, so you can watch them during a test. **Resource utilisation** is how much of a resource, such as CPU or memory, is being used. It's often the first thing to check when response times climb.

![A diagram with the load test at the top seeing response times, errors and throughput. Below is the system with an API, an order service and a database, each with CPU and memory bars, and the database CPU near its limit. Telemetry flows down to an observability box with metrics, logs and traces.](https://cdn.hashnode.com/uploads/covers/6ae39fea70d93e191f662f3a/e893ba8f-4a4c-4a69-a9cd-18643c466c2c.png)

Common tools include OpenTelemetry for collecting the data, Grafana (with a storage backend such as Prometheus) or Datadog for storing and viewing it. Before you run a serious load test, find out what your team already has, and ask for access to dashboards for the system you're testing.

Part 3 shows how to put client-side and server-side data together to find the actual bottleneck.

## Common mistakes and the ethics of load testing

Here they are in one place.

- **No goal.** Running a test without writing down the question it should answer.
- **Skipping the smoke test.** Then discovering after an hour that the script was failing from the first minute.
- **Averages only.** Hiding the tail that real users feel.
- **Ignoring errors.** Fast responses can be fast failures.
- **Unrealistic load.** No think time, one endpoint, one user and one product.
- **Hard-coded dynamic values.** Tokens that expire halfway through the test.
- **An overloaded load generator.** The machine running k6 runs out of CPU or network, and the test measures that instead of your system.
- **Unclear environment.** Comparing results from a small test environment with production expectations without saying so.
- **One run, one conclusion.** Results vary, so repeat important runs before deciding.
- **Testing alone.** Not telling the teams who own the environment, the database or the alerts.

Now the ethics, stated plainly. **Only test systems you own or have written permission to test.** Permission should name the system, the environment, the time window and the expected load. Never test public websites, including demo sites, and never test third-party services connected to your system without their agreement. Tell the people who are on call before you start, and have a way to stop the test quickly. In k6, pressing Ctrl+C stops a running test.

## Recap and what's next

Here are the key ideas.

- Functional tests ask "does it work?" Performance tests ask "does it still work, fast and reliably, under load?"
- VUs, response time, think time and throughput are linked. Read them together, along with the error rate.
- Averages hide slow requests. Use the median and percentiles like p95 and p99.
- Pick the test type from the question. Smoke first, then load, stress, spike, soak or breakpoint as needed.
- Realism matters. Use journeys, a workload mix, think time, varied data and correlation, and be honest about the environment.
- Checks report what happened, and thresholds decide pass or fail.
- Load tests show symptoms, and observability helps find causes.

Part 2 is about trust, making sure your numbers mean what you think they mean. Part 3 is about diagnosis, using observability to find out *why* a system slowed down. Both are hands-on, with tests you can run yourself.

How does your team approach performance testing today? Do you run load tests in CI, before releases, or only after something goes wrong? Share your experience in the comments.

## References

1. Grafana k6 documentation, "Test types". https://grafana.com/docs/k6/latest/testing-guides/test-types/
2. Grafana k6 documentation, "Smoke testing". https://grafana.com/docs/k6/latest/testing-guides/test-types/smoke-testing/
3. Grafana k6 documentation, "Load testing". https://grafana.com/docs/k6/latest/testing-guides/test-types/load-testing/
4. Grafana k6 documentation, "Soak testing". https://grafana.com/docs/k6/latest/testing-guides/test-types/soak-testing/
5. Grafana k6 documentation, "Open and closed models". https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/
6. Grafana k6 documentation, "Install k6". https://grafana.com/docs/k6/latest/set-up/install-k6/
7. Grafana k6 documentation, "Checks". https://grafana.com/docs/k6/latest/using-k6/checks/
8. Grafana k6 documentation, "Thresholds". https://grafana.com/docs/k6/latest/using-k6/thresholds/
9. Grafana k6 documentation, "Built-in metrics". https://grafana.com/docs/k6/latest/using-k6/metrics/reference/
10. Grafana k6 documentation, "End-of-test summary". https://grafana.com/docs/k6/latest/results-output/end-of-test/
11. Beyer, Jones, Petoff and Murphy (eds.), *Site Reliability Engineering*, chapter "Service Level Objectives". Google, 2016. https://sre.google/sre-book/service-level-objectives/
12. OpenTelemetry documentation, "Observability primer". https://opentelemetry.io/docs/concepts/observability-primer/
13. go-httpbin, a local HTTP test service. https://github.com/mccutchen/go-httpbin
14. Grafana, "setup-k6-action" and "run-k6-action" for GitHub Actions. https://github.com/grafana/setup-k6-action and https://github.com/grafana/run-k6-action