software-tester-skills

Scalability & Capacity Testing

Determining how a system scales and how much capacity it needs — load models, capacity planning, autoscaling validation, bottleneck identification (USE/RED), stress/spike/soak/breakpoint tests, and cost-per-request analysis with k6, Locust, JMeter, Gatling.

Goal: answer "How much can it take, what breaks first, and how do we scale?" — with numbers.

Test types

TestLoad shapeQuestion
Baseline/Loadexpected peak, steadySLO met at normal peak?
Stress / Breakpointramp until SLO breaks / errorsWhere is the limit & failure mode?
Spike10× jump in secondsAutoscale/queues/caches cope? recover?
Soak / Endurance70% load for 4–24 hLeaks, drift, log/disk growth (memory-leak-resource-testing)
Scalabilitystep 1×,2×,4× resourcesLinear? Where diminishing returns?
Capacitypeak × growth × headroomHow many nodes/DB size for next year?

Workload model (garbage in, garbage out)

Use production analytics: request mix by endpoint, think-time, session length, data-size distribution, peak RPS/concurrency, geographic spread, cache-hit ratios. Little's Law: concurrent users = arrival rate × avg session time; concurrency in service = RPS × latency. Use open model (arrival rate) for public traffic; closed model (fixed users) for internal systems.

k6 example: ramping arrival-rate + SLO thresholds

export const options = {
  scenarios: { peak: { executor: 'ramping-arrival-rate', startRate: 20, timeUnit: '1s',
    preAllocatedVUs: 200, maxVUs: 2000,
    stages: [{target: 100, duration: '5m'}, {target: 400, duration: '10m'}, {target: 0, duration: '2m'}]}},
  thresholds: { http_req_failed: ['rate<0.01'], http_req_duration: ['p(95)<500','p(99)<1200'] },
};
k6 run --out influxdb=http://influx:8086/k6 script.js
locust -f locustfile.py --headless -u 2000 -r 50 -t 30m --host https://stg

Find the bottleneck — USE & RED

  • USE per resource (CPU, memory, disk, network, DB conns, threads): Utilization, Saturation, Errors.
  • RED per service: Rate, Errors, Duration (p50/p95/p99, not average).
  • Follow the saturation: LB → app CPU/GC → thread/conn pools → DB (slow queries, locks, IOPS) → cache → downstream APIs → network/egress limits.
  • Correlate with traces (observability-testing); profile hot path (flame graphs: py-spy, async-profiler, pprof).

Autoscaling validation

Scale-out trigger & lag (metrics interval + provisioning + warm-up), scale-in safe (draining, connection reuse), min/max bounds, per-AZ balance, stateful components, DB/cache not scaling (usually the real limit), cost cap.

Report

Max sustainable throughput at SLO · latency curve (knee point) · first bottleneck + evidence · failure mode (graceful 429/503 vs cascading failure) · recommended capacity with 30–50% headroom · cost per 1k requests · retest plan.

Pitfalls

Load generator is the bottleneck (monitor it; distribute) · unrealistic data (all cache hits) · testing prod-different infra · ignoring downstream rate limits · coordinated omission (use tools that correct it) · averaging latencies.

performance-testing, chaos-resilience-testing, cloud-infrastructure-testing, web-performance-testing, observability-testing