software-tester-skills

Production Monitoring & Synthetic Testing (Shift-Right)

Testing in production safely (shift-right) — synthetic monitoring, canary checks, uptime probes, Playwright/Checkly/Datadog synthetics, real-user monitoring, SLO/SLI validation, alert testing, dark launches, traffic shadowing and observability-driven QA.

Pre-prod can't reproduce real data, traffic, third parties or geography. Continuously prove production works from the user's point of view, safely.

Synthetic monitor types

TypeFrequencyExample
Uptime/HTTP30–60 s, multi-regionGET /healthz 200 + body assertion, TLS expiry > 14 d
API transaction1–5 minlogin → create → read → delete with test tenant
Browser journey5–15 minPlaywright: search → add to cart → checkout (to payment sandbox stub)
DNS/CDN/cert5 minrecords, propagation, chain validity
Third-party dependency5 minpayment/email/auth provider reachability
Cron/heartbeatper jobdead-man's-switch (Healthchecks.io, Cronitor)

Tools: Checkly (Playwright-native), Datadog/New Relic/Grafana Synthetic, Pingdom, UptimeRobot, Better Stack, k6 Cloud browser, Sentry Crons, self-hosted Playwright in CI cron.

// Checkly/Playwright synthetic — runs every 10 min from 3 regions
import { test, expect } from '@playwright/test';
test('critical: user can search and reach checkout', async ({ page }) => {
  await page.goto('https://shop.example.com');
  await page.getByRole('searchbox').fill('headphones'); await page.keyboard.press('Enter');
  await page.getByRole('link', { name: /headphones/i }).first().click();
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await expect(page.getByTestId('cart-count')).toHaveText('1');
});

Safe testing in production

  • Dedicated test tenant/accounts, flagged so they're excluded from analytics, billing, emails, inventory, and search ranking.
  • Read-mostly journeys; writes use sandbox payment mode/void afterwards; auto-clean created data; idempotency keys.
  • No load tests against prod unless planned/throttled; rate-limit synthetics; exclude from SLO if you must, but include their pass/fail in SLI.
  • Dark launch/shadow traffic: mirror requests to new version, compare responses/latency without user impact (goreplay, service mesh mirroring).
  • Feature flags + canary (release-readiness-testing, configuration-feature-flag-testing).

SLIs, SLOs & alerts

Define SLIs from user experience: availability (good/total requests), latency (p95 < X), correctness/freshness. SLO e.g. 99.9%/30 d → error budget 43 min. Alert on burn rate (multi-window: 2%/1 h fast, 5%/6 h slow), not raw thresholds. Test your alerts: deliberately trigger (kill synthetic, inject error via flag) and confirm page arrives, routes to right team, links runbook, and clears on recovery; test on-call escalation quarterly.

RUM (real-user monitoring)

Web vitals by route/device/geo (web-performance-testing), JS errors (Sentry), API failures per release, funnel drop-offs; segment by version to detect regressions within minutes.

Feedback loop into QA

Every incident → (1) regression test, (2) new/tuned monitor, (3) runbook fix. Track: detection time, false-alarm rate, coverage of top-10 user journeys by synthetics.

observability-testing, smoke-sanity-testing, chaos-resilience-testing, release-readiness-testing, e2e-testing