software-tester-skills

CI/CD Testing

Designing or debugging test execution in CI/CD pipelines — PR gates, test tiering (smoke/regression/nightly), parallelization/sharding, quality gates, flaky-test tracking, and failure notification wiring. Covers GitHub Actions, GitLab CI, Jenkins and similar.

Test tiering (don't run everything on every PR)

  • PR gate (minutes): unit tests + fast API contract checks + a thin smoke E2E suite covering the money path. This blocks merge — keep it fast and reliable or people start bypassing it.
  • Pre-deploy / merge-to-main: fuller regression suite, still bounded (10-20 min).
  • Nightly / scheduled: full regression, cross-browser matrix, performance baseline, security scan (SAST/DAST) — anything too slow to gate a PR.
  • Post-deploy smoke: a small suite that runs against the just-deployed environment before traffic fully shifts, to catch environment-specific breakage config/secrets differences that staging didn't have.

Quality gates

Define explicit, machine-checkable pass/fail criteria per stage, not "look at the dashboard":

  • Merge gate: 0 failing required checks, coverage delta not regressed below threshold (if coverage-gating is used — see the caution about coverage-as-a-metric in qa-fundamentals).
  • Deploy gate: regression suite green, no new Critical/Blocker security findings (security-testing), perf within budget (performance-testing).
  • Make gates fail loud and specific — a gate that just says "checks failed" without linking the failing job wastes the next person's time.

Parallelization & sharding

  • Split large suites across N workers by file/test-duration balancing (not just alphabetical file split, which produces uneven shard times) — most modern runners (Playwright, pytest-xdist) support this natively.
  • Keep tests independent (no shared mutable fixture/test-order dependency) — sharding surfaces hidden ordering dependencies as flaky failures, which is actually useful for finding them.
  • Cache dependencies (node_modules, pip packages, browser binaries) between runs to cut pipeline time; invalidate cache correctly on lockfile change.

Flaky test handling

  • Don't auto-retry-until-green and call it done — track a flakiness rate per test (fails X% of runs over N runs) and treat crossing a threshold as its own bug to investigate, per test-automation's flaky-test triage.
  • Quarantine a known-flaky test (mark it non-blocking, keep it running, alert on its pattern) rather than deleting it or leaving it silently blocking merges.
  • Track flaky-test rate as a first-class pipeline health metric — a suite trending flakier is losing signal even if it's still "green."

Artifact & evidence collection

  • On failure, always retain: screenshots, video/trace (Playwright trace: 'retain-on-failure'), console logs, network HAR, the exact commit SHA and environment. Without this, a CI failure is unreproducible days later.
  • Set explicit retention windows — indefinite artifact storage silently becomes an infra cost problem.

Notifications & bug creation

  • Route failures to the owning team/channel automatically (based on changed files or test tags), not a firehose channel everyone mutes.
  • Auto-create/update a tracking ticket on a new regression-suite failure (not on every transient CI blip) — link the failing run, commit range, and evidence.
  • De-duplicate: if the same test has an open ticket, comment/bump it rather than opening a new one every run — see qa-code-intelligence for duplicate-detection approach.

Platform notes

  • GitHub Actions: matrix builds for cross-browser/OS; actions/cache; required status checks on branch protection; reusable workflows to avoid duplicating pipeline logic across repos.
  • GitLab CI: rules:/only:changes: to scope jobs to relevant file changes; DAG (needs:) to parallelize instead of purely sequential stages.
  • Jenkins: declarative pipelines over scripted for readability/maintainability; shared libraries for common QA steps across multiple pipelines.

Reporting

When a pipeline change itself introduces a regression (e.g. a gate that stopped actually blocking), treat it with the same severity discipline as a product bug — a broken quality gate is a Critical finding because it silently removes a safety net. See bug-reporting.