software-tester-skills

CLI Testing

Testing command-line tools and scripts — exit codes, stdout/stderr contracts, flags and argument parsing, help/version output, stdin/pipes, env/config precedence, cross-platform paths, signals, idempotency, and Bats/pytest/expect-based automation.

A CLI's API is: args + env + stdin → stdout + stderr + exit code + side effects. Test that contract.

Contract checklist

AspectVerify
Exit codes0 success; non-zero on failure (documented: 1 general, 2 usage, 64–78 sysexits); consistent
StreamsData → stdout; diagnostics/progress → stderr; nothing extra on stdout (pipe-safe)
Help/version-h/--help exit 0, lists all flags, examples; --version machine-readable
ArgsUnknown flag → usage error (exit 2); missing required arg; -- separator; flag=value & flag value; short-flag bundling -abc; repeated flags; negative numbers
Inputstdin -; empty stdin; huge stdin; binary; CRLF; TTY vs pipe detection
Output modes--json valid & stable schema; --quiet/--verbose; color disabled when not TTY or NO_COLOR set
Config precedenceflag > env > project config > user config > defaults (test each layer)
FilesNonexistent, directory instead of file, no permission, symlink, spaces/unicode paths, ~ expansion, overwrite prompts/--force, atomic writes
IdempotencyRunning twice is safe; --dry-run changes nothing
SignalsSIGINT/SIGTERM → cleanup, partial files removed, exit 130/143; SIGPIPE handled (`head`)
InteractivityPrompts have non-interactive override (--yes); no hang in CI
Cross-platformWindows paths/CRLF/quoting, macOS BSD vs GNU coreutils, locale LANG=C
SecurityNo secrets in argv (visible in ps); shell-injection via args/filenames; tmp files 0600; safe deserialization
PerformanceStartup time; large input streaming (constant memory)

Tooling

# Bats (bash)
#!/usr/bin/env bats
@test "prints version" { run mycli --version; [ "$status" -eq 0 ]; [[ "$output" =~ ^mycli\ [0-9]+\.[0-9]+ ]]; }
@test "rejects unknown flag" { run mycli --nope; [ "$status" -eq 2 ]; [[ "$output" == *"Usage"* ]]; }
# pytest + subprocess
import subprocess, json
def run(*a, inp=None): return subprocess.run(["mycli",*a], input=inp, capture_output=True, text=True)
def test_json_mode():
    r = run("list","--json"); assert r.returncode == 0 and r.stderr == ""
    assert isinstance(json.loads(r.stdout), list)

Other: cram/prysk (transcript tests), expect/pexpect/node-pty for interactive TTY flows, snapshot-golden-testing for help text and reports, shellcheck + shfmt for shell scripts, Go testscript, Rust assert_cmd + predicates, Node execa.

Test the installed artifact

Package (npm/pip/brew/cargo/binary), install into a clean container, run smoke commands, verify shebang/permissions/completions (bash, zsh, fish), man page, --version matches tag.

unit-testing, integration-testing, snapshot-golden-testing, installation-upgrade-testing, fuzz-testing