Why 'No Errors Found' Can Mean Your Tooling Couldn't Check

Midway through investigating a cluster of recurring pod errors, the log sweeps started coming back clean. Every kubectl logs / kubectl get call against the known-problem pods returned nothing. For a few minutes, that looked like good news.

It wasn’t. It was this:

Unable to connect to the server: remote error: tls: expired certificate

The credential had a 16-minute lifetime

The cluster used a short-lived admin credential mechanism — client certificates minted with a deliberately tight validity window, regenerated on demand rather than stored long-term. Good practice for reducing standing access. But the credential had expired over a week earlier, and nothing about a kubectl command returning empty output distinguished “genuinely zero matching log lines” from “couldn’t authenticate, therefore zero results by default.”

kubectl itself failed loudly and unambiguously — the TLS error above is about as clear as errors get. The problem was one layer up: a log-sweep script that greps command output and reports what it finds treats “the command produced no matching lines” and “the command produced no output at all” identically, if nothing checks which one happened.

Two states, not one

Any automated check that reports “no issues found” is making an implicit claim: the check actually ran. That claim needs to be verified, not assumed. A cheap guard — confirm the tool connected before trusting its output — turns a silent false-clean into a loud, immediate failure:

kubectl get pods -n target-namespace > /dev/null || {
  echo "auth/connectivity check failed — skipping sweep, not reporting clean" >&2
  exit 1
}

It’s a few lines, and it changes the failure mode from “confidently wrong” to “loudly blocked,” which is always the better of the two. The second one gets fixed immediately. The first one gets discovered days later, if at all — usually by someone re-running the same investigation and wondering why it’s back.

The generalizable rule

Short-lived credentials, ephemeral tokens, and just-in-time access are good security defaults. They also mean your tooling’s assumptions about “the check ran and found nothing” get invalidated more often than a long-lived credential would. If a script’s success path and its can’t-even-try path produce the same visible output, that’s a gap worth closing before you trust it during an incident — which is exactly the moment you’re least likely to notice.