Rolling Out NHI Discovery and Governance Without Breaking Production

A phased approach for platform engineering and security teams

Non-human identities don't fail gracefully. There's no login prompt, no "your session expired" message — just a hard crash, a broken CI/CD pipeline, a silent data pipeline going dark, or a pager alert at 2 a.m. That's what keeps high-risk legacy credentials alive indefinitely in most environments, and it's why rolling out NHI discovery and governance has to be staged carefully.

Most engineering organizations treat rotating a database password or an SSH key as routine: a low-stakes, scheduled task that is well understood. Rotating an unmapped, multi-cloud service account key is a different proposition. The asymmetry between the low cost of leaving a stale credential alone and the high cost of touching it and getting it wrong is exactly what this guide is structured around.

The four phases below — low-impact discovery, ownership and context mapping, a usage-delta audit, and staged enforcement — describe the actual technical mechanisms involved, regardless of whether they're assembled by a platform team using existing cloud-native tooling or automated by a dedicated NHI platform. The mechanism is the same either way; what differs is how much of it has already been built.

Phase 1: Low-Impact Discovery

The first principle of safe discovery is event-driven ingestion, not active polling. Repeatedly sweeping a cloud environment's APIs to enumerate every identity and credential — especially on a tight interval — risks API throttling from the cloud provider, which can disrupt active deployment pipelines that share the same rate limits. The alternative is to consume the audit trail each provider already produces, asynchronously, and react only to the events that matter: identity creation, credential issuance, and privilege changes.

Each major cloud provider has its own audit trail and its own vocabulary for these events — and getting the vocabulary right matters, both for building accurate detection rules and for not confusing teams who'll recognize the API names on sight.

AWS: CloudTrail and EventBridge

AWS identity-generation activity flows through CloudTrail, which can be aggregated into a centralized S3 bucket across an organization's accounts. An EventBridge rule can then trigger a near-real-time notification — to a discovery system, a ticketing queue, or a SIEM — whenever specific events occur. The events worth filtering for:

• CreateUser, CreateRole — new IAM identities
• CreateAccessKey — new long-lived credentials for an IAM user
• CreateLoginProfile — console access enabled for a user
• AttachUserPolicy, AttachRolePolicy, PutRolePolicy — privilege grants to an identity
• CreateServiceLinkedRole — AWS-managed service identities
• CreateKeyPair — EC2 SSH key pair generation

AssumeRole is worth tracking separately. It's extremely high-volume in normal operation, so it's better handled with filters for unusual patterns — cross-account assumption, assumption by a principal that's never assumed that role before — than as a blanket alert.

GCP: Cloud Audit Logs and Pub/Sub

GCP's equivalent pattern routes Cloud Audit Logs through a sink to Pub/Sub, which a discovery system subscribes to. The key events:

• google.iam.admin.v1.CreateServiceAccount — GCP's identity-creation event; this is the GCP analog of AWS's CreateUser/CreateRole, not an AWS API
• google.iam.admin.v1.CreateServiceAccountKey — credential issuance for an existing service account, analogous to AWS's CreateAccessKey
• SetIamPolicy — often more important than account creation itself, since this is where privilege actually gets attached to an identity that may already exist

Azure: Entra ID Audit Logs and Event Hub

Azure Monitor Diagnostic Settings can stream Administrative and Security log categories to an Event Hub for asynchronous consumption. The events to watch:

• "Add application" / "Add service principal" — identity creation via app registration, Azure's primary model for non-human identities
• "Add password to application" (Microsoft Graph addPassword) — client secret issuance for an application; this is Azure's analog to AWS's CreateAccessKey, not an AWS event
• "Add service principal credentials" — the certificate-based equivalent of the above

Exact event-name strings can shift slightly across API versions, so these should be checked against current provider documentation before being encoded into detection rules. The category-level mapping — which provider owns "service account" as a concept, which owns "app registration plus client secret" — is the durable part.

Code Repository Scanning: Webhook-Based Diff Scanning

Hardcoded secrets are best caught at the point of introduction rather than through periodic full-history scans, which are resource-intensive and easy to schedule around peak development hours without thinking about coverage gaps. A webhook configured at the organization level on a Git provider fires on every push event, sending a payload containing just the diff — the changed lines — to a scanning system.

The scan itself looks for two things: high-entropy strings (a heuristic for randomly-generated secrets) and structural patterns matching known credential formats — github_pat_ prefixes, AWS access key patterns (AKIA[0-9A-Z]{16}), and similar provider-specific formats. Scanning only the diff keeps this lightweight enough to run on every push without becoming a bottleneck in the development workflow.

Phase 2: Ownership and Context Mapping

A list of thousands of discovered credentials is not actionable on its own. Before anything gets touched, the question that matters is: who owns this, and what depends on it? When resource tags don't answer that — which is most of the time — two mechanisms can reconstruct the answer from data that already exists.

Git Blame Lineage Trace

When a credential or identity reference is found in an application's configuration or infrastructure-as-code, its origin can often be traced directly through version control history:

# Locate the commit, author, and timestamp where this identity was introduced
git log -S "service-account-prod" --pretty=format:"%h - %an, %ae : %ad" -n 1

The author's email can then be cross-referenced against an identity provider (Okta, Entra ID) to determine team membership. If that team maps to an active engineering group — say, Engineering-DataPlatform — ownership of the credential can be routed to that team for review. This can run as a one-off script during initial triage, or be wired into an automated pipeline that runs whenever a new credential is discovered; the underlying logic is the same either way.

Container Runtime Extraction (Kubernetes)

For secrets injected dynamically into running containers, the orchestration layer's API is a more reliable source than guessing based on image or deployment names. Querying the Kubernetes API server for a pod's specification reveals exactly how a credential is wired in:

spec:
  containers:
  - name: payment-processor
    env:
    - name: DB_SECRET
      valueFrom:
        secretKeyRef:
          name: payment-db-credentials
          key: password

If that pod carries a label like app.kubernetes.io/part-of: billing-service, the credential's operational context — and therefore its risk profile if compromised — can be associated with the billing team. Whether that association lands in a spreadsheet, a ticketing system, or an NHI inventory platform, the extraction mechanism is the same.

Phase 3: The 30-to-90 Day Usage Delta Audit

Least-privilege enforcement requires knowing what an identity actually does, not just what it's permitted to do — and many identities have legitimate but infrequent uses (end-of-month reconciliation jobs, quarterly backup processes) that a short observation window would miss entirely. A 90-day window is long enough to capture most of these cycles.

Building the Usage Delta Matrix

For each identity, comparing assigned permissions against actually-invoked actions over the observation window typically surfaces a meaningful gap:

This example, for an identity like analytics-sync-svc, is a common pattern: an identity that's clearly active and necessary for its core function, but that's also been granted administrative-tier permissions it has never exercised — often inherited from a broad managed policy applied for convenience at creation time.

Shadow Policies and Dry-Run Validation

The unused permissions shouldn't be stripped immediately — a 90-day window doesn't guarantee every legitimate use case has occurred. Instead, the right-sized permission set can be saved as a shadow policy: a proposed replacement that isn't yet enforced.

For a further validation period — commonly two weeks — every API call the identity makes is checked against the shadow policy without being blocked. If a call would have been denied under the shadow policy but is actually occurring in production, that's a signal the policy is too narrow — the policy is updated to permit it, and the validation window restarts. Once a full validation period passes with no such conflicts, the shadow policy can be promoted to the identity's actual permission set with much higher confidence that nothing will break.

Phase 4: Enforcement Without Downtime

Once monitoring has identified specific risks, the transition to active enforcement is where the asymmetry from the introduction becomes concrete. Two patterns make this transition safer.

The Conditional Deny Pattern

For an orphaned identity — one whose creator has left and whose continued necessity is unclear, but which is still making active API calls — outright deletion is risky: if something still depends on it, deletion breaks it with no warning.

A more controlled approach is a scoped Deny policy targeting only the highest-risk actions available to that identity (for example, iam:CreateUser, s3:DeleteBucket, or other actions inconsistent with the identity's apparent function) while leaving its other permissions intact.

The equivalent pattern at an API gateway layer (Kong, MuleSoft) is a proxy-level rule that blocks specific routes or methods for a given API key while logging the attempts, providing the same kind of narrow, observable signal before a full key revocation.

Staged JIT Secret Migration

For legacy applications relying on static credentials in .env files or environment variables, migrating to short-lived, dynamically-issued credentials is a multi-stage process:

• 1
  Provision the ephemeral secret engine — infrastructure stage

  Configure a secrets management system (HashiCorp Vault, AWS Secrets Manager, or equivalent) to dynamically generate short-lived database credentials — a max TTL of 60 minutes is a common starting point.

• 2
  Inject a sidecar agent — deployment stage

  Add a sidecar container to the application's Kubernetes deployment that authenticates to the secrets engine using the pod's native service account token, and mounts the dynamically-generated credentials into an in-memory volume.

• 3
  Update application code — code refactor stage

  Modify the application's connection logic to read credentials from the mounted in-memory path rather than parsing environment variables, and implement a hot-reload so the application picks up rotated credentials when a connection drops rather than requiring a restart.

• 4
  Decommission the static credential — cleanup stage

  Once metrics confirm the application is consistently using the dynamic credentials, the original static credential can be deleted from wherever it was stored.

This is the architecture regardless of whether a platform automates the sidecar injection and rotation orchestration, or a platform team wires it up directly with Vault and Kubernetes primitives — the difference is in how much of steps 1–3 comes pre-built versus assembled in-house.

Implementation Risk Reference

Even read-only ingestion isn't literally zero-risk — there's monitoring overhead and the audit data itself is sensitive — but Phases 1 and 2 are low-risk relative to Phases 3 and 4, where actual policy changes and credential migrations occur. The risk profile climbs sharply at Phase 4, which is the rationale for the staged, observable approach throughout the earlier phases: by the time enforcement happens, most of the uncertainty about what will break has already been resolved.

Not sure which phase is most relevant to your environment yet? The risk self-check takes nine questions about how credentials are created, tracked, and managed today and points to the phase above worth starting with. The access-review work in Phase 3 also maps directly onto several controls covered in the audit and compliance mapping guide, including SOC 2 CC6.1–6.3 and NIST 800-53 AC-2.