Managing secrets in Kubernetes is one of those challenges that every platform team eventually faces. Base64-encoded Kubernetes Secrets stored in etcd are not encryption — they’re encoding. If you’re running anything beyond a hobby cluster, you need a proper secrets management solution.

This guide is your one-stop reference for HashiCorp Vault + Kubernetes. It covers the architecture, compares every integration method, and links to detailed implementation guides for each approach.

How Vault + Kubernetes Auth Actually Works

Before looking at any implementation, you need to understand the components involved and how they interact. This section covers the fundamentals that every integration method builds upon.

The Key Components

┌─────────────────────────────────────────────────────────────────────────┐
│                                                                         │
│  ┌─────────────────┐   ┌───────────────────┐   ┌────────────────────┐  │
│  │ Service Account  │   │  SA JWT Token     │   │  TokenReview API   │  │
│  │ (K8s identity)   │   │  (the credential) │   │  (the validator)   │  │
│  │                  │   │                   │   │                    │  │
│  │ Identifies your  │   │ Cryptographically │   │ K8s API endpoint   │  │
│  │ pod — like a     │   │ signed proof of   │   │ that verifies if   │  │
│  │ username for     │   │ identity, mounted │   │ a JWT is valid and │  │
│  │ workloads        │   │ in every pod at   │   │ who it belongs to  │  │
│  │                  │   │ /var/run/secrets/ │   │                    │  │
│  └─────────────────┘   └───────────────────┘   └────────────────────┘  │
│                                                                         │
│  ┌─────────────────┐   ┌───────────────────┐   ┌────────────────────┐  │
│  │ Token Reviewer   │   │  Vault Auth Role  │   │  Vault Policy      │  │
│  │ SA               │   │  (the gatekeeper) │   │  (the permissions) │  │
│  │                  │   │                   │   │                    │  │
│  │ A separate SA    │   │ Maps K8s identity │   │ Defines which      │  │
│  │ that Vault uses  │   │ to Vault access — │   │ secret paths can   │  │
│  │ to call the      │   │ "SA X from NS Y   │   │ be read/written    │  │
│  │ TokenReview API  │   │  gets policy Z"   │   │ by the token       │  │
│  └─────────────────┘   └───────────────────┘   └────────────────────┘  │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

The Authentication Flow (Simplified)

Every Vault + Kubernetes integration follows this core flow:

Pod (webapp-sa)                    Vault                     K8s API Server
     │                               │                            │
     │  1. "Here's my JWT,           │                            │
     │     I want role 'webapp'"     │                            │
     │──────────────────────────────►│                            │
     │                               │  2. "K8s, is this         │
     │                               │     JWT valid?"           │
     │                               │───────────────────────────►│
     │                               │                            │
     │                               │  3. "Yes, it belongs to   │
     │                               │     webapp-sa in webapp-ns"│
     │                               │◄───────────────────────────│
     │                               │                            │
     │                               │  4. Checks: is webapp-sa   │
     │                               │     allowed role 'webapp'? │
     │                               │     → YES, issue token     │
     │                               │                            │
     │  5. Vault token returned      │                            │
     │     (policies: webapp-policy)  │                            │
     │◄──────────────────────────────│                            │
     │                               │                            │
     │  6. "Give me secret at        │                            │
     │     secret/data/webapp/config" │                            │
     │──────────────────────────────►│                            │
     │                               │                            │
     │  7. Secret data returned      │                            │
     │◄──────────────────────────────│                            │

What’s happening at each step:

  1. The pod’s JWT (mounted at /var/run/secrets/kubernetes.io/serviceaccount/token) is sent to Vault
  2. Vault doesn’t decode the JWT itself — it forwards it to the K8s TokenReview API for validation
  3. K8s API checks the signature, expiration, and SA existence, then returns the identity
  4. Vault matches the identity against the role’s bound_service_account_names and bound_service_account_namespaces
  5. If matched, Vault issues a short-lived token with the role’s policies
  6. The pod (or agent) uses this Vault token to read secrets
  7. Vault checks the token’s policies and returns the secret if allowed

Deep dive: For a complete breakdown of each component (JWT internals, TokenReview payloads, RBAC bindings, K8s 1.24+ changes), see the Kubernetes Auth Method Guide.

Architecture Overview

Topology 1: Vault Inside the Cluster

┌─────────────────────────────────────────────────────┐
│                  Kubernetes Cluster                  │
│                                                     │
│  ┌─────────────┐    TokenReview    ┌──────────────┐ │
│  │  Vault       │◄────────────────►│  K8s API     │ │
│  │  (StatefulSet)│                  │  Server      │ │
│  └──────┬───────┘                  └──────────────┘ │
│         │                                           │
│    ┌────▼────┐   ┌──────────┐   ┌───────────────┐  │
│    │ Agent   │   │ CSI      │   │ Secrets       │  │
│    │ Injector│   │ Provider │   │ Operator(VSO) │  │
│    └────┬────┘   └────┬─────┘   └──────┬────────┘  │
│         │             │                │            │
│    ┌────▼─────────────▼────────────────▼────────┐  │
│    │            Application Pods                 │  │
│    └─────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────┘

Best for: Single-cluster setups, dev/staging, simplified networking.

Topology 2: Vault Outside the Cluster (External)

┌──────────────────┐         ┌─────────────────────────────────┐
│  External Vault  │         │       Kubernetes Cluster         │
│  (VM / Managed)  │         │                                  │
│                  │◄──TLS──►│  ┌──────────────┐               │
│  vault.corp.com  │         │  │  K8s API     │               │
│  :8200           │         │  │  Server      │               │
│                  │         │  └──────────────┘               │
│                  │         │                                  │
│                  │◄──TLS──►│  ┌──────────┐  ┌────────────┐  │
│                  │         │  │  Agent   │  │  App Pods   │  │
│                  │         │  │  Injector│──│             │  │
│                  │         │  └──────────┘  └────────────┘  │
└──────────────────┘         └─────────────────────────────────┘

Best for: Multi-cluster, production, centralized secrets management.

Complete Integration Matrix

Auth Methods

Auth Method How It Works Vault Needs K8s API Access? Static Creds Required? Best For
Kubernetes Auth Pod’s SA JWT validated via TokenReview API ✅ Yes ❌ No In-cluster or API-reachable Vault
AppRole Auth RoleID + SecretID pair ❌ No ✅ Yes (SecretID) External Vault, CI/CD pipelines
JWT/OIDC Auth OIDC-compliant JWT tokens ❌ No ❌ No Multi-cloud, federated identity
Token Auth Direct Vault token ❌ No ✅ Yes (token) Dev/testing only

Secret Delivery Methods

Method Architecture Resource Overhead Dynamic Secrets Rotation Secrets in etcd Templating GitOps
Agent Injector Sidecar per pod 🔴 High ✅ Full ✅ Automatic ❌ No (tmpfs) ✅ Advanced ⚠️ Annotations
CSI Provider DaemonSet per node 🟢 Low ❌ Limited ❌ Pod restart ❌ No ❌ Basic ⚠️ Volume config
Secrets Operator (VSO) Single controller 🟢 Lowest ✅ Full ✅ Auto + restart ✅ Yes ✅ Transformations ✅ CRDs
ArgoCD Vault Plugin Sync-time resolution 🟢 None ❌ Static ❌ Re-sync ✅ Yes ✅ Placeholders ✅ Native GitOps
External Secrets Operator Controller + CRDs 🟢 Low ⚠️ Partial ✅ Refresh interval ✅ Yes ⚠️ Basic ✅ CRDs

Decision Matrix: Which Combination to Use?

Scenario Auth Method Delivery Method
Simple in-cluster, legacy apps Kubernetes Auth Agent Injector
In-cluster, GitOps with ArgoCD Kubernetes Auth ArgoCD Vault Plugin
In-cluster, modern K8s-native Kubernetes Auth Vault Secrets Operator
In-cluster, minimal overhead Kubernetes Auth CSI Provider
External Vault, API reachable Kubernetes Auth Agent Injector or VSO
External Vault, API unreachable AppRole External Secrets Operator
CI/CD pipelines AppRole Direct API or Agent
Multi-cluster, single Vault Kubernetes Auth (per cluster) VSO or Agent Injector
High-security, no etcd secrets Kubernetes Auth Agent Injector or CSI
Dynamic DB credentials needed Kubernetes Auth Agent Injector or VSO

Detailed Implementation Guides

Each guide below is a complete, step-by-step walkthrough with working manifests:

Authentication Methods

  • 📖 Kubernetes Auth Method — SA token validation, in-cluster and external Vault setup, RBAC, production hardening
  • 📖 AppRole Auth Method — RoleID/SecretID setup, Agent sidecar config, ESO integration, SecretID rotation

Secret Delivery Methods

  • 📖 Vault Agent Injector — Sidecar injection, annotation reference, templating examples, dynamic secrets, resource tuning
  • 📖 Vault CSI Provider — Ephemeral volumes, SecretProviderClass, K8s Secret syncing
  • 📖 Vault Secrets Operator (VSO) — CRD-based secrets (Static, Dynamic, PKI), auto-rotation, rollout restarts, transformations

GitOps Integration

  • 📖 ArgoCD Vault Plugin (AVP) — Placeholder syntax, Helm/Kustomize support, multi-environment setup, secret refresh strategies

Quick Start: Which Path Should You Take?

                    ┌──────────────────┐
                    │ Can Vault reach  │
                    │ K8s API server?  │
                    └────────┬─────────┘
                      Yes ┌──┴──┐ No
                          │     │
                    ┌─────▼──┐ ┌▼──────────┐
                    │  Use   │ │   Use      │
                    │  K8s   │ │  AppRole   │
                    │  Auth  │ │   Auth     │
                    └───┬────┘ └─────┬──────┘
                        │            │
                    ┌───▼────────────▼───┐
                    │ How to deliver     │
                    │ secrets to pods?   │
                    └────────┬──────────┘
              ┌──────────────┼──────────────┐
              │              │              │
      ┌───────▼──────┐ ┌────▼─────┐ ┌──────▼──────┐
      │ Need dynamic │ │ Want     │ │ Using       │
      │ secrets or   │ │ K8s-     │ │ ArgoCD      │
      │ templating?  │ │ native?  │ │ for GitOps? │
      │              │ │          │ │             │
      │ Agent        │ │ VSO or   │ │ AVP         │
      │ Injector     │ │ CSI      │ │ Plugin      │
      └──────────────┘ └──────────┘ └─────────────┘

Production Hardening Checklist

Vault Server

  • Deploy in HA mode (3+ replicas with Raft or Consul backend)
  • Enable auto-unseal via cloud KMS (AWS KMS / GCP CKMS / Azure Key Vault)
  • Enable audit logging and ship to centralized logging
  • Enforce TLS everywhere — no plaintext Vault traffic
  • Use namespaces (Enterprise) for multi-tenant isolation

Authentication

  • Prefer Kubernetes Auth over AppRole wherever possible
  • Use Service Account UIDs in role bindings for stronger identity
  • Set short TTLs (1h token, 4h max) — force frequent renewal
  • Bind roles to specific namespaces and service accounts — no wildcards

Secret Management

  • Use dynamic secrets (database, AWS STS) instead of static KV where possible
  • Enable secret versioning (KV v2) for rollback capability
  • Implement lease revocation procedures for incident response
  • Use VSO rolloutRestartTargets to auto-restart pods on secret rotation

Network Security

  • Restrict Vault API access with NetworkPolicies
  • For external Vault: use private endpoints or VPN — never expose to internet
  • Configure Vault listener to bind only to required interfaces

Troubleshooting Quick Reference

Problem Likely Cause Quick Fix
permission denied Wrong SA/namespace in role Check bound_service_account_names and namespaces
Sidecar not injecting Webhook missing or namespace excluded kubectl get mutatingwebhookconfigurations | grep vault
Pod stuck in Init Agent can’t reach Vault Check network + kubectl logs <pod> -c vault-agent-init
Secret file empty Wrong path or missing policy Verify path in Vault: vault kv get secret/app/config
VSO not syncing VaultAuth misconfigured Check operator logs + VaultAuth status
AVP placeholder not resolved Missing annotation or wrong path Verify avp.kubernetes.io/path annotation

Wrapping Up

The right Vault + Kubernetes integration depends on your topology, security requirements, and operational model. Use the decision matrix and detailed guides above to pick the combination that fits.

The key principle: never store real secrets in Git or etcd. Use Vault as the single source of truth and let the integration layer handle delivery.