Deep Dive: Vault Kubernetes Auth Method

This is part of the Vault + Kubernetes Integration Guide. Return to the main guide for the full architecture overview.

The Kubernetes Auth Method is the recommended way for pods to authenticate with Vault. It uses native Kubernetes Service Account tokens (JWTs), eliminating the need to distribute or rotate static credentials.

But before jumping into commands, let’s understand what each component is, why it exists, and how they work together.

Understanding the Components

The Kubernetes auth flow involves five key components working together. Let’s break down each one.

1. Service Account (SA)

A Service Account is a Kubernetes identity for processes running inside pods. Unlike user accounts (for humans), service accounts are for workloads.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: webapp-sa
  namespace: webapp-ns

What it does in the Vault flow:

• Acts as the identity of your pod. When your pod presents itself to Vault, it’s saying: “I am webapp-sa in namespace webapp-ns.”
• Every pod in Kubernetes runs under a service account. If you don’t specify one, it runs as default.
• Vault uses the SA name and namespace to determine what secrets the pod is allowed to access.

Key detail: Starting from Kubernetes 1.24, service accounts no longer automatically get a long-lived token secret. Tokens are now projected and time-bound via the TokenRequest API.

2. Service Account JWT Token

Every pod in Kubernetes gets a JWT (JSON Web Token) automatically mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. This is a cryptographically signed token issued by the Kubernetes API server.

What’s inside the JWT:

{
  "iss": "https://kubernetes.default.svc.cluster.local",
  "sub": "system:serviceaccount:webapp-ns:webapp-sa",
  "aud": ["https://kubernetes.default.svc.cluster.local"],
  "exp": 1717200000,
  "iat": 1717113600,
  "kubernetes.io": {
    "namespace": "webapp-ns",
    "serviceaccount": {
      "name": "webapp-sa",
      "uid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
    },
    "pod": {
      "name": "webapp-7d8f9b6c5-x2k9p",
      "uid": "f9e8d7c6-b5a4-3210-fedc-ba0987654321"
    }
  }
}

Key JWT fields explained:

What it does in the Vault flow:

• This JWT is the credential your pod presents to Vault. It proves the pod’s identity.
• The pod sends this JWT to Vault’s /auth/kubernetes/login endpoint.
• Vault doesn’t blindly trust this JWT — it validates it by calling the TokenReview API.

3. TokenReview API

The TokenReview API is a Kubernetes API endpoint (/apis/authentication.k8s.io/v1/tokenreviews) that lets external systems verify if a JWT token is valid and who it belongs to.

How it works:

┌─────────────┐                     ┌──────────────────┐
│   Vault     │  POST /tokenreviews │   K8s API Server  │
│             │────────────────────►│                    │
│  "Is this   │                     │  Checks:           │
│   JWT       │                     │  1. Signature valid?│
│   valid?"   │                     │  2. Token expired?  │
│             │◄────────────────────│  3. SA still exists?│
│             │  Response:           │                    │
│             │  authenticated: true│                    │
│             │  user: webapp-sa    │                    │
└─────────────┘                     └──────────────────┘

What Vault sends to TokenReview:

{
  "apiVersion": "authentication.k8s.io/v1",
  "kind": "TokenReview",
  "spec": {
    "token": "<the pod's JWT>"
  }
}

What K8s returns:

{
  "status": {
    "authenticated": true,
    "user": {
      "username": "system:serviceaccount:webapp-ns:webapp-sa",
      "uid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "groups": [
        "system:serviceaccounts",
        "system:serviceaccounts:webapp-ns"
      ]
    }
  }
}

Why this matters:

• Vault never decodes or validates the JWT itself. It delegates validation entirely to the Kubernetes API server.
• This means Vault doesn’t need the K8s signing keys — it just needs network access to the API server.
• This is also why Vault needs a service account with system:auth-delegator permissions — it needs to be able to call the TokenReview API.

4. Token Reviewer Service Account

This is a separate service account specifically for Vault to authenticate itself to the Kubernetes API when making TokenReview calls. It is NOT the same as the application’s service account.

┌──────────────────────────────────────────────────────────────┐
│  Two different Service Accounts with two different purposes: │
│                                                              │
│  1. vault-auth SA  ──► Used by VAULT to call TokenReview API │
│     (reviewer)         Needs: system:auth-delegator role     │
│                                                              │
│  2. webapp-sa      ──► Used by YOUR APP to identify itself   │
│     (workload)         Needs: no special K8s permissions     │
└──────────────────────────────────────────────────────────────┘

Why it needs system:auth-delegator:

The system:auth-delegator ClusterRole grants permission to:

• Create TokenReview objects (validate tokens)
• Create SubjectAccessReview objects (check permissions)

Without this binding, Vault would get a 403 Forbidden when trying to validate your pod’s JWT.

# What the ClusterRoleBinding looks like internally
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator  # Built-in K8s role
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: vault

5. Vault Auth Role

A Vault auth role is the bridge between Kubernetes identity and Vault permissions. It defines:

• WHO can authenticate (which SAs from which namespaces)
• WHAT they get access to (which Vault policies)
• HOW LONG the access lasts (token TTL)

┌───────────────────────────────────────────────────────────┐
│                    Vault Auth Role                         │
│                                                           │
│  Name: webapp                                             │
│                                                           │
│  WHO:  bound_service_account_names = ["webapp-sa"]        │
│        bound_service_account_namespaces = ["webapp-ns"]   │
│                                                           │
│  WHAT: policies = ["webapp-policy"]                       │
│        → webapp-policy grants read on secret/data/webapp/*│
│                                                           │
│  HOW LONG: ttl = 1h, max_ttl = 4h                        │
│                                                           │
│  If SA=webapp-sa AND namespace=webapp-ns → ALLOW          │
│  Issue a Vault token with webapp-policy attached          │
└───────────────────────────────────────────────────────────┘

The Complete Authentication Flow

Now let’s put it all together — here’s exactly what happens when your pod needs a secret from Vault:

┌──────────────────────────────────────────────────────────────────────────┐
│                          COMPLETE AUTH FLOW                              │
│                                                                          │
│  ┌─────────┐  Step 1: Read JWT from   ┌────────────────────────────┐    │
│  │ App Pod │  /var/run/secrets/...     │ SA Token (JWT)             │    │
│  │         │◄──────────────────────────│ sub: webapp-sa             │    │
│  │         │                           │ ns:  webapp-ns             │    │
│  └────┬────┘                           └────────────────────────────┘    │
│       │                                                                  │
│       │ Step 2: POST /v1/auth/kubernetes/login                          │
│       │         body: { role: "webapp", jwt: "<token>" }                │
│       ▼                                                                  │
│  ┌──────────┐                                                           │
│  │  Vault   │  Step 3: Vault extracts the JWT and calls TokenReview     │
│  │  Server  │──────────────────────────────────────────────┐            │
│  │          │                                               │            │
│  │          │  Step 6: Vault checks:                        │            │
│  │          │  • Is "webapp-sa" in role's                   │            │
│  │          │    bound_service_account_names? ✅            ▼            │
│  │          │  • Is "webapp-ns" in role's        ┌──────────────────┐   │
│  │          │    bound_service_account_namespaces?│  K8s API Server  │   │
│  │          │    ✅                               │                  │   │
│  │          │  • All checks pass →                │  Step 4: Validate│   │
│  │          │    Issue Vault token with           │  - Signature ✅  │   │
│  │          │    webapp-policy                    │  - Expiry ✅     │   │
│  │          │                                    │  - SA exists ✅  │   │
│  │          │◄───────────────────────────────────│                  │   │
│  └────┬─────┘  Step 5: Returns authenticated=true│  Step 5: Return  │   │
│       │                 user=webapp-sa            │  TokenReview     │   │
│       │                                          └──────────────────┘   │
│       │ Step 7: Returns Vault token                                     │
│       │         (hvs.CAESI..., policies=["webapp-policy"])              │
│       ▼                                                                  │
│  ┌─────────┐  Step 8: Uses Vault token to read                         │
│  │ App Pod │  GET /v1/secret/data/webapp/config                        │
│  │         │────────────────────────────────────►  Vault returns secret │
│  └─────────┘                                                            │
└──────────────────────────────────────────────────────────────────────────┘

Step-by-step summary:

1. Pod reads its JWT from the projected volume at /var/run/secrets/kubernetes.io/serviceaccount/token
2. Pod (or Agent) sends the JWT to Vault’s login endpoint along with the desired role name
3. Vault forwards the JWT to the Kubernetes TokenReview API using its reviewer SA credentials
4. K8s API validates the token — checks the cryptographic signature, expiration, and that the SA still exists
5. K8s API returns the validation result with the SA identity (name, namespace, UID, groups)
6. Vault checks the identity against the role’s bound_service_account_names and bound_service_account_namespaces
7. If all checks pass, Vault issues a short-lived Vault token with the policies from the matching role
8. Pod uses the Vault token to read secrets from the paths allowed by the attached policy

How K8s 1.24+ Changed Things (Bound Service Account Tokens)

Before K8s 1.24, every service account automatically got a long-lived, non-expiring token stored in a Secret. This was a security risk — if the token leaked, it was valid forever.

K8s 1.24+ (BoundServiceAccountTokenVolume):

Impact on Vault integration:

• The Token Reviewer SA (vault-auth) still needs a long-lived token, so we explicitly create a kubernetes.io/service-account-token Secret for it
• The workload SA (webapp-sa) uses the projected token — which is short-lived and more secure
• Vault must handle the issuer field correctly, since bound tokens have a specific issuer claim

Prerequisites

• Vault server (v1.14+) — inside or outside the cluster
• Kubernetes cluster (v1.24+ recommended for bound service account tokens)
• kubectl access with cluster-admin privileges
• vault CLI configured to talk to your Vault instance

Scenario 1: Vault Running Inside the Cluster

Step 1: Deploy Vault via Helm

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Production HA deployment
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set server.ha.enabled=true \
  --set server.ha.replicas=3 \
  --set server.ha.raft.enabled=true \
  --set injector.enabled=true \
  --set server.dataStorage.size=10Gi \
  --set server.auditStorage.enabled=true \
  --set server.auditStorage.size=10Gi

What this creates:

Step 2: Initialize and Unseal Vault

# Initialize (first time only)
kubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=5 -key-threshold=3 -format=json > vault-init.json

# Unseal (repeat for each replica)
UNSEAL_KEY_1=$(jq -r '.unseal_keys_b64[0]' vault-init.json)
UNSEAL_KEY_2=$(jq -r '.unseal_keys_b64[1]' vault-init.json)
UNSEAL_KEY_3=$(jq -r '.unseal_keys_b64[2]' vault-init.json)

for pod in vault-0 vault-1 vault-2; do
  kubectl exec -n vault $pod -- vault operator unseal $UNSEAL_KEY_1
  kubectl exec -n vault $pod -- vault operator unseal $UNSEAL_KEY_2
  kubectl exec -n vault $pod -- vault operator unseal $UNSEAL_KEY_3
done

# Set the root token for initial configuration
export VAULT_TOKEN=$(jq -r '.root_token' vault-init.json)

Why unsealing is needed: Vault encrypts all data at rest using an encryption key, which is itself encrypted by a master key. The master key is split into key shares using Shamir’s Secret Sharing. You need a threshold number of shares to reconstruct the master key and “unseal” Vault. In production, use auto-unseal with a cloud KMS instead.

Step 3: Create the Token Reviewer Service Account

This SA is only for Vault to call the TokenReview API. It has nothing to do with your application.

# Create the service account
kubectl create serviceaccount vault-auth -n vault

# Create a long-lived token secret (required for K8s 1.24+)
# The projected tokens are short-lived and bound to pods,
# but Vault needs a persistent token to call TokenReview
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-token
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
EOF

# Grant token review permissions
# system:auth-delegator allows creating TokenReview and
# SubjectAccessReview resources
kubectl create clusterrolebinding vault-tokenreview-binding \
  --clusterrole=system:auth-delegator \
  --serviceaccount=vault:vault-auth

Why a long-lived token? Vault stores this token in its config and uses it every time a pod tries to authenticate. If we used a projected token (which expires in 1 hour), Vault would lose the ability to validate tokens after it expires. The explicit kubernetes.io/service-account-token Secret creates a non-expiring token that persists.

Step 4: Enable and Configure Kubernetes Auth

# Port-forward to Vault (if not using ingress)
kubectl port-forward -n vault svc/vault 8200:8200 &

export VAULT_ADDR="http://127.0.0.1:8200"

# Extract the reviewer SA's token and the cluster CA certificate
SA_TOKEN=$(kubectl get secret vault-auth-token -n vault \
  -o jsonpath='{.data.token}' | base64 -d)

SA_CA_CRT=$(kubectl get secret vault-auth-token -n vault \
  -o jsonpath='{.data.ca\.crt}' | base64 -d)

# For in-cluster Vault, use the internal API server address
K8S_HOST="https://kubernetes.default.svc:443"

# Enable the auth method
vault auth enable kubernetes

# Configure it with the three pieces of information Vault needs:
vault write auth/kubernetes/config \
  token_reviewer_jwt="$SA_TOKEN" \
  kubernetes_host="$K8S_HOST" \
  kubernetes_ca_cert="$SA_CA_CRT" \
  issuer="https://kubernetes.default.svc.cluster.local"

What each config parameter does:

Scenario 2: Vault Running Outside the Cluster (External)

When Vault runs on an external VM or as a managed service, the key difference is networking — Vault must be able to reach the Kubernetes API server to call TokenReview.

Network Requirements

┌────────────────────┐                 ┌──────────────────────┐
│   External Vault   │                 │   Kubernetes Cluster  │
│   vault.corp.com   │                 │                       │
│                    │  TokenReview    │  ┌─────────────────┐  │
│   Must be able to  │────────────────►│  │  K8s API Server │  │
│   reach K8s API    │  (HTTPS/6443)   │  │  :6443          │  │
│                    │                 │  └─────────────────┘  │
│                    │                 │                       │
│                    │◄───────────────►│  ┌─────────────────┐  │
│                    │  Auth requests  │  │  Application    │  │
│                    │  (HTTPS/8200)   │  │  Pods           │  │
└────────────────────┘                 │  └─────────────────┘  │
                                       └──────────────────────┘

Two network paths required:

1. Pods → Vault (HTTPS/8200): For authentication and secret reads
2. Vault → K8s API (HTTPS/6443): For TokenReview validation

If Vault cannot reach the K8s API, use AppRole Auth instead.

Step 1: Extract Cluster Credentials

# Get the API server's EXTERNAL endpoint (not the in-cluster one)
K8S_HOST=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

# Create the reviewer SA and token (same as in-cluster)
kubectl create serviceaccount vault-auth -n vault
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-token
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
EOF

kubectl create clusterrolebinding vault-tokenreview-binding \
  --clusterrole=system:auth-delegator \
  --serviceaccount=vault:vault-auth

SA_TOKEN=$(kubectl get secret vault-auth-token -n vault \
  -o jsonpath='{.data.token}' | base64 -d)

SA_CA_CRT=$(kubectl get secret vault-auth-token -n vault \
  -o jsonpath='{.data.ca\.crt}' | base64 -d)

Step 2: Configure Vault (from Vault server)

export VAULT_ADDR="https://vault.corp.com:8200"

# Use a unique mount path per cluster
vault auth enable -path=kubernetes-prod kubernetes

vault write auth/kubernetes-prod/config \
  token_reviewer_jwt="$SA_TOKEN" \
  kubernetes_host="$K8S_HOST" \
  kubernetes_ca_cert="$SA_CA_CRT"

Multi-cluster tip: Use unique auth mount paths per cluster (e.g., kubernetes-prod, kubernetes-staging) so a single Vault instance can serve multiple clusters. Each mount gets its own reviewer JWT and K8s host, pointing to a different cluster’s API server.

Creating Policies and Roles

This section applies to both topologies.

Vault Policies — What Can Be Accessed

A Vault policy defines which paths a token can access and what operations it can perform:

vault policy write webapp-policy - <<EOF
# KV v2 read access — note the "data/" prefix for KV v2
path "secret/data/webapp/*" {
  capabilities = ["read", "list"]
}

# Metadata access (for listing secret keys)
path "secret/metadata/webapp/*" {
  capabilities = ["read", "list"]
}

# Dynamic database credentials
path "database/creds/webapp-readonly" {
  capabilities = ["read"]
}

# PKI certificate generation
path "pki/issue/webapp-cert" {
  capabilities = ["create", "update"]
}
EOF

Capabilities explained:

Vault Auth Roles — Who Can Authenticate

vault write auth/kubernetes/role/webapp \
  bound_service_account_names=webapp-sa \
  bound_service_account_namespaces=webapp-ns \
  policies=webapp-policy \
  audience=vault \
  ttl=1h \
  max_ttl=4h

Role parameters explained:

Create the Workload Service Account

kubectl create namespace webapp-ns
kubectl create serviceaccount webapp-sa -n webapp-ns

This SA doesn’t need any special Kubernetes RBAC. Its sole purpose is to identify the pod to Vault.

Testing the Authentication

Verify your setup works before deploying any application:

# Generate a short-lived SA token (simulates what the pod would use)
SA_JWT=$(kubectl create token webapp-sa -n webapp-ns --duration=600s)

# Authenticate against Vault
vault write auth/kubernetes/login \
  role=webapp \
  jwt="$SA_JWT"

# You should see a successful login with the correct policies:
# Key                  Value
# token                hvs.CAESI...
# token_policies       ["default" "webapp-policy"]

Production Hardening

Use Service Account UIDs

Bind roles to the SA’s UID — not just the name. If someone deletes and recreates a SA with the same name, the old binding won’t work:

SA_UID=$(kubectl get sa webapp-sa -n webapp-ns -o jsonpath='{.metadata.uid}')

vault write auth/kubernetes/role/webapp \
  bound_service_account_names=webapp-sa \
  bound_service_account_namespaces=webapp-ns \
  alias_name_source=serviceaccount_uid \
  policies=webapp-policy \
  ttl=1h

Auto-Unseal with Cloud KMS

Never manually unseal in production:

# values-production.yaml for Helm
server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
  extraEnvironmentVars:
    VAULT_SEAL_TYPE: awskms
    AWS_REGION: us-east-1
    VAULT_AWSKMS_SEAL_KEY_ID: "mrk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Audit Logging

vault audit enable file file_path=/vault/audit/vault-audit.log
vault audit enable syslog tag="vault" facility="AUTH"

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vault-access
  namespace: vault
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: vault
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              vault-access: "true"
      ports:
        - port: 8200
          protocol: TCP
  policyTypes:
    - Ingress

Label namespaces that need Vault access:

kubectl label namespace webapp-ns vault-access=true

Common Errors and Fixes

Next: Learn how to deliver these secrets to your pods with the Vault Agent Injector Guide or the Vault Secrets Operator Guide.