Kubernetes Storage Mastery: Persistent Volumes, Claims, and StatefulSets

Introduction

Containers are ephemeral by nature - when they restart, all data is lost. For stateful applications like databases, this is unacceptable. Kubernetes provides a robust storage abstraction through Persistent Volumes (PV), Persistent Volume Claims (PVC), and Storage Classes to solve this challenge.

In this guide, I’ll show you how to implement persistent storage in Kubernetes, from basic concepts to production-ready StatefulSets.

Understanding Kubernetes Storage

The Problem:

# Without persistent storage - data lost on restart!
containers:
- name: mysql
  image: mysql:8.0
  # Data stored in container filesystem - EPHEMERAL!

The Solution:

Persistent Volumes (PV): Cluster-level storage resources
Persistent Volume Claims (PVC): User requests for storage
Storage Classes: Dynamic provisioning templates
StatefulSets: Manage stateful applications with stable storage

Storage Architecture

Application Pod
    ↓
PVC (Request)
    ↓
PV (Actual Storage)
    ↓
Physical Storage (NFS, Cloud Disk, etc.)

Part 1: Persistent Volumes (PV)

Persistent Volumes are cluster-wide resources that represent physical storage. They exist independently of Pods and have their own lifecycle.

Key Concepts:

Lifecycle: Independent of Pods - survives Pod deletion
Provisioning: Can be static (admin-created) or dynamic (auto-created)
Binding: One-to-one mapping with PVC
Phases: Available → Bound → Released → Failed

Creating a Persistent Volume (Static Provisioning):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    type: local
    environment: production
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: "/mnt/data"
    type: DirectoryOrCreate

Understanding Access Modes:

Access Mode	Abbreviation	Description	Use Case
ReadWriteOnce	RWO	Volume mounted read-write by single node	Databases, single-instance apps
ReadOnlyMany	ROX	Volume mounted read-only by many nodes	Shared configuration, static content
ReadWriteMany	RWX	Volume mounted read-write by many nodes	Shared file systems, collaborative apps
ReadWriteOncePod	RWOP	Volume mounted read-write by single pod	Kubernetes 1.22+

Important Notes:

Not all storage types support all access modes
RWX requires special storage (NFS, CephFS, GlusterFS)
Cloud providers have specific limitations

Reclaim Policies Explained:

1. Retain (Recommended for Production):

persistentVolumeReclaimPolicy: Retain

PV remains after PVC deletion
Data preserved for manual recovery
Admin must manually reclaim/delete
Use when: Data is critical, need backup before deletion

2. Delete:

persistentVolumeReclaimPolicy: Delete

PV and underlying storage deleted with PVC
Data permanently lost
Use when: Temporary data, development environments

3. Recycle (Deprecated):

Performs basic scrub (rm -rf /volume/*)
Use Delete with dynamic provisioning instead

Storage Backend Types:

HostPath (Development Only):

spec:
  hostPath:
    path: "/mnt/data"
    type: DirectoryOrCreate  # Creates if doesn't exist

Types: Directory, DirectoryOrCreate, File, FileOrCreate, Socket, CharDevice, BlockDevice

⚠️ Warning: HostPath is NOT recommended for production - data tied to specific node!

NFS (Network File System):

spec:
  nfs:
    server: 10.255.255.10
    path: /exported/path
    readOnly: false

Advantages:

Supports RWX (multiple pods, multiple nodes)
Simple setup
Works across nodes

Setup NFS Server (Ubuntu):

# On NFS Server
sudo apt-get install nfs-kernel-server
sudo mkdir -p /srv/nfs/kubedata
sudo chown nobody:nogroup /srv/nfs/kubedata
echo "/srv/nfs/kubedata *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports
sudo exportfs -a
sudo systemctl restart nfs-kernel-server

# On K8s Nodes
sudo apt-get install nfs-common

AWS EBS (Elastic Block Store):

spec:
  awsElasticBlockStore:
    volumeID: vol-0123456789abcdef0
    fsType: ext4
  accessModes:
    - ReadWriteOnce  # EBS only supports RWO

Create EBS Volume:

aws ec2 create-volume \
  --availability-zone us-west-2a \
  --size 10 \
  --volume-type gp3 \
  --encrypted

Azure Disk:

spec:
  azureDisk:
    diskName: myAKSDisk
    diskURI: /subscriptions/<subscription-id>/resourceGroups/<rg>/providers/Microsoft.Compute/disks/myAKSDisk
    kind: Managed
    cachingMode: ReadWrite
  accessModes:
    - ReadWriteOnce

GCE Persistent Disk:

spec:
  gcePersistentDisk:
    pdName: my-data-disk
    fsType: ext4
  accessModes:
    - ReadWriteOnce

Create GCE Disk:

gcloud compute disks create my-data-disk \
  --size=10GB \
  --zone=us-central1-a \
  --type=pd-ssd

iSCSI (Internet Small Computer Systems Interface):

spec:
  iscsi:
    targetPortal: 10.0.2.15:3260
    iqn: iqn.2001-04.com.example:storage.disk1
    lun: 0
    fsType: ext4
    readOnly: false

CephFS (Ceph File System):

spec:
  cephfs:
    monitors:
      - 10.16.154.78:6789
      - 10.16.154.82:6789
    user: admin
    secretRef:
      name: ceph-secret
    readOnly: false
  accessModes:
    - ReadWriteMany  # CephFS supports RWX

Volume Modes:

spec:
  volumeMode: Filesystem  # Default - mounts as directory
  # OR
  volumeMode: Block      # Raw block device

Block vs Filesystem:

Filesystem: Traditional file system (ext4, xfs)
Block: Raw block device (databases needing direct disk access)

Node Affinity for PV:

spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1
          - node2

Part 2: Persistent Volume Claims (PVC)

Persistent Volume Claims are requests for storage by users/applications. They abstract storage details from developers.

PVC Lifecycle:

Pending: Waiting for PV binding
Bound: Successfully bound to PV
Lost: PV deleted but PVC still exists

Creating a Basic PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: manual

PVC with Selector (Static Provisioning):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: manual
  selector:
    matchLabels:
      type: local
      environment: production
    matchExpressions:
    - key: tier
      operator: In
      values:
      - database

Selector Operators:

In: Label value in set
NotIn: Label value not in set
Exists: Label key exists
DoesNotExist: Label key doesn’t exist

PVC with Volume Mode:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block  # Raw block device
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd

Using PVC in Pod (Multiple Ways):

Method 1: Simple Mount:

apiVersion: v1
kind: Pod
metadata:
  name: mysql-pod
spec:
  containers:
  - name: mysql
    image: mysql:8.0
    env:
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mysql-secret
          key: password
    volumeMounts:
    - name: mysql-storage
      mountPath: /var/lib/mysql
  volumes:
  - name: mysql-storage
    persistentVolumeClaim:
      claimName: mysql-pvc

Method 2: With SubPath:

apiVersion: v1
kind: Pod
metadata:
  name: multi-app-pod
spec:
  containers:
  - name: app1
    image: nginx
    volumeMounts:
    - name: shared-storage
      mountPath: /usr/share/nginx/html
      subPath: app1  # Mounts /app1 subdirectory
  - name: app2
    image: nginx
    volumeMounts:
    - name: shared-storage
      mountPath: /usr/share/nginx/html
      subPath: app2  # Mounts /app2 subdirectory
  volumes:
  - name: shared-storage
    persistentVolumeClaim:
      claimName: shared-pvc

Method 3: Read-Only Mount:

apiVersion: v1
kind: Pod
metadata:
  name: readonly-pod
spec:
  containers:
  - name: app
    image: nginx
    volumeMounts:
    - name: config-storage
      mountPath: /etc/config
      readOnly: true  # Mount as read-only
  volumes:
  - name: config-storage
    persistentVolumeClaim:
      claimName: config-pvc

Method 4: Block Device:

apiVersion: v1
kind: Pod
metadata:
  name: block-pod
spec:
  containers:
  - name: app
    image: database-app
    volumeDevices:  # Use volumeDevices for block
    - name: block-storage
      devicePath: /dev/xvda
  volumes:
  - name: block-storage
    persistentVolumeClaim:
      claimName: block-pvc

PVC in Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx
        volumeMounts:
        - name: web-storage
          mountPath: /usr/share/nginx/html
      volumes:
      - name: web-storage
        persistentVolumeClaim:
          claimName: web-pvc

⚠️ Important: All replicas share the same PVC! Use StatefulSet for per-pod storage.

Expanding PVC (if supported by storage class):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: expandable-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi  # Original size
  storageClassName: expandable-storage

To expand:

kubectl edit pvc expandable-pvc
# Change storage: 10Gi to storage: 20Gi

Check expansion:

kubectl describe pvc expandable-pvc
kubectl get pvc expandable-pvc -o yaml

PVC Commands:

# Create PVC
kubectl apply -f pvc.yaml

# List PVCs
kubectl get pvc
kubectl get pvc -o wide
kubectl get pvc --all-namespaces

# Describe PVC
kubectl describe pvc mysql-pvc

# Check PVC status
kubectl get pvc mysql-pvc -o jsonpath='{.status.phase}'

# See bound PV
kubectl get pvc mysql-pvc -o jsonpath='{.spec.volumeName}'

# Delete PVC
kubectl delete pvc mysql-pvc

Part 3: Storage Classes

Storage Classes enable dynamic provisioning - PVs are automatically created when PVCs are created.

Key Benefits:

No manual PV creation needed
Automatic provisioning on-demand
Different storage tiers (fast SSD, slow HDD)
Cloud provider integration

Storage Class Components:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"  # Make default
provisioner: kubernetes.io/aws-ebs  # Storage provider
parameters:  # Provider-specific parameters
  type: gp3
  iopsPerGB: "10"
  encrypted: "true"
reclaimPolicy: Delete  # What happens when PVC deleted
allowVolumeExpansion: true  # Allow resizing
volumeBindingMode: WaitForFirstConsumer  # When to provision
mountOptions:  # Mount options
  - debug

Volume Binding Modes:

1. Immediate (Default):

volumeBindingMode: Immediate

PV created immediately when PVC created
May create PV in wrong zone
Use when: Single-zone cluster

2. WaitForFirstConsumer (Recommended):

volumeBindingMode: WaitForFirstConsumer

PV created when Pod using PVC is scheduled
Ensures PV in same zone as Pod
Use when: Multi-zone cluster, topology-aware

Cloud Provider Storage Classes:

AWS EBS:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: aws-ebs-gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
  kmsKeyId: arn:aws:kms:us-east-1:123456789:key/xxx
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

EBS Volume Types:

gp3: General Purpose SSD (recommended)
gp2: General Purpose SSD (older)
io1/io2: Provisioned IOPS SSD
st1: Throughput Optimized HDD
sc1: Cold HDD

Azure Disk:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-disk-premium
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_LRS
  kind: Managed
  cachingMode: ReadOnly
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Azure SKUs:

Standard_LRS: Standard HDD
StandardSSD_LRS: Standard SSD
Premium_LRS: Premium SSD
UltraSSD_LRS: Ultra SSD

GCE Persistent Disk:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gce-pd-ssd
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-ssd
  replication-type: regional-pd
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

GCE Disk Types:

pd-standard: Standard persistent disk
pd-balanced: Balanced persistent disk
pd-ssd: SSD persistent disk
pd-extreme: Extreme persistent disk

NFS Dynamic Provisioning:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs-server.example.com
  share: /exported/path
mountOptions:
  - hard
  - nfsvers=4.1
reclaimPolicy: Retain
volumeBindingMode: Immediate

Local Storage:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Using Storage Class in PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-storage  # Reference storage class
  resources:
    requests:
      storage: 10Gi

Default Storage Class:

# Set as default
kubectl patch storageclass fast-storage \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

# Remove default
kubectl patch storageclass fast-storage \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

# List storage classes
kubectl get storageclass
kubectl get sc  # Short form

PVC without storageClassName uses default:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: default-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  # No storageClassName - uses default

Multiple Storage Classes Example:

# Fast SSD for databases
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "5000"
allowVolumeExpansion: true
---
# Standard for general use
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
allowVolumeExpansion: true
---
# Archive for backups
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: archive
provisioner: ebs.csi.aws.com
parameters:
  type: sc1  # Cold HDD
reclaimPolicy: Retain

Storage Class Commands:

# List storage classes
kubectl get storageclass
kubectl get sc

# Describe storage class
kubectl describe sc fast-storage

# Get default storage class
kubectl get sc -o jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}'

# Delete storage class
kubectl delete sc fast-storage

Part 4: StatefulSets - Managing Stateful Applications with Persistent Storage

What are StatefulSets? StatefulSets are Kubernetes controllers designed specifically for stateful applications that require:

Stable, unique network identifiers (predictable pod names)
Stable, persistent storage (each pod gets its own PVC)
Ordered, graceful deployment and scaling (pods created/deleted in sequence)
Ordered, automated rolling updates

Why Use StatefulSets? Unlike Deployments where all pods are identical and interchangeable, StatefulSets are used when:

Your application needs persistent identity (databases, message queues)
Pods need to maintain state across restarts
Order matters for startup/shutdown (leader election, distributed systems)
Each pod needs its own dedicated storage

StatefulSet vs Deployment:

Feature	StatefulSet	Deployment
Pod Names	Predictable (mysql-0, mysql-1)	Random (mysql-7d8f9-x8k2p)
Storage	Unique PVC per pod	Shared PVC for all pods
Scaling	Ordered (0→1→2)	Parallel
Network Identity	Stable DNS	Changes on restart
Use Case	Databases, queues	Stateless apps

Common Use Cases:

Databases: MySQL, PostgreSQL, MongoDB, Cassandra
Message Queues: Kafka, RabbitMQ, NATS
Distributed Systems: Elasticsearch, Zookeeper, etcd
Caching: Redis Cluster, Memcached

How StatefulSets Work:

Ordered Creation: Pods created sequentially (0, then 1, then 2…)
Stable Names: Each pod gets predictable name: <statefulset-name>-<ordinal>
Persistent Storage: Each pod gets its own PVC from volumeClaimTemplates
Stable Network: Each pod gets stable DNS: <pod-name>.<service-name>.<namespace>.svc.cluster.local
Ordered Deletion: Pods deleted in reverse order (2, then 1, then 0)

Complete StatefulSet Example with Explanations:

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
  - port: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 3
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-storage
      resources:
        requests:
          storage: 10Gi

StatefulSet Features:

Stable pod names: mysql-0, mysql-1, mysql-2
Ordered deployment and scaling
Stable network identities
Persistent storage per pod

Production Example: MySQL with Persistent Storage

Complete setup:

# Secret for MySQL password
apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
type: Opaque
stringData:
  password: MySecurePassword123!
---
# Storage Class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mysql-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Retain
---
# StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        volumeMounts:
        - name: mysql-data
          mountPath: /var/lib/mysql
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
  volumeClaimTemplates:
  - metadata:
      name: mysql-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: mysql-storage
      resources:
        requests:
          storage: 20Gi
---
# Service
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  clusterIP: None
  selector:
    app: mysql
  ports:
  - port: 3306

Storage Management Commands

# PV commands
kubectl get pv
kubectl describe pv mysql-pv
kubectl delete pv mysql-pv

# PVC commands
kubectl get pvc
kubectl describe pvc mysql-pvc
kubectl delete pvc mysql-pvc

# Storage Class commands
kubectl get storageclass
kubectl describe storageclass fast-storage

# StatefulSet commands
kubectl get statefulsets
kubectl scale statefulset mysql --replicas=3
kubectl delete statefulset mysql

Best Practices

1. Use Storage Classes for Dynamic Provisioning 2. Set Appropriate Reclaim Policies 3. Implement Backup Strategies 4. Monitor Storage Usage 5. Use StatefulSets for Stateful Apps 6. Set Resource Limits 7. Test Disaster Recovery

Troubleshooting Storage Issues - Common Problems and Solutions

Storage issues can prevent applications from starting or cause data loss. Here’s how to diagnose and fix common problems:

Problem 1: PVC Stuck in Pending State

What it means: PVC cannot find a suitable PV to bind to.

Symptoms:

kubectl get pvc
NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
mysql-pvc   Pending                                      manual

Diagnosis:

kubectl describe pvc mysql-pvc
# Look for events at the bottom

Common Causes & Solutions:

Cause 1: No matching PV available

# Check if any PVs exist
kubectl get pv

# Solution: Create a PV or use dynamic provisioning
kubectl apply -f pv.yaml

Cause 2: Insufficient storage

# PVC requests 10Gi but PV only has 5Gi
# Solution: Create PV with sufficient capacity or reduce PVC request

Cause 3: Access mode mismatch

# PVC requests ReadWriteMany but PV only supports ReadWriteOnce
# Solution: Match access modes or use appropriate storage type (NFS for RWX)

Cause 4: Storage class mismatch

# PVC specifies storageClassName: "fast" but no such class exists
# Solution: Create storage class or use existing one
kubectl get storageclass

Cause 5: Selector doesn’t match any PV

# PVC has selector that no PV satisfies
# Solution: Adjust PVC selector or add matching labels to PV

Problem 2: Pod Can’t Mount Volume

What it means: Pod created but volume mount fails, pod stuck in ContainerCreating.

Symptoms:

kubectl get pods
NAME      READY   STATUS              RESTARTS   AGE
mysql-0   0/1     ContainerCreating   0          5m

Diagnosis:

kubectl describe pod mysql-0
# Look for mount errors in Events section

kubectl get events --sort-by='.lastTimestamp'

Common Causes & Solutions:

Cause 1: PVC not bound

# Check PVC status
kubectl get pvc
# Solution: Fix PVC binding issue first (see Problem 1)

Cause 2: Volume already mounted on another node (RWO)

# RWO volumes can only mount on one node
# If pod moves to different node, volume must detach first
# Solution: Wait for detachment or delete old pod
kubectl delete pod <old-pod> --force --grace-period=0

Cause 3: NFS server unreachable

# Check NFS server connectivity from node
ssh <node>
showmount -e <nfs-server-ip>
# Solution: Fix network/firewall, ensure NFS server running

Cause 4: Permission issues

# Volume mounted but app can't write
# Solution: Check fsGroup, runAsUser in pod security context
spec:
  securityContext:
    fsGroup: 999  # MySQL group
    runAsUser: 999  # MySQL user

Cause 5: Node missing required packages

# For NFS: nfs-common not installed
# Solution: Install on all nodes
sudo apt-get install nfs-common

Problem 3: Data Loss After Pod Restart

What it means: Pod restarts but previous data is gone.

Diagnosis:

# Check if PVC is actually being used
kubectl describe pod <pod-name> | grep -A 5 Volumes
kubectl describe pod <pod-name> | grep -A 5 Mounts

Common Causes & Solutions:

Cause 1: Volume not mounted

# Pod spec missing volume mount
# Solution: Add volumeMounts and volumes sections

Cause 2: Wrong mount path

# Mounted to /data but app writes to /var/lib/mysql
# Solution: Correct mountPath in volumeMounts

Cause 3: Using emptyDir instead of PVC

# emptyDir is ephemeral
# Solution: Replace with PVC
volumes:
- name: data
  persistentVolumeClaim:
    claimName: mysql-pvc

Cause 4: PV reclaim policy is Delete

# PV deleted when PVC deleted
# Solution: Use Retain policy for important data
persistentVolumeReclaimPolicy: Retain

Problem 4: Storage Full

What it means: Volume has no space left.

Diagnosis:

# Check from inside pod
kubectl exec -it <pod-name> -- df -h

# Check PVC size
kubectl get pvc

Solutions:

Solution 1: Expand PVC (if supported)

# Check if storage class allows expansion
kubectl get sc <storage-class> -o yaml | grep allowVolumeExpansion

# Expand PVC
kubectl edit pvc <pvc-name>
# Increase storage size

Solution 2: Clean up data

kubectl exec -it <pod-name> -- bash
# Remove old logs, temporary files

Solution 3: Create new larger PVC

# Backup data, create new PVC, restore data

Problem 5: Slow Storage Performance

What it means: Application slow due to storage I/O bottleneck.

Diagnosis:

# Check I/O from inside pod
kubectl exec -it <pod-name> -- bash
dd if=/dev/zero of=/data/testfile bs=1M count=1000

Solutions:

Solution 1: Use faster storage class

# Switch from HDD to SSD
storageClassName: fast-ssd  # gp3, Premium_LRS, pd-ssd

Solution 2: Increase IOPS (cloud providers)

# AWS EBS example
parameters:
  type: gp3
  iops: "5000"  # Increase IOPS

Solution 3: Use local storage for temporary data

# Use emptyDir for cache/temp
volumes:
- name: cache
  emptyDir: {}

Problem 6: StatefulSet Pod Won’t Start

What it means: StatefulSet pod stuck, won’t create next pod.

Diagnosis:

kubectl get statefulset
kubectl describe statefulset <name>
kubectl get pods -l app=<app-name>

Common Causes & Solutions:

Cause 1: Previous pod not ready

# StatefulSets create pods sequentially
# Pod 0 must be ready before pod 1 starts
# Solution: Fix pod 0 issues first

Cause 2: PVC creation failed

# Check PVCs created by volumeClaimTemplates
kubectl get pvc
# Solution: Fix storage class or provisioner issues

Cause 3: Headless service missing

# StatefulSets require headless service (clusterIP: None)
kubectl get svc
# Solution: Create headless service

Useful Debugging Commands

# Check all storage resources
kubectl get pv,pvc,sc

# Watch PVC status
kubectl get pvc -w

# Check storage events
kubectl get events --field-selector involvedObject.kind=PersistentVolumeClaim

# Check node storage
kubectl describe node <node-name> | grep -A 10 "Allocated resources"

# Force delete stuck PVC
kubectl patch pvc <pvc-name> -p '{"metadata":{"finalizers":null}}'
kubectl delete pvc <pvc-name> --force --grace-period=0

# Check volume attachment
kubectl get volumeattachment

Conclusion

Kubernetes storage provides robust solutions for stateful applications. Key takeaways:

Use PVCs for storage requests
Implement Storage Classes for dynamic provisioning
Use StatefulSets for stateful applications
Plan backup strategies carefully
Monitor storage usage proactively

Next: Advanced Scheduling with Node Affinity and Taints

Introduction#

Understanding Kubernetes Storage#

Storage Architecture#

Part 1: Persistent Volumes (PV)#

Part 2: Persistent Volume Claims (PVC)#

Part 3: Storage Classes#

Part 4: StatefulSets - Managing Stateful Applications with Persistent Storage#

Production Example: MySQL with Persistent Storage#

Storage Management Commands#

Best Practices#

Troubleshooting Storage Issues - Common Problems and Solutions#

Problem 1: PVC Stuck in Pending State#

Problem 2: Pod Can’t Mount Volume#

Problem 3: Data Loss After Pod Restart#

Problem 4: Storage Full#

Problem 5: Slow Storage Performance#

Problem 6: StatefulSet Pod Won’t Start#

Useful Debugging Commands#

Conclusion#

Resources#

Introduction

Understanding Kubernetes Storage

Storage Architecture

Part 1: Persistent Volumes (PV)

Part 2: Persistent Volume Claims (PVC)

Part 3: Storage Classes

Part 4: StatefulSets - Managing Stateful Applications with Persistent Storage

Production Example: MySQL with Persistent Storage

Storage Management Commands

Best Practices

Troubleshooting Storage Issues - Common Problems and Solutions

Problem 1: PVC Stuck in Pending State

Problem 2: Pod Can’t Mount Volume

Problem 3: Data Loss After Pod Restart

Problem 4: Storage Full

Problem 5: Slow Storage Performance

Problem 6: StatefulSet Pod Won’t Start

Useful Debugging Commands

Conclusion

Resources