What happens when you create a Pod ?

1. Pending Stage

What Happens:

API Server Registration:
- The kubectl command sends the pod specification to the API server.
- The API server validates the request against admission controllers (e.g., resource quotas, pod security policies).
- The pod object is stored in etcd, the cluster’s distributed key-value store.
Scheduling:
- The Kubernetes Scheduler evaluates all nodes in the cluster.
- It checks:
  - Resource Requests: Are there enough CPU, memory, and other resources available?
  - Constraints: Affinity/anti-affinity rules, taints, and tolerations.
  - Node Conditions: Node health, disk pressure, and network readiness.
- Once a suitable node is found, the pod is bound to that node.
Resource Allocation:
- Any required Persistent Volumes (PVs) are bound to Persistent Volume Claims (PVCs).
- The pod remains in the Pending state until:
  - A node is assigned.
  - All requested volumes are ready.

2. ContainerCreating Stage

What Happens:

Node Preparation:
- The kubelet (node agent) on the assigned node detects the new pod.
- It checks if the node has the required resources available.
Image Pulling:
- For each container in the pod, the kubelet pulls the specified container image from a container registry (e.g., Docker Hub, private registry).
- If the image is already available in the node’s local cache, it skips pulling.
Volume Mounting:
- Volumes (e.g., Persistent Volumes, ConfigMaps, Secrets) specified in the pod manifest are attached to the node and mounted into the container filesystem.
- If the pod uses dynamically provisioned volumes, the required storage class provisions the volume.
Networking Setup:
- The pod is assigned an IP address by the network plugin (CNI, e.g., Calico, Flannel).
- Network policies are enforced if configured.
Container Runtime Initialization:
- The kubelet communicates with the container runtime (e.g., containerd, CRI-O) to prepare the container.
- The runtime creates container namespaces for isolation (network, PID, etc.).
- Resource limits (CPU, memory) are applied as cgroups.

3. Running Stage

What Happens:

Container Start:
- The container runtime starts each container in the pod.
- The kubelet monitors the container status and ensures all containers are running.
Health Monitoring:
- Probes (if defined) check the container's health:
  - Liveness Probe: Determines if the container is still running. Restarts it if it fails.
  - Readiness Probe: Checks if the container is ready to serve requests.
  - Startup Probe: Ensures the container initializes properly before readiness/liveness probes begin.
Workload Execution:
- The pod now serves its intended purpose, such as processing requests, running a database, or executing jobs.

4. Succeeded Stage

What Happens:

Task Completion:
- For pods running one-off tasks (e.g., batch jobs), the containers complete their execution successfully (exit code 0).
Resource Cleanup:
- Kubernetes releases resources allocated to the pod (e.g., volumes, memory, CPU).
Pod Termination:
- The pod object remains in the cluster unless explicitly deleted.

5. Failed Stage

What Happens:

Task or Initialization Failure:
- One or more containers exit with a non-zero status.
- Common reasons include:
  - Incorrect application configuration.
  - Resource limits exceeded.
  - Missing dependencies (e.g., failed volume mounts).
Kubernetes Logging:
- The kubelet and container runtime log failure events.
- These logs help diagnose the root cause.
Retry (if Configured):
- If the pod is part of a Job or Deployment, it may be retried based on the specified restart policy.

6. CrashLoopBackOff (Special Case)

What Happens:

Repeated Container Failures:
- A container repeatedly crashes during startup or runtime.
- Causes may include:
  - Incorrect initialization.
  - External dependency failures (e.g., database connection issues).
Exponential Backoff:
- Kubernetes delays subsequent restarts to avoid overwhelming the system.
- Logs and events provide details about the failure.

7. Terminating Stage

What Happens:

Pod Deletion Request:
- A user or controller sends a request to delete the pod.
PreStop Hook Execution:
- If defined, the preStop hook runs to perform tasks like notifying other services of shutdown.
Signal to Containers:
- The kubelet sends a SIGTERM signal to containers, allowing them to shut down gracefully.
Grace Period:
- Containers are given a default of 30 seconds (configurable via terminationGracePeriodSeconds) to complete their tasks.
- If containers do not terminate within the grace period, a SIGKILL signal forcibly stops them.
Resource Cleanup:
- Volumes are detached and unmounted.
- The pod is removed from the node and the cluster.

8. Evicted (Special Case)

What Happens:

Node Resource Pressure:
- The kubelet detects low resources (CPU, memory, disk space) on the node.
Eviction Decision:
- Pods are prioritized for eviction based on their PriorityClass or Quality of Service (QoS).
- Lower-priority pods or pods with the BestEffort QoS class are evicted first.
Pod Termination:
- The pod is removed from the node but may be rescheduled to another node.

Summary

Pending: Scheduler assigns a node.
ContainerCreating: Image pulling, volume mounting, network setup.
Running: Containers start, probes monitor health.
Succeeded/Failed: Task completion or error handling.
Terminating: Graceful shutdown and resource cleanup.
Evicted: Pods removed due to resource constraints.

Each stage involves multiple system-level interactions between Kubernetes components (API server, kubelet, scheduler) and underlying infrastructure, ensuring the pod is deployed and managed efficiently.