CI/CD & DevOps — from commit to production, deeply

1First principles: what problem does CI/CD solve?

Integration risk grows superlinearly with batch size. Merge two weeks of changes at once and you get combinatorial conflicts plus a huge haystack when something breaks. CI/CD is the discipline of shrinking batches:

• CI (Continuous Integration): merge to main frequently; every merge is automatically built + tested, so breakage is detected within minutes of introduction — while the diff is small and the author still has context.
• CD (Continuous Delivery): every green build is deployable — packaging, config, migrations all automated; a human pushes the button. Continuous Deployment removes the button.

2Pipeline anatomy

A canonical backend pipeline, stage by stage:

push/PR
  → lint + format + typecheck     # seconds; fail fastest on the cheapest checks
  → unit tests                    # fast, parallelized
  → build artifact (docker image) # build ONCE; same bytes flow to every env
  → integration tests             # against real containerized deps (Postgres, Kafka)
  → security scans                # dep audit, image CVEs (Trivy), secret detection
  → push image to registry        # tagged with git SHA — immutable, traceable
  → deploy → staging              # automatic
  → smoke tests / e2e on staging
  → deploy → prod                 # gated (approval) or automatic
  → post-deploy verification      # health checks, error-rate watch, auto-rollback

Principles to articulate:

• Fail fast, cheapest first. Lint before tests before builds — most feedback in least time.
• Build once, promote everywhere. Never rebuild per environment — rebuilds can differ (new base image, new transitive dep). Promote the same image digest from staging to prod. Config differs per env; the artifact doesn't. (This is 12-factor build/release/run — §6.)
• Immutable, traceable artifacts. Tag images with the git SHA, not latest — "what exactly is running in prod?" must have a one-word answer.
• Pipeline speed is a feature. Over 10–15 min to green and people batch changes — exactly the anti-pattern CI fights. Tools: dependency caching, test parallelization/sharding, path filters (don't run the full suite for a README change), tiered suites (fast per-commit, full pre-merge/nightly).
• Deployment strategies: rolling (replace pods gradually — K8s default, zero infra cost), blue-green (two full environments, switch traffic atomically, instant rollback, 2x cost), canary (route 1→5→25→100% of traffic while watching error rates — safest for high-traffic services).

3GitHub Actions — the concrete tool

Model: a workflow (YAML in .github/workflows/) is triggered by events (push, pull_request, schedule, workflow_dispatch) and runs jobs (parallel by default, each on a fresh runner VM; sequence with needs:); each job runs steps — shell commands or reusable actions.

name: ci
on:
  pull_request:
  push: { branches: [main] }

jobs:
  test:
    runs-on: ubuntu-latest
    services:                    # sidecar containers for integration tests
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: test }
        ports: ["5432:5432"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12", cache: "pip" }  # dep caching
      - run: pip install -e ".[dev]"
      - run: ruff check . && ruff format --check .
      - run: pytest -x -q --cov
  build-and-push:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write }  # least-privilege token
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: ghcr.io/org/app:${{ github.sha }}
          cache-from: type=gha       # layer cache across runs
          cache-to: type=gha,mode=max

Details that signal real use:

• Secrets via ${{ secrets.X }} — encrypted, masked in logs, scoped per repo/environment. Better still: OIDC federation to cloud providers — the runner exchanges a short-lived identity token for cloud credentials, so no long-lived cloud keys sit in GitHub at all. Name-dropping OIDC here is high-value.
• Matrix builds (strategy: matrix: python: [3.11, 3.12]) — one job definition, N parallel variants.
• Environments with required reviewers = the manual gate before prod.
• Concurrency groups (concurrency: { group: deploy-prod, cancel-in-progress: false }) — serialize deploys; cancel superseded PR runs to save minutes.

4Docker, deeply

The core idea

A container is not a VM. It's a normal Linux process, isolated by kernel features:

• Namespaces — the process gets its own view of PIDs, network, mounts, hostname, users.
• cgroups — CPU/memory limits.
• Union filesystem — its own filesystem assembled from image layers.

No guest OS, no hypervisor → millisecond starts, near-zero overhead. All containers share the host kernel — which is also the isolation caveat vs VMs (one sentence of security awareness).

What Docker actually solved: "works on my machine." The image packages app + runtime + system deps + filesystem into one immutable, content-addressed artifact that runs identically on a laptop, CI, and prod — dev/prod parity as an artifact property.

Images and layers

An image is an ordered stack of read-only layers; each Dockerfile instruction that changes the filesystem creates one. Layers are content-addressed and shared — 10 images on python:3.12-slim store the base once. At runtime the container adds a thin writable layer on top (copy-on-write) — which is why container-local writes are ephemeral and disposable.

FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["uvicorn", "app.main:app"]

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["uvicorn", "app.main:app"]

Multi-stage builds

Build-time tooling (compilers, headers, dev deps) shouldn't ship to prod — size, attack surface, CVE noise. Multi-stage: build in a fat stage, COPY --from= only the outputs into a slim final stage.

FROM python:3.12 AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --prefix=/install -r requirements.txt  # needs gcc etc. — stays here

FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /install /usr/local
COPY app/ app/
RUN useradd -m appuser
USER appuser          # non-root: container escape ≠ root on host
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Hygiene checklist to rattle off: slim/distroless base; .dockerignore (.git, .env, caches — build-context bloat and secret leaks); non-root USER; exec-form CMD (JSON array — your process is PID 1 and receives SIGTERM directly, which is what makes K8s graceful shutdown work); never ARG/ENV secrets into an image — they persist in layer history (docker history shows them); use BuildKit --mount=type=secret for build-time credentials; pin image digests for reproducibility.

5Kubernetes — core objects and why each exists

K8s in one sentence: a declarative reconciliation engine — you write desired state into the API server; controllers run a loop comparing desired vs actual and act to converge. Everything below is "what desired state can I declare?"

Ingress routing, concretely:

rules:
- host: api.example.com
  http:
    paths:
    - path: /
      backend: { service: { name: api, port: { number: 80 } } }

The glue that makes it all work: probes

• Liveness — "restart me if this fails" (hung process).
• Readiness — "don't send me traffic yet/now" (warming up, dependency down). Readiness gating is what makes rolling deploys zero-downtime: new pods receive traffic only when actually ready.
• Startup — patience for slow-booting apps before liveness kicks in.

612-Factor apps — the ones that matter

Twelve-factor is the why behind container/K8s design. The load-bearing factors, compressed:

7Environment & secret management

• Config hierarchy: code defaults → env-specific config → env vars/secrets at runtime. Pydantic BaseSettings is the idiomatic FastAPI answer — typed, validated config from env, fails fast at startup on missing values.
• Secrets lifecycle: never in git (enforce with pre-commit hooks + CI secret scanning — gitleaks/trufflehog). A secret once committed is compromised forever via history — rotate it, don't just delete the line. Runtime delivery: platform secret stores (hardened K8s Secrets, or Vault/cloud managers with dynamic short-lived credentials).
• Rotation must be a non-event — support two valid credentials simultaneously so you roll without downtime (same pattern as dual API keys).
• Identity over secrets where possible: workload identity / IAM roles / OIDC — the platform attests "this pod is service X" and mints short-lived credentials; nothing long-lived to leak. The direction the industry is moving; one sentence earns points.
• Don't log secrets: redaction in logging middleware; masked in CI logs; and — an LLM-era addition you can own — keep secrets out of prompt payloads and LLM traces. Observability tooling that records full prompts will happily capture a connection string a naive RAG pipeline stuffed into context.

8Infrastructure as Code — intuition level

The idea: infrastructure (clusters, DBs, queues, DNS, IAM) is declared in versioned files and created/updated by a tool — instead of console clicking that produces undocumented, unreproducible snowflake environments.

• Declarative + reconciliation, again: Terraform is the same mental loop as K8s — desired state (HCL) vs actual state (state file + provider APIs) → terraform plan shows the diff, apply converges it. If you understand K8s controllers, you already understand Terraform's model; saying that sentence shows synthesis.
• Why it wins: review infra changes in PRs; reproduce whole environments (staging = prod's config with smaller numbers); disaster recovery becomes apply; drift becomes detectable.
• Vocabulary: Terraform/OpenTofu (declarative, provider ecosystem), Pulumi/CDK (real languages compiling to the same model), Helm (K8s package templating), GitOps (ArgoCD/Flux: a controller in the cluster pulls desired state from git and reconciles continuously — git becomes the deploy interface and the audit log; git revert is a rollback).

You don't need deep Terraform expertise — "I understand the declarative model well from K8s, I've consumed IaC-managed infra, and I'd be productive in it quickly" is honest and lands.

9Rapid-fire interview Q&A

10Self-test

← Clean code & patterns LLM fundamentals →