CI/CD Pipeline Architecture
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Developer Push โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ CI Pipeline โ
โ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โ
โ โ Lint โโ โ Test โโ โ Build โโ โ Push โ โ
โ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ CD Pipeline โ
โ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โ
โ โ Staging โโ โ E2E โโ โ Prod โโ โ Monitor โ โ
โ โ Deploy โ โ Tests โ โ Deploy โ โ Rollout โ โ
โ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โโโโโโโโโโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
GitHub Actions Pipeline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
| name: CI/CD Pipeline
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: test
options: >-
--health-cmd pg_isready
--health-interval 10s
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run tests
run: pytest --cov=app --cov-report=xml
- name: Upload coverage
uses: codecov/codecov-action@v4
build:
needs: test
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@v4
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
push: true
tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
deploy:
needs: build
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- name: Deploy to Kubernetes
uses: azure/k8s-deploy@v4
with:
manifests: k8s/
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
|
Deployment Strategies
| Strategy | Use Case | Tradeoff |
|---|
| Rolling | Most deployments | Gradual, safe |
| Blue-Green | Instant rollback needed | 2x infrastructure |
| Canary | Risk-sensitive | Complex routing |
| Recreate | Breaking changes | Brief downtime |
Kubernetes Deployment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
| # Deployment with health checks and rolling updates
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-service
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: api-service
template:
metadata:
labels:
app: api-service
spec:
containers:
- name: api
image: myapp:latest
ports:
- containerPort: 8000
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 15
periodSeconds: 20
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
|
Technologies I Use
- CI/CD: GitHub Actions, GitLab CI, Jenkins, Cloud Build
- Containers: Docker, containerd
- Orchestration: Kubernetes, GKE, EKS
- IaC: Terraform, Pulumi
- Monitoring: Prometheus, Grafana, Datadog
- Secrets: HashiCorp Vault, AWS Secrets Manager
Pipeline Best Practices
1
2
3
4
5
6
7
8
9
10
11
12
| # Multi-stage builds for smaller images
FROM python:3.11-slim as builder
WORKDIR /app
COPY requirements.txt .
RUN pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt
FROM python:3.11-slim
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-cache /wheels/*
COPY . .
CMD ["gunicorn", "app:app", "-b", "0.0.0.0:8000"]
|
Frequently Asked Questions
What is CI/CD pipeline development?
CI/CD (Continuous Integration/Continuous Deployment) automates building, testing, and deploying code. Pipeline development involves setting up automated workflows that run tests, perform security scans, build artifacts, and deploy to various environments on every code change.
How much does CI/CD setup cost?
CI/CD development typically costs $100-150 per hour. A basic pipeline starts around $5,000-10,000, while thorough enterprise pipelines with multiple environments, security scanning, and complex deployment strategies range from $20,000-50,000+.
GitHub Actions vs GitLab CI vs Jenkins: which should I use?
Choose GitHub Actions for: GitHub repos, simpler needs, marketplace actions. Choose GitLab CI for: GitLab repos, built-in container registry. Choose Jenkins for: complex enterprise needs, maximum customization. GitHub Actions is the simplest for most teams.
What should a production CI/CD pipeline include?
Essential: linting, unit tests, security scanning (dependencies, secrets), build step, and deployment. Better: integration tests, staging deployment, smoke tests, rollback capability, notifications, and approval gates for production.
Can you help speed up slow CI/CD pipelines?
Yes. I optimize: parallelization, caching (dependencies, build artifacts), test splitting, incremental builds, smaller Docker images, and removing unnecessary steps. I’ve reduced pipeline times by 50-80% for slow builds.
Experience:
Related Technologies: GitHub Actions, Docker/Kubernetes, Terraform, GCP, AWS