Skip to content

CI/CD Pipelines

Carbon Connect uses GitHub Actions for continuous integration and deployment. The project has 11 workflow files covering quality, security, deployment, and repository management.


Workflow Overview

flowchart TB
    subgraph PR["Pull Request"]
        TDD[PR TDD Gate]
        BCI[Backend CI]
        FCI[Frontend CI]
        QG[Quality Gates]
        SS[Security Scan]
        DR[Dependency Review]
        AL[Auto Label]
    end

    subgraph Merge["Merge to Main"]
        DOC[Deploy Docs]
    end

    subgraph Release["Release / Deploy"]
        DS[Deploy Staging]
        DP[Deploy Production]
        REL[Release]
    end

    subgraph Scheduled["Scheduled"]
        ST[Stale Management]
        SS2[Weekly Security Scan]
    end

Pipeline Details

1. Backend CI

File: .github/workflows/backend-ci.yml

Triggers: Push to main/master, PRs (when backend/** changes)

Job Description Dependencies
lint Ruff linter + formatter check None
test pytest with coverage, pgvector, Valkey lint

Services: PostgreSQL 16 (pgvector), Redis 7

Key steps:

  • Enables pgvector extension before running migrations
  • Runs alembic upgrade head for database schema
  • Executes pytest tests/ --maxfail=1 --cov=backend
  • Uploads coverage to Codecov

2. Frontend CI

File: .github/workflows/frontend-ci.yml

Triggers: Push to main/master, PRs (when frontend/** changes)

Job Description Dependencies
lint ESLint check None
build TypeScript compilation + Next.js build lint
test Vitest with coverage lint (continue-on-error)

TDD Workflow

Frontend tests use continue-on-error: true to report status without blocking PRs. This supports the TDD workflow where failing tests indicate pending implementations.


3. PR TDD Gate

File: .github/workflows/pr-tdd-gate.yml

Triggers: PR opened, edited, synchronized, reopened

Enforces TDD compliance by checking the PR description for:

  • A ## TDD Proof section
  • Checked items for:
    • "Tests written before production code"
    • "Evidence of failing test run included"
    • "All tests passing locally"

Blocks merge if any required checklist item is missing or unchecked.


4. Quality Gates

File: .github/workflows/quality-gates.yml

Triggers: Push to main/master/staging, all PRs

This is the comprehensive quality check that gates merges:

Job Description Blocking
backend-quality Ruff lint + format + mypy + bandit Yes
frontend-quality ESLint + TypeScript + Prettier Yes
test-coverage pytest with --cov-fail-under=69 Yes
security-scan Trivy vulnerability scanner + TruffleHog secrets Advisory
dependency-audit pip-audit + npm audit Advisory
build-check Docker build (API + Celery) + frontend build Yes
terraform-validate terraform fmt + validate + tfsec Advisory
quality-gate Aggregates all results, fails if critical checks fail Final gate

5. Security Scan

File: .github/workflows/security-scan.yml

Triggers: Push to main/master, PRs, weekly (Monday 6 AM UTC)

Job Description
codeql-python CodeQL analysis (security-extended + quality queries)
codeql-javascript CodeQL for TypeScript/JavaScript
dependency-scan Safety (Python) + npm audit
secret-scan TruffleHog verified secrets scan
sast Bandit static analysis for Python

6. Dependency Review

File: .github/workflows/dependency-review.yml

Triggers: PRs to main/master/staging

  • Scans new dependencies for vulnerabilities (fails on HIGH/CRITICAL)
  • Enforces license policy:
    • Allowed: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC
    • Denied: GPL-2.0, GPL-3.0, AGPL-3.0 (copyleft)
  • Posts findings as PR comments

7. Deploy to Staging

File: .github/workflows/deploy-staging.yml

Triggers: Push to staging branch, manual dispatch

Job Description
test Run pytest (skippable via manual dispatch)
build Build + push Docker image to ECR (staging-{sha})
deploy Update ECS tasks: API, Worker, Beat
migrate Run Alembic migrations via one-off ECS task
notify Report deployment status

Platform: Builds for linux/arm64 (Graviton).


8. Deploy to Production

File: .github/workflows/deploy-production.yml

Triggers: GitHub Release published, manual dispatch

Job Description Dependencies
validate Verify image exists in ECR None
backup Create RDS snapshot validate
migrate Run database migrations validate, backup
deploy-api Blue/green API deployment validate, backup, migrate
deploy-workers Deploy Worker + Beat validate, deploy-api
smoke-test Health check + endpoint verification deploy-api, deploy-workers
notify Success/failure notification All
rollback Rollback to previous task definition On failure

Production Safety

  • cancel-in-progress: false -- never cancels in-flight production deployments
  • Requires production environment approval
  • Creates RDS snapshot before every deployment
  • Includes automatic rollback on failure

9. Release

File: .github/workflows/release.yml

Triggers: Push tag v*.*.*, manual dispatch

  • Generates changelog from merged PRs using labels
  • Creates GitHub Release with categorized changelog
  • Supports pre-releases (tags containing -)

10. Auto Label

File: .github/workflows/auto-label.yml

Triggers: PR opened, synchronized, reopened

  • Labels PRs by file paths (backend, frontend, infra, docs)
  • Labels PRs by size (XS: <10, S: <50, M: <200, L: <500, XL: >500 lines)
  • Warns on XL pull requests

11. Stale Issue Management

File: .github/workflows/stale.yml

Triggers: Daily at midnight UTC

Target Stale After Close After Exempt Labels
Issues 60 days 14 days pinned, security, bug, in-progress
PRs 30 days 7 days pinned, security, dependencies, in-progress

12. Deploy Documentation

File: .github/workflows/docs.yml

Triggers: Push to main (when docs/** or mkdocs.yml changes), manual dispatch

  • Builds MkDocs site with mkdocs build --strict
  • Deploys to GitHub Pages via gh-pages branch

Concurrency Controls

All workflows use concurrency groups to prevent parallel runs:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true  # Except production deploys

Required Secrets

Secret Used By Description
AWS_ACCOUNT_ID Deploy workflows AWS account for OIDC role
CODECOV_TOKEN CI workflows Coverage upload token
GITHUB_TOKEN All workflows Auto-provided by GitHub
PROD_SUBNET_IDS Production deploy VPC subnet IDs for ECS tasks
PROD_SECURITY_GROUP Production deploy Security group for ECS tasks
STAGING_SUBNET_IDS Staging deploy VPC subnet IDs
STAGING_SECURITY_GROUP Staging deploy Security group