CI Rollback Guide

Quick Reference

Emergency Rollback (Fastest)

# If you need to rollback CI immediately
git checkout main
cp .github/workflows/ci.yml.backup-20251111 .github/workflows/ci.yml
git add .github/workflows/ci.yml
git commit -m "revert(ci): restore working CI configuration"
git push origin main

When to use: CI is completely broken, blocking all PRs

Rollback Scenarios

1. PR Tests Failing (No Rollback Needed)

Situation: Your PR with CI changes has failing tests

Solution:

# Fix the workflow on your branch
vim .github/workflows/ci.yml
git add .github/workflows/ci.yml
git commit -m "fix(ci): correct configuration"
git push

Why no rollback: Main branch is untouched, PR auto-updates

2. Merged Changes Breaking Main

Situation: CI changes merged to main, now CI is failing

Solution A: Revert Commit (Recommended)

# Find the problematic commit
git log --oneline -10

# Revert it
git revert <commit-sha>
git push origin main

Pros:

• Clean git history
• Preserves audit trail
• Safe for team collaboration

Solution B: Restore from Backup

git checkout main
cp .github/workflows/ci.yml.backup-20251111 .github/workflows/ci.yml
git add .github/workflows/ci.yml
git commit -m "revert(ci): restore from backup"
git push origin main

Pros:

• Guaranteed working state
• Fast recovery
• No need to identify problematic commit

3. Complex Git History

Situation: Multiple commits need rollback, git history is messy

Solution: Reset to Last Good Commit (Nuclear Option)

# Find last known good commit
git log --oneline -20
gh run list --limit 20  # Find last successful CI run

# Reset to that commit
git checkout main
git reset --hard <last-good-commit>

# Force push (⚠️ COORDINATE WITH TEAM FIRST)
git push --force origin main

⚠️ Warnings:

• Destroys commits after reset point
• Requires team coordination
• Can break others' branches

Prerequisites:

• Team is notified
• No one else is pushing to main
• You have backup of important commits
• You understand you're rewriting history

4. Backup File is Missing

Situation: Need to rollback but backup file is gone

Solution: Use Git History

# Find commit before CI changes
git log --oneline .github/workflows/ci.yml

# Checkout old version
git checkout <old-commit> -- .github/workflows/ci.yml

# Commit it
git add .github/workflows/ci.yml
git commit -m "revert(ci): restore previous working version"
git push origin main

Verification Steps

After any rollback, verify CI works:

1. Check GitHub Actions

# View recent runs
gh run list --limit 5

# Watch latest run
gh run watch

2. Test Locally

# Test each service
cd services/catalog && bun test
cd ../identity && bun test
cd ../media && bun test
cd ../payment && bun test
cd ../search && bun run test:ci

3. Create Test PR

# Make trivial change
echo "# Test" >> README.md
git checkout -b test-ci
git commit -am "test: verify CI works"
git push -u origin test-ci
gh pr create --title "Test: Verify CI" --body "Testing CI after rollback"

Prevention

Before Merging CI Changes

1. Test on feature branch first

   git checkout -b ci-test
   # Make changes
   git push -u origin ci-test
   gh pr create
   # Wait for CI to pass

2. Verify locally

   for dir in services/*/; do
     echo "Testing $(basename $dir)..."
     (cd "$dir" && bun test) || echo "Failed: $(basename $dir)"
   done

3. Create backup

   cp .github/workflows/ci.yml .github/workflows/ci.yml.backup-$(date +%Y%m%d)
   git add .github/workflows/ci.yml.backup-*
   git commit -m "docs: backup CI before changes"

4. Use feature flags for risky changes

   # Example: gradual rollout
   if: github.ref == 'refs/heads/develop'  # Test on develop first

Branch Protection

Configure GitHub branch protection:

1. Go to repo Settings → Branches
2. Add rule for main:
   • ✅ Require status checks to pass
   • ✅ Require branches to be up to date
   • ✅ Require review from Code Owners
3. Select required checks: lint, test

Backup Files

Current Backups

• .github/workflows/ci.yml.backup-20251111 - Pre-matrix migration (committed: 2025-11-11)

Creating New Backup

# Before making CI changes
cp .github/workflows/ci.yml .github/workflows/ci.yml.backup-$(date +%Y%m%d)
git add .github/workflows/ci.yml.backup-*
git commit -m "docs: backup CI configuration"

Listing Backups

ls -lh .github/workflows/*.backup*

Removing Old Backups

# After verifying new CI works for 1+ week
git rm .github/workflows/ci.yml.backup-OLD
git commit -m "chore: remove old CI backup"

Troubleshooting

Rollback Didn't Fix CI

Possible causes:

1. Infrastructure issue (GitHub Actions outage)
2. Dependency issue (package.json changes)
3. External service down (MongoDB, Elasticsearch)

Diagnostics:

# Check GitHub status
curl -s https://www.githubstatus.com/api/v2/status.json | jq

# Check recent commits
git log --oneline -10

# Review all recent changes
git diff HEAD~5 HEAD

Can't Push to Main

Error: ! [rejected] main -> main (non-fast-forward)

Cause: Someone else pushed while you were rolling back

Solution:

git pull --rebase origin main
git push origin main

Force Push Failed

Error: ! [remote rejected] main -> main (protected)

Cause: Branch protection prevents force push (good!)

Solution: Use revert instead of reset:

git revert <bad-commit-1> <bad-commit-2>
git push origin main

Recovery Checklist

After rollback:

• GitHub Actions passing
• All services tested locally
• Test PR created and passes
• Team notified of rollback
• Root cause identified
• Fix planned for next iteration
• Documentation updated

Contact / Escalation

If Rollback Fails

1. Notify team immediately in Slack/Discord
2. Stop all merges to main
3. Check GitHub Actions logs for error details
4. Review git reflog for history: git reflog
5. Consider temporary workaround:

   # Disable CI temporarily (last resort)
   mv .github/workflows/ci.yml .github/workflows/ci.yml.disabled
   git commit -am "chore: temporarily disable CI for debugging"
   git push

After Resolution

1. Re-enable CI
2. Document what went wrong
3. Update rollback guide with lessons learned
4. Add preventive measures

Related Documentation

• CI_ARCHITECTURE.md - Full CI architecture documentation
• .github/workflows/ci.yml - Current CI configuration
• .github/workflows/ci.yml.backup-20251111 - Backup CI configuration