CROP
Shared

GCP Infrastructure & API Gateway Plan

Date: 2025-12-05 Status: In Progress Environment: Development (all services = prod, separation only in database: dev/stage/prod)

GCP Infrastructure & API Gateway Plan

Date: 2025-12-05 Status: In Progress Environment: Development (all services = prod, separation only in database: dev/stage/prod)

Executive Summary

Full GCP infrastructure audit revealed several critical issues:

  1. API Gateway covers only ~30% of required endpoints
  2. crop-front-admin calls services directly, bypassing Gateway
  3. health-analytics-service is completely missing from Gateway
  4. Inconsistent IAM policies between services

1. Cloud Run Services - Current State

ServiceStatusPublic AccessAPI Gateway SAIn Gateway
search-serviceReadyallUsers + Gateway SAPresentPartial
payment-serviceReadyallUsers + Gateway SAPresentFull
delivery-apiReadyallUsersMissingNo
delivery-workerReadyNo (correct)N/AN/A
health-analyticsReadyallUsersMissingNo
payment-stagingFAILED---

URLs

search-service:        https://search-service-222426967009.us-east1.run.app
payment-service:       https://payment-service-222426967009.us-east1.run.app
delivery-api:          https://delivery-api-222426967009.us-east1.run.app
health-analytics:      https://health-analytics-service-222426967009.us-east1.run.app

API Gateway

Gateway URL:     https://crop-gateway-2u6j9u5d.ue.gateway.dev
Cloudflare:      https://api.crop-dev.app (proxies to Gateway)
Current Config:  crop-api-v3

2. Gateway.yaml - Coverage Analysis

Current Coverage

ServiceIn GatewayTotal RoutesCoverage
search-service (public)7978%
search-service (admin)31421%
payment-service88100%
health-analytics020+0%
delivery-api050%

Search-Service Admin Routes

In Gateway (3):

  • /api/admin/sync
  • /api/admin/analyze-index
  • /api/admin/debug-query

NOT in Gateway (11):

  • /api/admin/monitoring/overview
  • /api/admin/monitoring/metrics
  • /api/admin/monitoring/alerts
  • /api/admin/monitoring/errors
  • /api/admin/field-completeness
  • /api/admin/validate-data
  • /api/admin/index-document
  • /api/admin/delete-by-query
  • /api/admin/collections
  • /api/admin/create-index
  • /api/admin/parts

Health-Analytics Service Routes

NOT in Gateway (all):

  • /api/analytics/field-completeness
  • /api/analytics/summary
  • /api/analytics/environments
  • /api/analytics/collections
  • /api/health/media/coverage
  • /api/health/media/gaps
  • /api/health/media/distribution
  • /api/health/media/dashboard/summary
  • /api/health/media/dashboard/trend
  • /api/health/media/dashboard/activity
  • /api/health/media/dashboard/vendors
  • /api/health/collections
  • /api/health/vendors
  • /api/health/vendors/:vendor/metrics
  • /api/health/quality-distribution
  • /api/admin/collections
  • /api/admin/collections/:name
  • /api/admin/collections/:name/query
  • /api/admin/collections/:name/aggregate

Delivery-API Routes

NOT in Gateway (all):

  • /health
  • /providers
  • /rates (POST)
  • /addresses/validate (POST)
  • /shipments (GET/POST)
  • /shipments/:tracking (GET/DELETE)

3. crop-front-admin - API Calls

Direct calls to search-service:

// lib/search-service.ts - baseURL = SEARCH_SERVICE_URL (direct Cloud Run)
/api/admin/monitoring/overview
/api/admin/monitoring/metrics
/api/admin/monitoring/alerts
/api/admin/monitoring/errors
/api/admin/field-completeness
/api/admin/validate-data
/api/admin/index-document
/api/admin/delete-by-query
/api/admin/sync

Direct calls to health-analytics:

// lib/health-analytics-client.ts - baseURL = HEALTH_ANALYTICS_API_URL (direct Cloud Run)
/api/health/media/coverage
/api/health/media/gaps
/api/health/media/dashboard/summary
/api/health/media/dashboard/trend
/api/analytics/environments
/api/analytics/summary
/api/analytics/field-completeness

Direct calls (health-analytics-api.ts):

// lib/health-analytics-api.ts - baseURL = NEXT_PUBLIC_HEALTH_API_URL
/overview
/vendors
/vendors/:vendorCode
/collections

4. Action Plan

Phase 1: Fix Critical Issues ✅ COMPLETED (2025-12-05)

Priority: HIGH

  • 1.1 Delete payment-service-staging (broken, not needed in dev)

    gcloud run services delete payment-service-staging --region=us-east1

  • 1.2 Add API Gateway IAM to delivery-api

    gcloud run services add-iam-policy-binding delivery-api \
  --region=us-east1 \
  --member="serviceAccount:api-gateway@noted-bliss-466410-q6.iam.gserviceaccount.com" \
  --role="roles/run.invoker"

  • 1.3 Add API Gateway IAM to health-analytics-service

    gcloud run services add-iam-policy-binding health-analytics-service \
  --region=us-east1 \
  --member="serviceAccount:api-gateway@noted-bliss-466410-q6.iam.gserviceaccount.com" \
  --role="roles/run.invoker"

Phase 2: Expand Gateway Configuration ✅ COMPLETED (2025-12-05)

Priority: MEDIUM

  • 2.1 Add search-service admin routes to gateway.yaml (11 routes):

    • /api/admin/monitoring/overview
    • /api/admin/monitoring/metrics
    • /api/admin/monitoring/alerts
    • /api/admin/monitoring/errors
    • /api/admin/field-completeness
    • /api/admin/validate-data
    • /api/admin/index-document
    • /api/admin/delete-by-query
    • /api/admin/collections
    • /api/admin/create-index
    • /api/admin/parts

  • 2.2 Add health-analytics routes to gateway.yaml (19 routes):

    • /api/analytics/field-completeness
    • /api/analytics/summary
    • /api/analytics/environments
    • /api/analytics/collections
    • /api/health/media/coverage
    • /api/health/media/gaps
    • /api/health/media/distribution
    • /api/health/media/dashboard/summary
    • /api/health/media/dashboard/trend
    • /api/health/media/dashboard/activity
    • /api/health/media/dashboard/vendors
    • /api/health/collections
    • /api/health/vendors
    • /api/health/vendors/{vendor}/metrics
    • /api/health/quality-distribution
    • /api/health/admin/collections/{name} (GET/POST/DELETE)
    • /api/health/admin/collections/{name}/query
    • /api/health/admin/collections/{name}/aggregate

  • 2.3 Add delivery-api routes to gateway.yaml (8 operations):

    • /api/v1/delivery/health (GET)
    • /api/v1/delivery/providers (GET)
    • /api/v1/delivery/rates (POST)
    • /api/v1/delivery/addresses/validate (POST)
    • /api/v1/delivery/shipments (GET, POST)
    • /api/v1/delivery/shipments/{tracking} (GET, DELETE)

Phase 3: Deploy Updated Gateway ✅ COMPLETED (2025-12-05)

Priority: MEDIUM

  • 3.1 Validate gateway.yaml

    bunx swagger-cli validate openapi/gateway.yaml
# Result: openapi/gateway.yaml is valid

  • 3.2 Create new API Gateway configuration

    gcloud api-gateway api-configs create crop-api-v4 \
  --api=crop-api \
  --openapi-spec=gateway.yaml \
  --backend-auth-service-account=api-gateway@noted-bliss-466410-q6.iam.gserviceaccount.com

  • 3.3 Update gateway to new configuration

    gcloud api-gateway gateways update crop-gateway \
  --location=us-east1 \
  --api=crop-api \
  --api-config=crop-api-v4

  • 3.4 Test all endpoints via api.crop-dev.app

    • Delivery: /api/v1/delivery/health → 200 OK
    • Delivery: /api/v1/delivery/providers → 200 OK ["ups","fedex","dhl"]
    • Health Analytics: /api/health/media/coverage → 401 (JWT required - correct)
    • Search Admin: /api/admin/monitoring/overview → 401 (token required - correct)
    • Existing endpoints still work (search, payment)

Phase 4: Migrate Frontend URLs ✅ COMPLETED (2025-12-05)

Priority: LOW

  • 4.1 crop-front-admin: Update SEARCH_SERVICE_URL → api.crop-dev.app
  • 4.2 crop-front-admin: Update HEALTH_ANALYTICS_API_URL → api.crop-dev.app
  • 4.3 CROP-front: Already using api.crop-dev.app (no changes needed)
  • 4.4 Added compatibility routes (/ready, /api/v1/health/analytics) to Gateway v5

Changes made:

  • crop-front-admin/.env.local: Updated all service URLs to api.crop-dev.app
  • crop-front-admin/.env.example: Updated documentation with Gateway URLs
  • CROP-front/.env.local: Fixed DELIVERY_API_URL to include /api/v1/delivery prefix
  • Gateway v5: Added /ready and /api/v1/health/analytics routes for compatibility

Important: Delivery API URL format:

# Correct (with prefix for Gateway routes):
DELIVERY_API_URL=https://api.crop-dev.app/api/v1/delivery

# Wrong (missing prefix - will get 404 on /rates, /shipments etc):
DELIVERY_API_URL=https://api.crop-dev.app

Verified endpoints:

  • All Search Service endpoints work via Gateway ✅
  • All Health Analytics endpoints work via Gateway ✅
  • All Delivery API endpoints work via Gateway ✅
  • All Payment Service endpoints work via Gateway ✅

5. Architecture Recommendations

Current Architecture (Issues)

┌─────────────────┐          ┌─────────────────┐
│ crop-front-admin│ ───────► │ search-service  │  (direct call)
└─────────────────┘          └─────────────────┘
        │                    ┌─────────────────┐
        └──────────────────► │health-analytics │  (direct call)
                             └─────────────────┘

┌─────────────────┐          ┌─────────────────┐
│   CROP-front    │ ───────► │  delivery-api   │  (direct call)
└─────────────────┘          └─────────────────┘
        │
        └──────────────────► ┌─────────────────┐
                             │  api.crop-dev   │  (via Gateway)
                             │  (search/pay)   │
                             └─────────────────┘

Target Architecture

┌─────────────────┐
│ crop-front-admin│ ─┐
└─────────────────┘  │
                     │      ┌─────────────────┐      ┌─────────────────┐
┌─────────────────┐  ├────► │   Cloudflare    │ ───► │   API Gateway   │
│   CROP-front    │ ─┤      │ api.crop-dev.app│      │   crop-gateway  │
└─────────────────┘  │      └─────────────────┘      └────────┬────────┘
                     │                                        │
┌─────────────────┐  │                           ┌────────────┼────────────┐
│  External APIs  │ ─┘                           │            │            │
└─────────────────┘                              ▼            ▼            ▼
                                          ┌──────────┐ ┌──────────┐ ┌──────────┐
                                          │ search   │ │ payment  │ │ delivery │
                                          └──────────┘ └──────────┘ └──────────┘
                                                 │            │
                                                 ▼            ▼
                                          ┌──────────┐ ┌──────────┐
                                          │ health-  │ │ identity │
                                          │ analytics│ └──────────┘
                                          └──────────┘

Benefits of Target Architecture

  1. Single entry point - all via api.crop-dev.app
  2. Cloudflare CDN - caching, DDoS protection
  3. Centralized rate limiting - in API Gateway
  4. Unified authentication format - Clerk JWT via Gateway
  5. Easier monitoring - all logs in one place

6. Database Environment Note

Important: We are in dev phase, all services point to the same infrastructure:

  • Cloud Run services: "prod" (only one set)
  • API Gateway: one (crop-gateway)
  • Cloudflare: one domain (api.crop-dev.app)

Separation only in MongoDB database:

  • crop_dev - development database
  • crop_stage - staging database (CURRENTLY USING)
  • crop_prod - production database

Services point to crop_stage via MONGODB_DB_NAME env var.

7. Testing Checklist

After each phase:

Phase 1 Tests

  • payment-service-staging deleted
  • gcloud run services list shows 5 services (without staging)
  • gcloud run services get-iam-policy delivery-api includes api-gateway SA
  • gcloud run services get-iam-policy health-analytics-service includes api-gateway SA

Phase 2-3 Tests

  • curl https://api.crop-dev.app/api/admin/monitoring/overview returns data
  • curl https://api.crop-dev.app/api/health/media/coverage returns data
  • curl https://api.crop-dev.app/api/v1/delivery/providers returns ["ups","fedex","dhl"]
  • All endpoints from gateway.yaml accessible via api.crop-dev.app

Phase 4 Tests

  • crop-front-admin dashboard loads
  • Media coverage page works
  • Monitoring page shows metrics
  • CROP-front checkout flow with delivery rates works

8. Rollback Plan

If something goes wrong:

# Rollback Gateway to previous version
gcloud api-gateway gateways update crop-gateway \
  --location=us-east1 \
  --api=crop-api \
  --api-config=crop-api-v3  # previous version

# Remove IAM binding (if needed)
gcloud run services remove-iam-policy-binding delivery-api \
  --region=us-east1 \
  --member="serviceAccount:api-gateway@noted-bliss-466410-q6.iam.gserviceaccount.com" \
  --role="roles/run.invoker"
  • /docs/CRITICAL_ANALYSIS.md - Env refactoring analysis
  • /docs/ENV_REFACTORING_STATUS.md - Env vars migration status
  • /docs/INFRASTRUCTURE_STATUS.md - Overall infrastructure status
  • /CROP-front/openapi/gateway.yaml - Current Gateway configuration

Next Steps: Start with Phase 1 - fix critical issues (delete staging, add IAM).

Production Pre-Deployment Checklist

> Purpose: This checklist ensures that code changes are production-ready before deployment. > Target: Bun + Hono + TypeScript monorepo ()

Clerk Frontend Integration Guide

Prerequisites: - Backend Phases 1 & 2 must be deployed - Phase 0 (Clerk Dashboard setup) must be completed - Environment variables from backend team

On this page

GCP Infrastructure & API Gateway PlanExecutive Summary1. Cloud Run Services - Current StateURLsAPI Gateway2. Gateway.yaml - Coverage AnalysisCurrent CoverageSearch-Service Admin RoutesHealth-Analytics Service RoutesDelivery-API Routes3. crop-front-admin - API CallsDirect calls to search-service:Direct calls to health-analytics:Direct calls (health-analytics-api.ts):4. Action PlanPhase 1: Fix Critical Issues ✅ COMPLETED (2025-12-05)Phase 2: Expand Gateway Configuration ✅ COMPLETED (2025-12-05)Phase 3: Deploy Updated Gateway ✅ COMPLETED (2025-12-05)Phase 4: Migrate Frontend URLs ✅ COMPLETED (2025-12-05)5. Architecture RecommendationsCurrent Architecture (Issues)Target ArchitectureBenefits of Target Architecture6. Database Environment Note7. Testing ChecklistPhase 1 TestsPhase 2-3 TestsPhase 4 Tests8. Rollback PlanRelated Documents