Last Updated: 2025-12-29 Environment: Dev/Staging Platform: Google Cloud Platform (GCP) Runtime: Bun 1.3.4

CROP Microservices Architecture

Last Updated: 2025-12-29 Environment: Dev/Staging Platform: Google Cloud Platform (GCP) Runtime: Bun 1.3.4

Overview
System Architecture
Services
Infrastructure
Data Flow
Networking
Security
Monitoring

Overview

CROP Microservices is a cloud-native application built on GCP, providing search, catalog, payment, and identity management services for agricultural parts marketplace.

Key Technologies

Layer	Technology
Runtime	Bun 1.3.4
Platform	Google Cloud Run (Gen2, serverless)
Databases	MongoDB Atlas (primary), Elasticsearch (search)
Networking	VPC, Private Service Connect (PSC)
CI/CD	GitHub Actions + Cloud Build
Monitoring	Cloud Logging, Cloud Trace

Project Structure

microservices/
├── services/              # Microservices
│   ├── search/           # ✅ DEPLOYED - Full-text search API
│   ├── health-analytics/ # ✅ DEPLOYED - Health monitoring
│   ├── catalog/          # Product catalog management
│   ├── user/             # User accounts and equipment (garage)
│   ├── media/            # Image and file handling
│   ├── payment/          # Stripe payment processing
│   └── analytics/        # Usage analytics
├── packages/             # Shared libraries
├── docs/                 # Documentation
└── infrastructure/       # IaC configs (if any)

System Architecture

High-Level Architecture

┌─────────────────────────────────────────────────────────┐
│                     Internet/CDN                         │
└────────────────────┬────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────┐
│              Google Cloud Load Balancer                  │
└────────┬────────────────────────────────────────────────┘
         │
         ├──────────────┬──────────────┬──────────────────┐
         │              │              │                  │
         ▼              ▼              ▼                  ▼
    ┌─────────┐   ┌─────────┐   ┌──────────┐      ┌──────────┐
    │ Search  │   │ Catalog │   │   User   │      │ Payment  │
    │ Service │   │ Service │   │ Service  │      │ Service  │
    └────┬────┘   └────┬────┘   └────┬─────┘      └────┬─────┘
         │             │             │                  │
         │             │             │                  │
         ▼             ▼             ▼                  ▼
    ┌──────────────────────────────────────────────────────┐
    │              VPC (crop-vpc)                          │
    │  ┌──────────────────────────────────────────┐       │
    │  │    VPC Access Connector (crop-connector)  │       │
    │  └──────────────────────────────────────────┘       │
    │                                                      │
    │  ┌─────────────┐      ┌──────────────────┐         │
    │  │ Elasticsearch│      │ Private Service  │         │
    │  │ 10.0.0.52   │      │ Connect Endpoints│         │
    │  └─────────────┘      └──────────────────┘         │
    └──────────────────────────────────────────────────────┘
                     │
                     ▼
    ┌─────────────────────────────────────────────────────┐
    │         MongoDB Atlas (crop-gcp-pl-0)               │
    │         Via PSC: pl-00-000 @ 10.0.0.2               │
    └─────────────────────────────────────────────────────┘

Service Communication

┌───────────────┐
│  Client App   │
└───────┬───────┘
        │ HTTPS
        ▼
┌───────────────┐     ┌──────────────┐
│ Search Service│────▶│ Elasticsearch│
└───────┬───────┘     └──────────────┘
        │
        │ Internal Call (if needed)
        ▼
┌───────────────┐     ┌──────────────┐
│ Catalog Service│───▶│   MongoDB    │
└───────────────┘     └──────────────┘

Services

1. Search Service ✅ DEPLOYED

Status: Production-ready (dev/staging) URL: https://api.crop-dev.app Revision: search-service-00023-kv6

Purpose: Full-text search API for agricultural parts

Tech Stack:

Runtime: Bun 1.3.4
Framework: Hono.js
Database: Elasticsearch 8.x
Data Source: MongoDB Atlas (via PSC)

Key Features:

Full-text search with facets
Autocomplete with multi-field support
Equipment fitment search
OpenAPI 3.0 documentation
Prometheus metrics endpoint

Dependencies:

MongoDB Atlas: Source of truth
Elasticsearch: Search index (10.0.0.52)
VPC Access Connector: Network access

Configuration:

Image: gcr.io/noted-bliss-466410-q6/search-service:fa5d076
Execution Environment: gen2
VPC Connector: crop-connector
VPC Egress: private-ranges-only
CPU: 1000m
Memory: 512Mi
Concurrency: 80

Environment Variables:

MONGODB_URI - MongoDB connection string
ELASTICSEARCH_URL - Elasticsearch endpoint
SEARCH_INDEX_NAME - Index alias (default: parts_current)
ALLOWED_ORIGINS - CORS origins
ADMIN_API_TOKEN - Admin endpoints authentication

Documentation:

API: /docs (Swagger UI)
OpenAPI: /openapi.json
See: services/search/CLAUDE.md

2. Health Analytics Service ✅ DEPLOYED

Status: Operational URL: https://api.crop-dev.app Revision: health-analytics-service-00002-wsv

Purpose: System health monitoring and analytics

Tech Stack:

Runtime: Bun
Framework: Hono.js

3. Catalog Service

Status: Development Path: services/catalog/

Purpose: Product catalog management API

Features:

Parts CRUD operations
Category and manufacturer management
Bulk import/export
Quality scoring

4. User Service

Status: Development Path: services/user/

Purpose: User accounts and equipment (garage) management

Features:

User profile management
Equipment/garage (vehicle) management
Clerk authentication integration

5. Media Service

Status: Development Path: services/media/

Purpose: Image and file handling

Features:

Image upload to GCS
CDN delivery via Cloud Storage

6. Payment Service

Status: Development Path: services/payment/

Purpose: Stripe payment processing

Features:

Stripe integration
Customer management
Payment intents

7. Analytics Service

Status: Development Path: services/analytics/

Purpose: Usage analytics and metrics

Infrastructure

GCP Project

Project ID: noted-bliss-466410-q6 Project Number: 222426967009 Region: us-east1 (primary)

Compute Resources

Resource	Type	Configuration
Search Service	Cloud Run	Gen2, 1 vCPU, 512MB RAM
Health Analytics	Cloud Run	Gen2, default
Elasticsearch VM	Compute Engine	e2-medium, 10.0.0.52

Networking

VPC: crop-vpc

Subnets:

crop-connector-subnet - For VPC Access Connector

VPC Access Connector:

Name: crop-connector
Status: READY
Min Instances: 2
Max Instances: 3
Throughput: 200-300 Mbps

Private Service Connect:

50 PSC forwarding rules (crop-mongodb-0 to crop-mongodb-49)
Service Attachments to MongoDB Atlas
DNS Private Zone: atlas-psc-dkwuhg

NAT Gateway: (Currently running, to be removed)

Name: crop-nat
Router: crop-router
Cost: $40/month

Databases

MongoDB Atlas:

Cluster: crop-gcp-pl-0
Tier: M0 (Free tier)
Region: us-east1
Replica Set: 3 nodes (pl-00-000, pl-00-001, pl-00-012)
Connection: Private Service Connect (single-node currently)

Elasticsearch:

Version: 8.x
Host: 10.0.0.52:9200
Deployment: Compute Engine VM
Index: parts_current (alias)

Storage

GCS Buckets:

crop_parts - Parts images and media
Image prefix: newholland/images

Data Flow

Search Query Flow

1. Client Request
   ↓ HTTPS
2. Cloud Run (Search Service)
   ↓ Via VPC Connector
3. Elasticsearch (10.0.0.52)
   ↓ Query Results
4. Search Service (Transform)
   ↓ JSON Response
5. Client

Data Sync Flow (MongoDB → Elasticsearch)

1. MongoDB Atlas
   ↓ Read via PSC
2. Search Service (Transform)
   ↓ Index
3. Elasticsearch
   ↓ Verify
4. Alias Switch (Zero-downtime)

Scripts:

sync-mongodb-to-es.ts - Full sync
migrate-index.ts - Zero-downtime migration

Networking

Private Service Connect (PSC)

Purpose: Secure connectivity to MongoDB Atlas without exposing public IPs

Architecture:

Cloud Run
    ↓ VPC Access Connector
VPC (crop-vpc)
    ↓ PSC Endpoint (10.0.0.2)
MongoDB Atlas (pl-00-000)

Components:

Service Attachments (Atlas side)
- 3 attachments for 3 replica set members
- Currently only 1 registered (pl-00-000)
PSC Endpoints (GCP side)
- 50 forwarding rules created
- Mapped to 10.0.0.x private IPs
DNS Private Zone
- Zone: atlas-psc-dkwuhg
- Records: pl-00-000.dkwuhg.mongodb.net → 10.0.0.2

Current Status:

✅ Working: Single-node connection to pl-00-000
🔄 Pending: Register pl-00-001 and pl-00-012 for full HA

VPC Access Connector

Purpose: Connect Cloud Run to VPC resources (Elasticsearch, MongoDB via PSC)

Configuration:

Name: crop-connector
Network: crop-vpc
Subnet: crop-connector-subnet
Egress: private-ranges-only
State: READY

Attached Services:

search-service (Gen2)
health-analytics-service (if configured)

Security

Authentication

Cloud Run Services:

Public: --allow-unauthenticated (for public APIs)
Internal: IAM-based authentication

Admin Endpoints:

Token-based: ADMIN_API_TOKEN header
Optional IP whitelist: ADMIN_IP_WHITELIST

Secrets Management

Current: Environment variables in Cloud Run Recommended: Migrate to Secret Manager before production

Secrets to migrate:

MongoDB credentials
Elasticsearch credentials (if any)
API tokens
OAuth keys

Network Security

Egress Control:

VPC Access Connector with private-ranges-only egress
Only RFC 1918 traffic uses VPC
Public internet access via default route

Database Access:

MongoDB: Private Service Connect (no public access)
Elasticsearch: Private IP in VPC

Service-to-Service:

Internal calls via Cloud Run service URLs
IAM authentication recommended

Monitoring

Health Checks

Search Service:

/health - Quick health check
/ready?deep=true - Deep health check (MongoDB + Elasticsearch)
/live - Liveness probe

Health Check Response:

{
  "mongodb": "ok",
  "elasticsearch": "ok",
  "docCount": 3740,
  "revision": "search-service-00023-kv6"
}

Logging

Cloud Logging:

Automatic log collection from Cloud Run
Structured logging with request IDs
Filter: resource.type=cloud_run_revision

Example Queries:

# Errors in search service
gcloud logging read "resource.type=cloud_run_revision \
  AND resource.labels.service_name=search-service \
  AND severity>=ERROR" \
  --limit=20 \
  --project=noted-bliss-466410-q6

# MongoDB connection logs
gcloud logging read "resource.type=cloud_run_revision \
  AND textPayload:MongoDB" \
  --limit=20 \
  --project=noted-bliss-466410-q6

Metrics

Prometheus Metrics:

Endpoint: /metrics (search-service)
Metrics: Request count, latency histograms, error rate
Collection: Prometheus or Cloud Monitoring

Cloud Monitoring:

Automatic metrics from Cloud Run
Request count, latency, memory, CPU usage

Tracing

Cloud Trace:

Automatic trace collection
Trace context in X-Cloud-Trace-Context header
View: GCP Console → Trace

Cost Analysis

Current Monthly Costs (Dev/Staging)

Resource	Cost	Notes
VPC Access Connector	$7	Required for VPC access
NAT Gateway	$40	To be removed after HA setup
Elasticsearch VM	~$30	e2-medium instance
Cloud Run (Search)	~$5	Low traffic
MongoDB Atlas	$0	Free tier M0
Total	~$82/month

After Optimization

Resource	Cost	Savings
VPC Access Connector	$7	-
NAT Gateway	$0	-$40
Elasticsearch VM	~$30	-
Cloud Run	~$5	-
MongoDB Atlas	$0	-
Total	~$42/month	$40/month

Savings: $480/year

Deployment Topology

Current (Dev/Staging)

Production Traffic
        ↓
   Cloud Run (Gen2)
        ↓
  VPC Connector
        ↓
┌──────────────────────┐
│   crop-vpc           │
│  ┌───────────────┐   │
│  │ Elasticsearch │   │
│  │   10.0.0.52   │   │
│  └───────────────┘   │
│                      │
│  ┌───────────────┐   │
│  │ PSC Endpoint  │   │
│  │   10.0.0.2    │───┼───▶ MongoDB Atlas
│  └───────────────┘   │     (pl-00-000)
│                      │
│  ┌───────────────┐   │
│  │ NAT Gateway   │───┼───▶ Public Internet
│  └───────────────┘   │     (to be removed)
└──────────────────────┘

Target (Production)

Production Traffic
        ↓
   Cloud Run (Gen2)
        ↓
  VPC Connector
        ↓
┌──────────────────────┐
│   crop-vpc           │
│  ┌───────────────┐   │
│  │ Elasticsearch │   │
│  │   10.0.0.52   │   │
│  └───────────────┘   │
│                      │
│  ┌───────────────┐   │
│  │ PSC Endpoints │   │
│  │  10.0.0.2     │───┼───▶ MongoDB Atlas
│  │  10.0.0.3     │───┼───▶ (3-node replica set)
│  │  10.0.0.4     │───┼───▶ Full HA
│  └───────────────┘   │
└──────────────────────┘

Development Workflow

Local Development

# Install dependencies
bun install

# Run service locally
cd services/search
bun run dev

# Test
bun test

Build & Deploy

# Build Docker image
docker build -f services/search/Dockerfile -t search-service .

# Push to GCR
docker tag search-service gcr.io/noted-bliss-466410-q6/search-service:latest
docker push gcr.io/noted-bliss-466410-q6/search-service:latest

# Deploy to Cloud Run
gcloud run deploy search-service \
  --image=gcr.io/noted-bliss-466410-q6/search-service:latest \
  --region=us-east1 \
  --project=noted-bliss-466410-q6

See: DEPLOYMENT_GUIDE.md for detailed instructions

Next Steps

Short-term (Dev/Staging)

Search service deployed and operational
VPC Access Connector configured
MongoDB connectivity via PSC
Deploy remaining services (catalog, identity, etc.)
Set up CI/CD pipelines for all services

Production Readiness

Complete MongoDB Atlas HA setup (register Service Attachments)
Remove NAT Gateway
Migrate secrets to Secret Manager
Set up Cloud Armor (DDoS protection)
Configure Cloud CDN
Set up alerting and monitoring dashboards
Implement rate limiting
Security audit and penetration testing

Documentation Index

ARCHITECTURE.md (this file) - System architecture overview
DEPLOYMENT_GUIDE.md - How to deploy services
NEW_SERVICE_TEMPLATE.md - Creating new microservices
CI_CD_GUIDE.md - CI/CD setup and workflows
services/search/CLAUDE.md - Search service documentation
services/search/docs/ - Detailed technical docs

Maintained by: Dev Team Last Review: 2025-11-19 Status: Living Document

CROP Microservices Architecture

On this page