Reactive Growth Microservices

A production-ready, cloud-native microservices architecture demonstrating modern backend development practices with reactive programming, event-driven design, and comprehensive observability.

Table of Contents

• Overview
• Architecture
• Tech Stack
• Microservices
• Key Features
• Prerequisites
• Getting Started
• API Documentation
• Monitoring & Observability
• Testing
• Project Structure
• Development
• Contributing

Overview

This project showcases a complete microservices ecosystem built with Kotlin and Spring Boot WebFlux, featuring fully reactive, non-blocking communication patterns. The system demonstrates enterprise-grade patterns including API gateway routing, event-driven architecture with Kafka, distributed tracing, and comprehensive monitoring.

What Makes This Project Stand Out

• 100% Reactive: End-to-end non-blocking architecture using Spring WebFlux and R2DBC
• Modern API Gateway: Armeria-based gateway with gRPC support, rate limiting, and circuit breakers
• Event-Driven: Asynchronous inter-service communication via Kafka
• Full Observability Stack: Integrated OpenTelemetry, Jaeger, Zipkin, Tempo, Grafana, and Prometheus
• Production-Ready: Includes health checks, metrics, distributed tracing, and proper error handling
• Type-Safe Database Access: JOOQ integration with automatic schema generation from Flyway migrations
• Container-First: Complete Docker Compose setup for local development and testing

Architecture

┌─────────────────┐
│   API Gateway   │  (Armeria - Port 5555)
│   Rate Limiting │
│   Load Balancing│
└────────┬────────┘
         │ gRPC
    ┌────┴──────────────┬──────────────┐
    │                   │              │
┌───▼────┐      ┌───────▼──┐    ┌─────▼─────┐
│Product │      │Inventory │    │   Order   │
│Service │      │ Service  │    │  Service  │
│:7001   │      │  :7002   │    │   :7003   │
└───┬────┘      └────┬─────┘    └─────┬─────┘
    │                │                │
    └────────┬───────┴────────────────┘
             │ Events
        ┌────▼─────┐
        │  Kafka   │
        └──────────┘
             │
    ┌────────┴────────────┐
    │                     │
┌───▼────────┐    ┌───────▼──────┐
│ PostgreSQL │    │ Observability│
│  Database  │    │   Stack      │
└────────────┘    └──────────────┘

Communication Patterns

• Synchronous: gRPC for low-latency request/response between API Gateway and services
• Asynchronous: Kafka for event-driven communication between microservices
• Database: R2DBC (reactive PostgreSQL driver) for non-blocking database operations

Tech Stack

Core Technologies

Category	Technology	Version
Language	Kotlin	2.0.21
Runtime	JVM	21
Framework	Spring Boot	3.3.5
Reactive Stack	Spring WebFlux	3.3.5
API Gateway	Armeria	1.30.1
Database	PostgreSQL	16-alpine
DB Migration	Flyway	10.17.3
DB Access	JOOQ	3.19.15
DB Driver	R2DBC PostgreSQL	Latest
Message Broker	Apache Kafka	7.7.1
RPC Protocol	gRPC	1.68.1
Build Tool	Gradle Kotlin DSL	8.x

Observability & Monitoring

Component	Technology	Purpose
Distributed Tracing	OpenTelemetry	Modern telemetry collection
Trace Visualization	Jaeger	Trace analysis and debugging
Trace Visualization	Zipkin	Alternative trace viewer
Trace Backend	Tempo	Grafana's trace storage
Metrics Collection	Prometheus	Time-series metrics database
Metrics Visualization	Grafana	Dashboards and alerting
Log Aggregation	Loki	Log collection and querying
Telemetry Pipeline	OTEL Collector	Centralized telemetry processing
Metrics Export	Micrometer	Application metrics

Testing & Quality

• JUnit 5: Unit testing framework
• Testcontainers: Integration testing with real dependencies (PostgreSQL, Kafka)
• Reactor Test: Reactive streams testing
• MockK: Kotlin-friendly mocking
• JaCoCo: Code coverage reporting

Additional Libraries

• Spring Security: Authentication and authorization
• Spring Cloud Sleuth: Distributed tracing integration
• Logstash Logback Encoder: Structured JSON logging
• Kotlin Coroutines: Async programming support
• Jackson Kotlin Module: JSON serialization
• SpringDoc OpenAPI: API documentation

Microservices

1. Product Service

Port: 7001 (HTTP), 7071 (gRPC), 7771 (Actuator)

Manages product catalog with full CRUD operations.

Features:

• Product creation, retrieval, update, and deletion
• gRPC endpoint for inter-service communication
• Kafka event publishing on product changes
• PostgreSQL database with R2DBC
• JOOQ type-safe queries

Endpoints:

• GET /products - List all products
• GET /products/{id} - Get product by ID
• POST /products - Create new product
• PUT /products/{id} - Update product
• DELETE /products/{id} - Delete product

2. Inventory Service

Port: 7002 (HTTP), 7072 (gRPC), 7772 (Actuator)

Tracks product inventory and stock levels.

Features:

• Real-time inventory management
• Stock reservation and release
• Kafka integration for order processing
• Reactive stock validation
• Automatic inventory creation on new products

Key Workflows:

• Listens to product creation events
• Validates inventory for order placement
• Publishes inventory validation results

3. Order Service

Port: 7003 (HTTP), 7073 (gRPC), 7773 (Actuator)

Handles customer orders and order lifecycle.

Features:

• Order creation and management
• Event-driven order processing
• Integration with inventory validation
• Order status tracking
• Transactional consistency with Saga pattern

Order Flow:

1. Receive order creation request
2. Publish order validation event to Kafka
3. Wait for inventory validation
4. Update order status based on validation result
5. Notify customer

4. Armeria API Gateway

Port: 5555

High-performance API gateway built on Armeria framework.

Features:

• gRPC client aggregation
• Rate limiting with Bucket4j
• Circuit breaker pattern
• Request/response logging
• Prometheus metrics endpoint
• Health check aggregation
• Composite service responses
• HTTP/2 and HTTP/3 support

Capabilities:

• Route requests to appropriate microservices
• Aggregate responses from multiple services
• Handle cross-cutting concerns (auth, logging, monitoring)
• Protocol translation (REST to gRPC)

Key Features

1. Reactive Architecture

• Non-blocking I/O: All services use Spring WebFlux for async, non-blocking operations
• Backpressure Handling: Proper reactive streams implementation
• R2DBC: Reactive database driver for PostgreSQL
• Reactive Kafka: Non-blocking Kafka producers and consumers

2. Event-Driven Design

• Kafka Integration: Decoupled service communication
• Event Sourcing: Track state changes through events
• CQRS Pattern: Separate read and write operations where beneficial
• Saga Pattern: Distributed transaction management

3. Observability

Distributed Tracing

• OpenTelemetry: Industry-standard telemetry collection
• Multiple Backends: Jaeger, Zipkin, and Tempo for trace analysis
• Context Propagation: Automatic trace context across service boundaries
• gRPC Tracing: Full visibility into gRPC calls

Metrics

• Prometheus Integration: Scrape metrics from all services
• Custom Metrics: Business-specific metrics (orders/sec, inventory levels)
• JVM Metrics: Heap, GC, threads, and performance metrics
• Kafka Metrics: Consumer lag, throughput, error rates

Logging

• Structured Logging: JSON format via Logstash encoder
• Correlation IDs: Track requests across services
• Loki Integration: Centralized log aggregation
• Log Levels: Configurable per package/class

Visualization

• Grafana Dashboards: Pre-configured dashboards for each service
• Alerting: Prometheus AlertManager integration
• Service Maps: Visualize service dependencies in Jaeger
• Real-time Monitoring: Live metrics and traces

4. Type-Safe Database Operations

• JOOQ Code Generation: Generate type-safe DAO from database schema
• Flyway Migrations: Version-controlled schema management
• Testcontainers: Automatic PostgreSQL container for code generation
• R2DBC Integration: Reactive database access

5. API Contract & Documentation

• Protocol Buffers: Type-safe gRPC service definitions
• OpenAPI/Swagger: REST API documentation
• Actuator Endpoints: Health, info, metrics endpoints

6. Resilience Patterns

• Rate Limiting: Prevent service overload (Bucket4j)
• Circuit Breaker: Fail fast when downstream services are unavailable
• Retry Logic: Automatic retry with exponential backoff
• Timeout Management: Prevent cascade failures

7. Security

• Spring Security: Authentication and authorization
• TLS Support: Secure communication between services
• Secure Configuration: Externalized secrets management

Prerequisites

• JDK 21 or higher
• Docker and Docker Compose
• Gradle (wrapper included)
• 8GB RAM minimum (for running all services)

Getting Started

1. Clone the Repository

git clone <repository-url>
cd reactive-growth-microservices

2. Start Infrastructure Services

docker-compose up -d postgres-reactive \
                     zookeeper-reactive \
                     kafka-reactive \
                     kafka-ui-reactive \
                     otel-collector-reactive \
                     jaeger-reactive \
                     zipkin-reactive \
                     tempo-reactive \
                     loki-reactive \
                     prometheus-reactive \
                     grafana-reactive

Wait for services to be healthy (approximately 30-60 seconds)

3. Build All Services

./gradlew clean build

This will:

• Compile Kotlin code
• Generate gRPC stubs from .proto files
• Run Flyway migrations
• Generate JOOQ classes
• Run unit and integration tests
• Generate code coverage reports

4. Run Services Individually

Product Service

cd product-service
./gradlew bootRun

Inventory Service

cd inventory-service
./gradlew bootRun

Order Service

cd order-service
./gradlew bootRun

API Gateway

cd armeria-api-gateway
./gradlew run

5. Run Everything with Docker Compose

docker-compose up --build

This will build and start all services in containers.

API Documentation

Access Points

Service	URL	Description
API Gateway	http://localhost:5555	Main entry point
Product Service	http://localhost:7001	Direct product access
Inventory Service	http://localhost:7002	Direct inventory access
Order Service	http://localhost:7003	Direct order access
Swagger UI (Product)	http://localhost:7001/swagger-ui.html	API docs

Sample API Calls

Create Product

curl -X POST http://localhost:5555/api/products \
  -H "Content-Type: application/json" \
  -d '{
    "name": "MacBook Pro M3",
    "description": "Latest MacBook Pro with M3 chip",
    "price": 2499.99,
    "category": "Electronics"
  }'

Get Product

curl http://localhost:5555/api/products/1

Create Order

curl -X POST http://localhost:5555/api/orders \
  -H "Content-Type: application/json" \
  -d '{
    "productId": 1,
    "quantity": 2,
    "customerId": "customer-123"
  }'

Check Inventory

curl http://localhost:5555/api/inventory/product/1

Monitoring & Observability

Access Dashboards

Tool	URL	Credentials	Purpose
Grafana	http://localhost:3000	admin/grafana	Metrics dashboards
Jaeger	http://localhost:16686	-	Distributed tracing
Zipkin	http://localhost:9411	-	Alternative trace viewer
Prometheus	http://localhost:9090	-	Metrics database
Kafka UI	http://localhost:9099	-	Kafka topic browser

Key Metrics to Monitor

1. Service Health

   • Actuator: http://localhost:7771/actuator/health (Product)
   • Actuator: http://localhost:7772/actuator/health (Inventory)
   • Actuator: http://localhost:7773/actuator/health (Order)

2. Prometheus Metrics

   • http://localhost:7771/actuator/prometheus (Product)
   • http://localhost:7772/actuator/prometheus (Inventory)
   • http://localhost:7773/actuator/prometheus (Order)

3. Application Metrics

   • Request rates and latencies
   • Error rates (4xx, 5xx)
   • Database connection pool stats
   • JVM memory and GC metrics
   • Kafka consumer lag

Tracing Example

After making API calls, view traces in:

• Jaeger: http://localhost:16686 → Select service → Find traces
• Zipkin: http://localhost:9411 → Query for traces

Traces show:

• End-to-end request flow through services
• Individual span timings
• gRPC call details
• Kafka message propagation
• Database query performance

Testing

Run Unit Tests

./gradlew test

Run Integration Tests

./gradlew integrationTest

Integration tests use Testcontainers to spin up:

• PostgreSQL database
• Kafka broker
• Required dependencies

Run All Tests with Coverage

./gradlew clean test integrationTest jacocoTestReport

Coverage reports: build/jacocoHtml/index.html

Test Categories

1. Unit Tests: Business logic, validation, transformations
2. Integration Tests: Database operations, Kafka messaging, external integrations
3. Contract Tests: gRPC service contracts
4. Reactive Tests: Reactive streams behavior and backpressure

Project Structure

reactive-growth-microservices/
├── armeria-api-gateway/          # API Gateway service
│   ├── src/
│   │   ├── main/kotlin/          # Gateway implementation
│   │   └── main/proto/           # gRPC service definitions
│   └── build.gradle.kts
│
├── product-service/              # Product microservice
│   ├── src/
│   │   ├── main/kotlin/          # Service implementation
│   │   ├── main/resources/
│   │   │   ├── application.yml   # Configuration
│   │   │   └── db/migration/     # Flyway SQL scripts
│   │   ├── main/proto/           # gRPC contracts
│   │   ├── test/                 # Unit tests
│   │   └── integration-test/     # Integration tests
│   └── build.gradle.kts
│
├── inventory-service/            # Inventory microservice
│   └── [similar structure]
│
├── order-service/                # Order microservice
│   └── [similar structure]
│
├── docker-compose-init/          # Configuration files
│   ├── grafana/                  # Grafana datasources
│   ├── otel-collector/           # OpenTelemetry config
│   ├── postgres/                 # Database init scripts
│   ├── prometheus/               # Prometheus config
│   └── tempo/                    # Tempo config
│
├── docker-compose.yaml           # Complete stack definition
├── settings.gradle.kts           # Multi-module project setup
└── README.md                     # This file

Service Internal Structure

Each microservice follows clean architecture:

service/src/main/kotlin/com/reactive/{service}/
├── {Service}Application.kt      # Spring Boot entry point
├── config/                       # Configuration classes
│   ├── DatabaseConfig.kt
│   ├── KafkaConfig.kt
│   └── SecurityConfig.kt
├── module/
│   └── {domain}/                 # Domain module (e.g., product, order)
│       ├── controller/           # REST controllers
│       ├── service/              # Business logic
│       ├── repository/           # Data access
│       ├── handler/
│       │   └── grpc/             # gRPC service implementations
│       ├── event/
│       │   └── kafka/            # Kafka producers/consumers
│       ├── dto/                  # Data transfer objects
│       └── entity/               # Domain entities
└── database/
    └── jooq/                     # Generated JOOQ classes

Development

Adding a New Service

1. Create new module in settings.gradle.kts
2. Copy and modify build.gradle.kts from existing service
3. Define database schema in Flyway migrations
4. Generate JOOQ classes: ./gradlew jooqCodegen
5. Define gRPC contract in .proto files
6. Implement business logic following reactive patterns
7. Add service to docker-compose.yaml
8. Configure observability (OTEL, metrics, health checks)

Database Schema Changes

1. Create new migration: src/main/resources/db/migration/V{version}__{description}.sql
2. Regenerate JOOQ: ./gradlew jooqCodegen
3. Update service layer to use new schema

gRPC Service Changes

1. Update .proto files in src/main/proto/
2. Regenerate stubs: ./gradlew generateProto
3. Implement new methods in gRPC service implementations
4. Update API Gateway if needed

Kafka Topics

Current topics:

• product-events: Product creation/update/deletion
• order-events: Order placement and status changes
• inventory-events: Stock level changes
• inventory-validation-events: Inventory check results

Add new topics in:

1. Service configuration (application.yml)
2. Kafka config class
3. Producer/Consumer implementations
4. Docker Compose Kafka UI configuration

Performance Characteristics

Benchmarks (Local Testing)

Metric	Value
API Gateway Throughput	~10,000 req/sec
Average Latency (p50)	~5ms
p95 Latency	~15ms
p99 Latency	~50ms
Concurrent Connections	1000+
Database Connections	40 per service

Scalability

• Horizontal Scaling: All services are stateless and can scale horizontally
• Database: PostgreSQL with connection pooling (40 connections per service)
• Kafka: Partitioned topics for parallel processing
• Resource Usage: Each service ~500MB RAM under normal load

Contributing

1. Fork the repository
2. Create feature branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open Pull Request

Code Style

• Follow Kotlin coding conventions
• Use meaningful variable and function names
• Write tests for new features
• Maintain > 80% code coverage
• Update documentation

Troubleshooting

Common Issues

Issue: Services can't connect to PostgreSQL

# Check if PostgreSQL is running
docker ps | grep postgres-reactive

# Check logs
docker logs postgres-reactive

# Verify connection
docker exec -it postgres-reactive psql -U postgres -l

Issue: Kafka connection errors

# Restart Kafka stack
docker-compose restart zookeeper-reactive kafka-reactive

# Check topic creation
docker exec -it kafka-reactive kafka-topics --list --bootstrap-server localhost:9092

Issue: JOOQ code generation fails

# Clean and regenerate
./gradlew clean tc-start flywayMigrate jooqCodegen tc-stop

Issue: Out of memory errors

# Increase Gradle heap
export GRADLE_OPTS="-Xmx4g -XX:MaxMetaspaceSize=512m"

# Or edit gradle.properties
echo "org.gradle.jvmargs=-Xmx4g -XX:MaxMetaspaceSize=512m" >> gradle.properties

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

For questions or feedback, please open an issue in the repository.

---

Built with Kotlin + Spring Boot WebFlux + Armeria + Kafka + PostgreSQL + OpenTelemetry

Demonstrating production-ready microservices architecture with reactive programming, event-driven design, and comprehensive observability.