Predictor Service

A Go-based weather prediction service that downloads and processes GRIB files for wind vector data extraction.

Features

• GRIB File Processing: Downloads and processes GRIB weather data files
• Wind Vector Extraction: Extracts wind vector data for given coordinates and time
• Redis Caching: Caches extraction results for improved performance
• Distributed Locking: Uses Redis-based distributed locks for safe concurrent downloads
• Scheduled Updates: Automatic GRIB file updates via configurable scheduler
• REST API: HTTP API for data extraction and service management
• Modular Architecture: Clean separation of concerns with dependency injection

Architecture

The service follows a modular architecture with clear separation of concerns:

• Service Layer: Business logic and orchestration
• GRIB Package: GRIB file processing and data extraction
• Redis Package: Caching and distributed locking
• Scheduler Package: Job scheduling and execution
• Transport Layer: HTTP API handling
• Jobs: Background tasks (GRIB updates, etc.)

Quick Start with Docker

Prerequisites

• Docker
• Docker Compose

Running the Service

1. Clone the repository:

   git clone <repository-url>
   cd predictor
   

2. Start the services:

   # Production
   docker-compose up -d
   
   # Development (with volume mounts)
   docker-compose -f docker-compose.dev.yml up -d
   

3. Check service status:

   docker-compose ps
   

4. View logs:

   # All services
   docker-compose logs -f
   
   # Specific service
   docker-compose logs -f predictor
   docker-compose logs -f redis
   

Using Make Commands

The project includes a Makefile for common operations:

# Build and start services
make up-build

# View logs
make logs

# Stop services
make down

# Clean up everything
make clean

# Show all available commands
make help

Configuration

The service is configured via environment variables with the prefix GSN_PREDICTOR_:

GRIB Configuration

• GSN_PREDICTOR_GRIB_DIR: Directory for GRIB files (default: /tmp/grib)
• GSN_PREDICTOR_GRIB_TTL: GRIB file TTL (default: 24h)
• GSN_PREDICTOR_GRIB_CACHE_TTL: Cache TTL (default: 1h)
• GSN_PREDICTOR_GRIB_PARALLEL: Parallel download workers (default: 4)
• GSN_PREDICTOR_GRIB_TIMEOUT: Download timeout (default: 30s)
• GSN_PREDICTOR_GRIB_DATASET_URL: GRIB data source URL

Redis Configuration

• GSN_PREDICTOR_REDIS_HOST: Redis host (default: localhost)
• GSN_PREDICTOR_REDIS_PORT: Redis port (default: 6379)
• GSN_PREDICTOR_REDIS_PASSWORD: Redis password (default: empty)
• GSN_PREDICTOR_REDIS_DB: Redis database (default: 0)

Scheduler Configuration

• GSN_PREDICTOR_SCHEDULER_ENABLED: Enable scheduler (default: true)

GRIB Updater Job Configuration

• GSN_PREDICTOR_GRIB_UPDATER_INTERVAL: Update interval (default: 6h)
• GSN_PREDICTOR_GRIB_UPDATER_TIMEOUT: Update timeout (default: 45m)

REST Transport Configuration

• GSN_PREDICTOR_REST_HOST: HTTP host (default: 0.0.0.0)
• GSN_PREDICTOR_REST_PORT: HTTP port (default: 8080)
• GSN_PREDICTOR_REST_READ_TIMEOUT: Read timeout (default: 30s)
• GSN_PREDICTOR_REST_WRITE_TIMEOUT: Write timeout (default: 30s)
• GSN_PREDICTOR_REST_IDLE_TIMEOUT: Idle timeout (default: 60s)

API Endpoints

The service exposes a REST API for wind data extraction:

• GET /health - Health check endpoint
• POST /predict - Extract wind data for given coordinates and time

Example API Usage

# Health check
curl http://localhost:8080/health

# Extract wind data
curl -X POST http://localhost:8080/predict \
  -H "Content-Type: application/json" \
  -d '{
    "latitude": 40.7128,
    "longitude": -74.0060,
    "altitude": 100,
    "timestamp": "2024-01-15T12:00:00Z"
  }'

Development

Local Development

1. Install dependencies:

   go mod download
   

2. Start Redis:

   docker run -d --name redis -p 6379:6379 redis:7.2-alpine
   

3. Set environment variables:

   cd cmd/api
   source .env
   

4. Run the service:

   go run .
   

Building Locally

# Build the binary
make build-local

# Run tests
make test

# Format code
make fmt

# Lint code
make lint

Docker Development

For development with hot reloading:

# Start development environment
docker-compose -f docker-compose.dev.yml up -d

# View logs
docker-compose -f docker-compose.dev.yml logs -f predictor

Docker Best Practices

The Dockerfile follows Go best practices:

• Multi-stage build: Separate builder and runtime stages
• Non-root user: Runs as non-root user for security
• Minimal runtime: Uses Alpine Linux for smaller image size
• Health checks: Built-in health monitoring
• Optimized layers: Efficient layer caching
• Security: No unnecessary packages or permissions

Monitoring and Health Checks

The service includes built-in health checks:

• Application health: HTTP endpoint at /health
• Docker health: Container health check with wget
• Redis health: Redis ping health check
• Service dependencies: Proper startup order with health checks

Troubleshooting

Common Issues

1. Redis connection refused:

   • Ensure Redis is running: docker-compose ps
   • Check Redis logs: docker-compose logs redis
   • Verify network connectivity

2. GRIB download failures:

   • Check internet connectivity
   • Verify GRIB data source URL
   • Check disk space in /tmp/grib

3. Service not starting:

   • Check logs: docker-compose logs predictor
   • Verify environment variables
   • Check port conflicts

Debug Commands

# Execute shell in container
docker-compose exec predictor sh

# Check Redis connectivity
docker-compose exec redis redis-cli ping

# View container resources
docker stats

# Check network connectivity
docker-compose exec predictor wget -O- http://redis:6379

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Run tests and linting
6. Submit a pull request

License

[Add your license information here]