feat: downloader

2025-06-23 04:19:26 +03:00 · 2025-06-23 04:19:26 +03:00 · 42e7924be9
commit 42e7924be9
parent b9c1a98895
37 changed files with 2422 additions and 94 deletions
--- a/README.md
+++ b/README.md
@ -0,0 +1,261 @@
+# Predictor Service
+
+A Go-based weather prediction service that downloads and processes GRIB files for wind vector data extraction.
+
+## Features
+
+- **GRIB File Processing**: Downloads and processes GRIB weather data files
+- **Wind Vector Extraction**: Extracts wind vector data for given coordinates and time
+- **Redis Caching**: Caches extraction results for improved performance
+- **Distributed Locking**: Uses Redis-based distributed locks for safe concurrent downloads
+- **Scheduled Updates**: Automatic GRIB file updates via configurable scheduler
+- **REST API**: HTTP API for data extraction and service management
+- **Modular Architecture**: Clean separation of concerns with dependency injection
+
+## Architecture
+
+The service follows a modular architecture with clear separation of concerns:
+
+- **Service Layer**: Business logic and orchestration
+- **GRIB Package**: GRIB file processing and data extraction
+- **Redis Package**: Caching and distributed locking
+- **Scheduler Package**: Job scheduling and execution
+- **Transport Layer**: HTTP API handling
+- **Jobs**: Background tasks (GRIB updates, etc.)
+
+## Quick Start with Docker
+
+### Prerequisites
+
+- Docker
+- Docker Compose
+
+### Running the Service
+
+1. **Clone the repository:**
+   ```bash
+   git clone <repository-url>
+   cd predictor
+   ```
+
+2. **Start the services:**
+   ```bash
+   # Production
+   docker-compose up -d
+   
+   # Development (with volume mounts)
+   docker-compose -f docker-compose.dev.yml up -d
+   ```
+
+3. **Check service status:**
+   ```bash
+   docker-compose ps
+   ```
+
+4. **View logs:**
+   ```bash
+   # All services
+   docker-compose logs -f
+   
+   # Specific service
+   docker-compose logs -f predictor
+   docker-compose logs -f redis
+   ```
+
+### Using Make Commands
+
+The project includes a Makefile for common operations:
+
+```bash
+# Build and start services
+make up-build
+
+# View logs
+make logs
+
+# Stop services
+make down
+
+# Clean up everything
+make clean
+
+# Show all available commands
+make help
+```
+
+## Configuration
+
+The service is configured via environment variables with the prefix `GSN_PREDICTOR_`:
+
+### GRIB Configuration
+- `GSN_PREDICTOR_GRIB_DIR`: Directory for GRIB files (default: `/tmp/grib`)
+- `GSN_PREDICTOR_GRIB_TTL`: GRIB file TTL (default: `24h`)
+- `GSN_PREDICTOR_GRIB_CACHE_TTL`: Cache TTL (default: `1h`)
+- `GSN_PREDICTOR_GRIB_PARALLEL`: Parallel download workers (default: `4`)
+- `GSN_PREDICTOR_GRIB_TIMEOUT`: Download timeout (default: `30s`)
+- `GSN_PREDICTOR_GRIB_DATASET_URL`: GRIB data source URL
+
+### Redis Configuration
+- `GSN_PREDICTOR_REDIS_HOST`: Redis host (default: `localhost`)
+- `GSN_PREDICTOR_REDIS_PORT`: Redis port (default: `6379`)
+- `GSN_PREDICTOR_REDIS_PASSWORD`: Redis password (default: empty)
+- `GSN_PREDICTOR_REDIS_DB`: Redis database (default: `0`)
+
+### Scheduler Configuration
+- `GSN_PREDICTOR_SCHEDULER_ENABLED`: Enable scheduler (default: `true`)
+
+### GRIB Updater Job Configuration
+- `GSN_PREDICTOR_GRIB_UPDATER_INTERVAL`: Update interval (default: `6h`)
+- `GSN_PREDICTOR_GRIB_UPDATER_TIMEOUT`: Update timeout (default: `45m`)
+
+### REST Transport Configuration
+- `GSN_PREDICTOR_REST_HOST`: HTTP host (default: `0.0.0.0`)
+- `GSN_PREDICTOR_REST_PORT`: HTTP port (default: `8080`)
+- `GSN_PREDICTOR_REST_READ_TIMEOUT`: Read timeout (default: `30s`)
+- `GSN_PREDICTOR_REST_WRITE_TIMEOUT`: Write timeout (default: `30s`)
+- `GSN_PREDICTOR_REST_IDLE_TIMEOUT`: Idle timeout (default: `60s`)
+
+## API Endpoints
+
+The service exposes a REST API for wind data extraction:
+
+- `GET /health` - Health check endpoint
+- `POST /predict` - Extract wind data for given coordinates and time
+
+### Example API Usage
+
+```bash
+# Health check
+curl http://localhost:8080/health
+
+# Extract wind data
+curl -X POST http://localhost:8080/predict \
+  -H "Content-Type: application/json" \
+  -d '{
+    "latitude": 40.7128,
+    "longitude": -74.0060,
+    "altitude": 100,
+    "timestamp": "2024-01-15T12:00:00Z"
+  }'
+```
+
+## Development
+
+### Local Development
+
+1. **Install dependencies:**
+   ```bash
+   go mod download
+   ```
+
+2. **Start Redis:**
+   ```bash
+   docker run -d --name redis -p 6379:6379 redis:7.2-alpine
+   ```
+
+3. **Set environment variables:**
+   ```bash
+   cd cmd/api
+   source .env
+   ```
+
+4. **Run the service:**
+   ```bash
+   go run .
+   ```
+
+### Building Locally
+
+```bash
+# Build the binary
+make build-local
+
+# Run tests
+make test
+
+# Format code
+make fmt
+
+# Lint code
+make lint
+```
+
+### Docker Development
+
+For development with hot reloading:
+
+```bash
+# Start development environment
+docker-compose -f docker-compose.dev.yml up -d
+
+# View logs
+docker-compose -f docker-compose.dev.yml logs -f predictor
+```
+
+## Docker Best Practices
+
+The Dockerfile follows Go best practices:
+
+- **Multi-stage build**: Separate builder and runtime stages
+- **Non-root user**: Runs as non-root user for security
+- **Minimal runtime**: Uses Alpine Linux for smaller image size
+- **Health checks**: Built-in health monitoring
+- **Optimized layers**: Efficient layer caching
+- **Security**: No unnecessary packages or permissions
+
+## Monitoring and Health Checks
+
+The service includes built-in health checks:
+
+- **Application health**: HTTP endpoint at `/health`
+- **Docker health**: Container health check with wget
+- **Redis health**: Redis ping health check
+- **Service dependencies**: Proper startup order with health checks
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Redis connection refused:**
+   - Ensure Redis is running: `docker-compose ps`
+   - Check Redis logs: `docker-compose logs redis`
+   - Verify network connectivity
+
+2. **GRIB download failures:**
+   - Check internet connectivity
+   - Verify GRIB data source URL
+   - Check disk space in `/tmp/grib`
+
+3. **Service not starting:**
+   - Check logs: `docker-compose logs predictor`
+   - Verify environment variables
+   - Check port conflicts
+
+### Debug Commands
+
+```bash
+# Execute shell in container
+docker-compose exec predictor sh
+
+# Check Redis connectivity
+docker-compose exec redis redis-cli ping
+
+# View container resources
+docker stats
+
+# Check network connectivity
+docker-compose exec predictor wget -O- http://redis:6379
+```
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Add tests
+5. Run tests and linting
+6. Submit a pull request
+
+## License
+
+[Add your license information here]