AWS SFTP-to-S3 Data Pipeline
Data Engineering Mynehub Ventures August 2023 - October 2024Background workers to parse DAT files from S3, publish JSON to SQS, process business logic, and trigger SMS notifications.
AWS SFTP-to-S3 Data Pipeline
Built data ingestion pipeline for Mynehub Ventures using AWS services.
Components
- AWS Lambda functions for SFTP Transfer Family-to-S3 workflows
- SQS consumers/publishers with .NET 8 Minimal APIs
- Background workers for DAT file parsing
- SMS notification triggers
Architecture
- Clean Architecture with CQRS
- Static Logging, OTEL, Grafana monitoring
- k6 load testing, xUnit/integration tests
DevOps
EC2 deployments via Jenkins, Jira Kanban workflow.