Senior Data Engineer for Mission-Critical Analytics Feed Build

CoreData Systems

Contract · Remote · Flexible hours

About the role

CoreData Systems is building a high‑throughput data platform to accelerate product releases. As a Data Engineer you will drive a CI/CD and platform automation sprint that reduces deployment latency and improves data pipeline reliability. You will work on the internal project code‑named "Velocity Stream" to deliver faster, repeatable releases.

Key responsibilities

• Design and implement Terraform scripts to provision AWS Redshift clusters and S3 data lakes in a fully automated pipeline.
• Build Airflow DAGs that trigger ETL jobs on every code push, integrating with GitHub Actions for end‑to‑end testing.
• Containerize Spark jobs using Docker and orchestrate them with Kubernetes, ensuring zero‑downtime rollouts.
• Develop Python‑based data validation frameworks that run as pre‑deployment checks in the CI pipeline.
• Collaborate with DevOps to embed dbt model testing into the release workflow, targeting a 30% reduction in post‑release defects.
• Document the CI/CD workflow, create runbooks, and train the data team on new automation tools.

Must-have skills

• Advanced Python programming for data pipelines.
• Strong experience with PostgreSQL and MySQL schema design.
• Proficiency in AWS services (S3, Redshift, Lambda, CloudFormation/Terraform).
• Hands‑on knowledge of CI/CD tools such as GitHub Actions, Jenkins, or GitLab CI.
• Familiarity with container orchestration (Docker, Kubernetes) and Airflow.

Nice to have

• Experience with dbt and data testing frameworks.
• Knowledge of Spark or Flink for large‑scale processing.