Lead Big Data Architect Needed for Stream Processing Setup

CloudWave Digital

Contract · Remote · Flexible hours

About the role

CloudWave Digital is building a next‑generation analytics platform. As a Big Data Architect you will lead the test automation uplift for the new "Nebula Sync" module, aiming to cut regression risk before each product launch. The work blends large‑scale data pipeline design with rigorous CI‑based validation.

Key responsibilities

• Design end‑to‑end data architecture for Nebula Sync, including schema evolution and partition strategy on AWS Redshift.
• Implement automated regression test suites using Python, PyTest and Docker containers that validate data integrity after each pipeline change.
• Integrate test pipelines into Jenkins/CodePipeline, ensuring nightly runs and fast feedback loops.
• Collaborate with data engineers to embed data quality checks (Deequ, Great Expectations) into ETL jobs.
• Document architecture decisions, test coverage metrics, and risk mitigation plans for stakeholder reviews.
• Mentor junior engineers on best practices for data testing and observability.

Must-have skills

• Deep expertise in designing scalable data lakes and warehouses on AWS.
• Proficiency with Python for data processing and test automation.
• Strong knowledge of PostgreSQL and SQL performance tuning.
• Hands‑on experience with Docker and container orchestration for reproducible test environments.
• Track record of implementing CI/CD for data pipelines.

Nice to have

• Familiarity with Apache Spark or Flink for streaming validation.
• Experience with Terraform for infrastructure as code.