Lead Big Data Architect Needed for Mission-Critical Data Pipeline Build

Keystone Digital

Contract · Remote · Flexible hours

About the role

Keystone Digital is building a unified data backbone to streamline billing, CRM, and internal service APIs. As a Big Data Architect you will design and lead the implementation of the "FusionSync" integration layer, ensuring data consistency and real‑time flow across systems.

Key responsibilities

• Architect the end‑to‑end data pipeline for FusionSync using Apache Spark and Kafka to ingest billing events, CRM updates, and API logs.
• Define data models in PostgreSQL and Snowflake that support transactional consistency and analytical queries.
• Implement CI/CD for data jobs with Docker, GitLab CI, and Terraform on AWS.
• Set up monitoring, alerting, and data quality checks using Grafana, Prometheus, and Great Expectations.
• Collaborate with billing and CRM product owners to map source schemas to the unified model and deliver milestone‑driven prototypes.
• Document architecture decisions, runbooks, and hand‑off procedures for the internal engineering team.

Must-have skills

• Deep expertise in big data frameworks (Spark, Flink, Kafka).
• Strong data modeling and ETL design experience with PostgreSQL and Snowflake.
• Extensive AWS experience (EMR, Redshift, S3, IAM).
• Proficiency in Python for data processing and orchestration.
• Containerization with Docker and infrastructure as code using Terraform.

Nice to have

• Experience with real‑time analytics platforms such as Kinesis or Druid.
• Knowledge of GDPR and data governance best practices.