Lead Big Data Architect – Warehouse Design

Silverline Labs

Contract · Remote · Flexible hours

About the role

Silverline Labs is building a greenfield SaaS platform to deliver real‑time analytics to globally distributed users. As the Big Data Architect you will define the data foundation for the Nimbus Stream module, shaping ingestion pipelines, storage strategy, and analytics layer from day one.

Key responsibilities

• Design and implement a scalable data lake on AWS S3 with partitioning schemes tailored for high‑velocity event streams.
• Architect end‑to‑end ETL workflows using Apache Spark and Airflow, targeting sub‑second latency for the Nimbus Stream dashboard.
• Select and configure a columnar warehouse (Redshift) and define data models that support ad‑hoc querying and machine‑learning feature extraction.
• Establish data governance policies, schema evolution processes, and CI/CD pipelines for data artifacts using Docker and GitLab CI.
• Collaborate with product engineers to prototype real‑time feature stores and ensure data quality through automated testing.

Must-have skills

• Deep expertise in designing data pipelines with Spark, Airflow, and Kafka.
• Strong command of AWS services (S3, Redshift, Glue, Kinesis).
• Proficiency in Python for data engineering and orchestration.
• Experience with PostgreSQL for transactional data integration.
• Containerization with Docker and orchestration knowledge.

Nice to have

• Familiarity with Terraform for infrastructure as code.
• Exposure to real‑time analytics frameworks such as Flink.