Data Platform Engineer
Job Description
1. End-to-End Data Architecture & Engineering :
- Design, build, and maintain end-to-end data pipelines, covering CDC ingestion from production databases to production-grade lakehouse storage.
- Develop and scale workflow orchestration using Temporal and data transformation pipelines using dbt.
- Build standardized dimensional data models and optimized relational/materialized views to support analytics and business intelligence.
- Modernize legacy data workflows by replacing traditional DAG runners and cron-based batch jobs with scalable orchestration frameworks.
2. Multi-Region Architecture & Data Residency :
- Design and implement multi-region data architectures across cloud environments, ensuring compliance with data residency and sovereignty requirements.
- Establish secure data isolation mechanisms to ensure sensitive patient and transaction data remains within permitted geographical boundaries.
- Build compliant cross-region replication, synchronization, failover, and data-routing mechanisms.
3. AI/ML Integration & Multimodal Data Processing :
- Integrate machine learning models into data pipelines to identify and redact sensitive PII/PHI across text, images, audio, and video.
- Build and operationalize embedding generation, LLM inference, and semantic search pipelines.
- Develop scalable workflows for AI-powered data processing and multimodal content analysis.
4. Lakehouse Infrastructure & Platform Enablement :
- Design and optimize open lakehouse architectures using Apache Iceberg.
- Implement efficient partitioning, ACID transactions, and time-travel capabilities for large-scale datasets.
- Provision and manage secure, scalable Jupyter Notebook environments for data science and exploratory analytics.
5. Cloud Infrastructure, DevOps & FinOps :
- Design, deploy, and manage AWS infrastructure using Infrastructure as Code (Terraform or AWS CDK).
- Establish cost-monitoring frameworks to track cost-per-outcome across data and AI pipelines.
- Own production reliability through monitoring, alerting, data quality checks, and pipeline freshness tracking.