1. Lakehouse‑Based Data Engineering (AWS & GCP)
- Design and implement Lakehouse architectures using object storage and open table formats (e.g., Apache Iceberg) to support ACID transactions, schema evolution, and time‑travel.
- Build and maintain batch and streaming data pipelines on:
- AWS (e.g., S3, Glue, Kinesis, Spark)
- GCP (e.g., GCS, Dataproc, Dataflow, Pub/Sub)
- Implement medallion architecture patterns (raw / bronze → silver → gold) consistently across clouds.
2. Data Product Development
- Develop curated, governed, and consumption‑ready data products aligned to business domains.
- Partner with Data Product Owners and stakeholders to translate requirements into robust technical implementations.
- Ensure data products are discoverable, reusable, and well‑documented, supporting analytics and downstream AI/ML use cases.
3. Data Modeling & Quality
- Design and maintain analytical data models optimized for BI, reporting, and advanced analytics.
- Implement data quality checks, validation rules, and monitoring within pipelines.
- Manage schema evolution and ensure adherence to enterprise data standards across AWS and GCP.
4. DataOps, CI/CD & Automation
- Apply DataOps and DevOps best practices by implementing CI/CD pipelines for data pipelines and data products.
- Enable automated testing, deployment, and rollback across cloud environments.
- Collaborate with platform and DevOps teams to improve observability, reliability, and operational maturity.
5. Performance, Reliability & Cost Optimization
- Optimize pipelines for performance, scalability, and cost efficiency on both AWS and GCP.
- Monitor and tune compute and storage usage in collaboration with platform and FinOps teams.
- Troubleshoot and resolve complex production issues across multi‑cloud environments.
6. Technical Leadership & Collaboration
- Act as a senior technical contributor and mentor for junior and mid‑level data engineers.
- Participate in design and code reviews to maintain high engineering standards.
- Contribute to reusable frameworks, templates, and best practices for data engineering.
Official notification