Data Engineer

amazon | Experience: 3+ Years | Chennai, Tamil Nadu, India | Posted 6 hours ago

Job Description

 You will build the foundational data layer that agentic systems depend on — schema-aware, version-controlled, and optimized for programmatic access.

2) You will engineer event-driven pipelines (Spark Structured Streaming, Kafka, Kinesis) that serve as the nervous system for agentic workflows — providing the low-latency data feeds that agents consume to trigger actions, detect anomalies, and orchestrate downstream processes.

3) You will define how agents discover data, resolve ambiguity across metric definitions, and compose multi-step analytical queries against our Curated Data Layer (CDL).

4) You will build intelligent agents that own the full DataOps loop: auto-detecting schema drift, self-healing broken pipelines, orchestrating metric onboarding workflows, running data quality checks, and publishing validated datasets — all with minimal human intervention.

5) You will design the trust layer for agentic data systems: output validation agents that verify SQL correctness before execution, confidence scoring for AI-generated metric definitions, automated regression testing against known-good baselines, and escalation workflows that route low-confidence outputs to human reviewers.

6) You will lead the transformation from static, manually-maintained pipelines to agent-operated infrastructure — migrating legacy ETL to Iceberg-based incremental processing, unifying metric definitions across 12+ surfaces through agent-driven reconciliation, and building the operational backbone that enables 4,000+ metrics to be governed, published, and consumed by both humans and AI agents.

⚡ Hot DevOps & Cloud Jobs Trending Now

SRE
Sr. SRE Engineer
Stripe | Bangalore
DEV
Backend Developer
Coinbase | Remote
Infra
Cloud Infra Lead
Datadog | Pune
ML
MLOps Architect
Anthropic | Hyderabad
SRE
Sr. SRE Engineer
Stripe | Bangalore
DEV
Backend Developer
Coinbase | Remote
Infra
Cloud Infra Lead
Datadog | Pune
ML
MLOps Architect
Anthropic | Hyderabad