Senior/Lead Big Data Engineer with Hadoop,Hive,Spark,SQL expertise
Job Description
Platform Optimization & Governance
Performance Tuning: Proactively troubleshoot and fine-tune complex Spark and Hive/HQL jobs to reduce latency, optimize memory usage, and manage cluster resource allocation (YARN).
Data Quality & Reliability: Establish automated data validation frameworks, lineage tracking, and strict SLA monitoring to guarantee data accuracy and platform reliability.
Platform Operations: Oversee on-prem big data platform stability, capacity planning, and workflow orchestration using Airflow.
Team Building & Technical Leadership
Build & Scale the Team: Partner with leadership to hire, onboard, and build a strong data engineering team from the ground up.
Mentorship & Guidance: Provide hands-on technical leadership, conduct architecture and code reviews, and mentor junior and mid-level engineers.
Engineering Standards: Establish and enforce best practices around Git version control, code quality, documentation, and Agile delivery methodologies.
Architecture & Pipeline Development
End-to-End ETL Ownership: Architect, develop, and maintain scalable, fault-tolerant batch ETL/ELT pipelines across on-premises Hadoop-based data platforms.
Large-Scale Data Processing: Design high-throughput data ingestion and transformation workflows for massive customer and behavioral datasets.
Data Modeling: Lead dimensional and relational data modeling efforts to support enterprise analytics, reporting, and downstream AI solutions.
Functional Alignment & Cross-Team Collaboration
Business & Functional Translation: Translate complex functional business requirements into scalable technical data architectures, ensuring data assets directly solve business challenges.
Cross-Functional Partnership: Collaborate closely with analytics, product, and AI teams to understand data needs and deliver reliable, analytics-ready datasets.