Senior Data Engineer

Technical Leadership

•             Lead a team of Data Engineers through assessment, development, migration, validation and production execution.

•             Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns.

•             Own technical estimates, engineering backlog, work allocation and technical delivery.

•             Conduct design and code reviews and enforce engineering standards.

•             Mentor Data Engineers and provide hands-on technical guidance.

•             Act as the primary engineering escalation point for complex migration and production issues.

•             Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.

•             Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions.

Hadoop-to-Iceberg Migration

•             Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg.

•             Harden and extend the reusable, configuration-driven migration framework developed during the POC.

•             Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.

•             Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing.

•             Design execution approaches for small, medium, large and very-large tables.

•             Optimize parallel migration across tables and partitions.

•             Implement checkpointing, restart/recovery, idempotency and safe retries.

•             Lead technical execution and troubleshooting of migration waves.

Data Modelling & Schema Engineering

•             Analyze existing Hive data models and translate them into approved Apache Iceberg physical models.

•             Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.

•             Perform source-to-target data mapping and schema/data-type mapping.

•             Define and implement table structures consistent with approved Iceberg standards.

•             Analyze:

o             entities and relationships,

o             primary/foreign-key relationships,

o             data types,

o             partition keys,

o             business keys,

o             reference/cross-reference structures.

•             Identify schema incompatibilities and recommend remediation.

•             Work with the Data Architect on partitioning, file-layout and model-design decisions.

•             Ensure migrated models preserve required business r