Big Data / PySpark Engineering Lead (12+)
citi | 15 days ago | Pune

Architecture & Design

  • Design and implement scalable, fault-tolerant batch and real-time data processing pipelines.
  • Develop robust data models and schema designs optimized for both performance and storage efficiency.
  • Evaluate and integrate emerging tools and frameworks (e.g., Spark, Flink, Kafka) into the existing stack.
  • Provide in-depth analysis with interpretive thinking to define issues and develop innovative solutions
  • Develop comprehensive knowledge of how areas of business, such as architecture and infrastructure, integrate to accomplish business goals

Data Modernization & Migration Leadership

  • Legacy Systems Decommissioning: Lead the strategic migration of data and logic from legacy platforms (e.g. on-premises SQL Servers) to a modern Data Lakehouse environment.
  • ETL/ELT Transformation: Re-engineer existing stored procedures and complex legacy ETL jobs into scalable, distributed processing frameworks using Spark (Python) and Starburst/Trino.
  • Validation & Parity Testing: Design and implement automated frameworks for Data Parity Testing to ensure 100% accuracy and consistency between legacy outputs and new big data results.
  • Schema Evolution: Map and transform rigid, legacy relational schemas into flexible, high-performance formats optimized for the cloud (e.g., Parquet, Avro, or Iceberg).
  • Phased Cutover Management: Orchestrate a phased migration strategy (Parallel Run, Shadow Execution) to ensure zero downtime for downstream business applications and reporting tools.
  • Performance Benchmarking: Establish performance baselines on legacy systems and ensure the new Big Data architecture meets or exceeds those benchmarks at scale.
  • Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards

2. Engineering Excellence

  • Write clean, high-performance code in Python.
  • Optimize complex SQL queries and fine-tune distributed computing clusters to reduce latency and costs.
  • Ensure data integrity and security by implementing rigorous validation and encryption standards.

3. DevOps & Reliability

  • Build and maintain CI/CD pipelines for automated testing and deployment of data jobs.
  • Monitor system health and troubleshoot performance bottlenecks across the data lifecycle.

4. Leadership & Strategy

  • Provide technical mentorship and conduct code reviews for junior and mid-level engineers. Serve as advisor or coach to mid-level developers and analysts, allocating work as necessary.
  • Translate complex business requirements into technical specifications.
  • Collaborate with Product Managers to ensure data availability for downstream analytics, business models and users
  • Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.
  • Partner with multiple management teams to ensure appropriate integration of functions to meet goals as well as identify and define necessary system enhancements to deploy new products and process improvements

 

Official notification

⚡ Hot Jobs Trending Now

SRE
Sr. SRE Engineer
Stripe | Bangalore, India
DEV
Backend Developer
Coinbase | Remote, India
Infra
Cloud Infra Lead
Datadog | Pune, India
ML
MLOps Architect
Anthropic | Hyderabad
Data
Fivetran Data Eng.
Fivetran | Mumbai
SRE
Sr. SRE Engineer
Stripe | Bangalore, India
DEV
Backend Developer
Coinbase | Remote, India
Infra
Cloud Infra Lead
Datadog | Pune, India
ML
MLOps Architect
Anthropic | Hyderabad
Data
Fivetran Data Eng.
Fivetran | Mumbai
SDE
Staff Software Eng.
Airbnb | Gurgaon, India
Prod
Platform Engineer
Databricks | Bangalore
QA
Quality Assurance
GitLab | Remote
Security
Cloud Security
Zscaler | Mumbai
UX
Product Designer
Figma | Pune, India
SDE
Staff Software Eng.
Airbnb | Gurgaon, India
Prod
Platform Engineer
Databricks | Bangalore
QA
Quality Assurance
GitLab | Remote
Security
Cloud Security
Zscaler | Mumbai
UX
Product Designer
Figma | Pune, India
Contact US

Let's work laptop charging together

Any question or remark? just write us a message

Send a message

If you would like to discuss anything related to payment, account, licensing,
partnerships, or have pre-sales questions, you’re at the right place.