Purpose of the Job/ Role:

• We are looking for a highly skilled and hands-on Data Engineer with strong expertise in Big Data technologies, distributed data processing, and backend engineering. The ideal candidate should have practical experience in building scalable batch and real-time data pipelines using frameworks such as PySpark, Kafka, and streaming platforms.

• The candidate should also possess strong Python programming skills and experience building APIs/microservices using modern backend frameworks.

• This role is best suited for engineers who enjoy solving large-scale data engineering problems and working closely with AI/GenAI-driven applications.

• We are specifically looking for candidates with strong hands-on coding and distributed systems experience, and not candidates whose experience is predominantly limited to traditional ETL tools, low-code cloud pipelines, or Snowflake-centric workflows.

Key Requisites:

• Design, develop, and optimize scalable batch and streaming data pipelines

• Build large-scale distributed data processing systems using PySpark

• Develop real-time streaming solutions using Kafka and/or Apache Flink

• Build and maintain APIs and backend services using Python-based microservice frameworks

• Work with structured and semi-structured datasets across multiple storage systems

• Optimize pipeline performance, scalability, and fault tolerance

• Collaborate with AI/ML and GenAI teams to support LLM and RAG-based applications

• Participate in architecture discussions, debugging, code reviews, and production deployments

• Ensure reliability, monitoring, and maintainability of data engineering systems

Mandatory Skills

• Strong hands-on experience with PySpark and distributed data processing (must-have)

• Strong programming expertise in Python

• Hands-on experience building APIs and backend services using:

• Flask

• FastAPI

• or similar microservice frameworks

• Good understanding of Big Data ecosystem and streaming architectures

Experience with:

• Apache Kafka

• Apache Flink (preferred)

• PostgreSQL

• Experience building end-to-end batch and real-time data engineering pipelines

• Good understanding of scalable system design, partitioning, optimization, and distributed processing concepts

• Understanding of microservices architecture and API integrations

• Experience working in Linux-based environments

• Strong analytical and problem-solving skills