Spark-Scala, Databricks

Key Responsibilities: Data Engineering & ETL Development • Design, develop, and maintain ETL pipelines using Spark-Scala on Databricks for batch and incremental processing. • Implement data transformations, joins, aggregations, and validations to ensure accurate and consistent outputs. • Build reusable Spark components and follow best practices for modular, maintainable code. Performance, Reliability & Operations • Tune Spark jobs (partitioning, caching, shuffle optimization) to improve performance and cost efficiency. • Monitor job execution, troubleshoot failures, and provide timely production support with root-cause analysis. • Implement logging, error handling, and data quality checks to improve pipeline reliability. Collaboration & Delivery • Work with cross-functional teams to gather requirements and translate them into technical solutions. • Participate in code reviews, documentation, and knowledge sharing to uplift team standards. • Support release cycles by validating outputs, ensuring backward compatibility, and maintaining deployment readiness.