Design, develop, test, and maintain scalable data processing applications using Python and PySpark.
• Build and maintain ETL/ELT pipelines for data ingestion, transformation, validation, and integration.
• Develop distributed data processing solutions using Apache Spark and PySpark to process large datasets efficiently.
• Design, implement, and manage cloud-based data pipelines using AWS services.
• Work with AWS services such as Amazon S3, AWS Glue, AWS Lambda, Amazon EMR, Amazon Redshift, AWS Step Functions, and Amazon CloudWatch, as applicable to the project.
• Develop data extraction, transformation, and loading processes from multiple data sources into data lakes and data warehouses.
• Implement data transformation logic, business rules, aggregations, joins, and data cleansing using Python and PySpark.
• Optimize Spark jobs through partitioning, caching, join strategies, and resource tuning to improve performance and reduce processing time.
• Develop and maintain reusable Python modules, libraries, utilities, and automation scripts.
• Implement data validation, reconciliation, error handling, logging, and exception management within data pipelines.
• Monitor pipeline execution, troubleshoot failures, and resolve performance bottlenecks in production environments.
• Work with relational databases and data platforms using SQL for data extraction, analysis, validation, and reconciliation.
• Implement secure data processing practices using appropriate AWS IAM roles, policies, encryption, and access controls.
• Support CI/CD pipelines, automated deployments, version control, and release management using tools such as Git and AWS-native or other DevOps solutions.
• Collaborate with Data Architects, Data Engineers, DevOps teams, and business stakeholders to translate requirements into technical solutions.
• Participate in code reviews, unit testing, integration testing, production deployments, and technical documentation.
• Perform root cause analysis and implement corrective actions for data pipeline failures and production incidents.
• Ensure adherence to coding standards, data governance requirements, security policies, and enterprise development practices.