Job Responsibilities/KRAs:

• Design, develop and maintain ETL processes using Spark (PySpark) to integrate data as Iceberg on ADLS from multiple source systems

• Petabyte size database migration to ADLS on Iceberg tables

• Real-time data processing from Azure Service Bus using Spark streaming/airbyte/flink

• Optimize ETL workflows to ensure efficient data processing and loading.

• Develop scripts to automate data processing and loading tasks.

• Implement data quality checks and validation processes within ETL workflows.

• Understanding business requirements/scope of projects, create ETL code as per business logic/process; be able to provide estimation for the tasks as required with supporting data points.

• Ensure data governance policies are adhered to, including data lineage and metadata management.

• Provide support for data-related issues and troubleshoot ETL-related problems.

• Create and maintain technical documentation and reports for stakeholders.

  Requirements –

• 10 years of total technical experience on designing, developing, and implementing ETL solutions using Apache Spark.

• Experience in complete Software Development Life Cycle which includes Systems Analysis of various applications in Client/Server Environment.

• Working with various software applications with advanced knowledge (FiveTran, Airbyte, HVR, Apache flink, SQL)

• Experience in creating Streaming data processing using Spark streaming, shell scripting, deployment activities, performance tuning and error handling skills.

• Excellent hands-on experience in various data sources, transformations, Partition/De-partition, Databases, Datasets and JSON/XML/Parquet file formats

• Experience in data replication from RDBMS source using HVR

• Experience in working on airbyte/Apache flink is preferred

• Analytical problem solving and business interaction skills.

• Effective communication with entire offshore team, customer and all concerned teams on day-to-day basis; provide daily status reporting to all stakeholders.

• Experience in the insurance (e.g., UW, Claim, Policy Issuance) or financial industry preferred.