Senior Data Engineer to create, manage, and monitor data pipelines using Azure Data Factory and Azure Synapse for a Government client

Our valued public sector client requires two Senior Data Engineers to create, manage, and monitor data pipelines using Azure Data Factory and Azure Synapse for a Government client.

Tasks:

• Azure Data Factory (ADF) and Azure Synapse Analytics for data ingestion, transformation, and processing experience
• Data integration and the development of data pipelines
• Data processing for cost-effectiveness in Azure's scalable and flexible cloud environment
• Creation, management, and optimization of data processing workflows within the Azure data platform for the clients data warehouse modernization initiative
• Python scripting for automation of storage-related tasks, configuration management, and system monitoring
• Manage workload using Azure Synapse for a data lake project.
• Utilize Azure Synapse to distribute the workload (performance management)
• Develop and maintain Python scripts for efficient data manipulation, migration, and integration with storage systems.
• Develop spark applications in python (PySpark) on distributed environment
• Creating Spark clusters and configuring high concurrency clusters
• Create and maintain documentation for compliance and knowledge transfer
• Design for scalability and optimize code for peak performance.
• Develop test cases for unit and integration tests
• Data acquisition
• Git branching strategies using Azure DevOps Repo & pipeline
• Develop data set processes
• Identify ways to improve data reliability, efficiency and quality
• Use large data sets to address business issues
• Deliver updates to stakeholders based on analytics

Must-haves:

1. 4+ years of experience using Azure Data Factory
2. 4+ years of experience using Azure Synapse
3. 2+ years of experience creating Git branching strategies using ADO Repo and Pipeline
4. 2+ years of Python development experience

Nice-to-haves:

1. Experience working for the Federal Government
2. Experience working with mainframe systems/files