Senior Data Engineer to implement a Data Lake in an Azure environment

Our valued Public Sector client requires a Senior Data Engineer to implement a Data Lake in an Azure environment. The successful resource will be responsible for designing and implementing Data Storage, Data Processing, and Data Security, as well as monitoring and optimizing data storage and data processing.

Knowledge and experience with the following would be considered a strong asset:

• Continuous Integration/Continuous Deployment (CI/CD)
• DevOps
• Azure Monitor
• Azure Data Lake Storage Gen2
• Azure Synapse
• Azure Databricks
• Azure DevOps
• Azure Data Factory
• Stream Analytics, Azure Event Hubs
• Apache Spark

Relevant certification: DP-203: Data Engineering on Microsoft Azure

Responsibilities include, but are not limited to:

• Architecting Microsoft Azure Solutions across multiple platforms
• Building governance and security around the Azure platform and how it fits into the solutions.
• Designing Identity and Access management with Azure AD and Azure resources
• Providing Microsoft Azure architecture collaboration with other technical
• Designing and building observability platform using modern monitoring and management tools
• Documentation of solutions (e.g. architecture, configuration and setup).
• Working within a project management/agile delivery methodology in a leading role as part of a wider team.
• Provide effective knowledge transfer and up skilling to relevant customer personnel to ensure an appropriate level of future self-sufficiency.
• Implementation and Delivery of Microsoft Azure projects
• Work with the IT project team & support the development of solution architecture, detail design, development & operational support of the infrastructure of systems
• Design Cost Effective data solutions including security, modernization, and optimizations in Azure Cloud, ADF, & Databricks
• Lead sessions in identifying process and systems solutions to meet business needs or to resolve business issues;
• Lead business process and technology walk-through and design sessions for the purpose of effectively scoping and assessing the impact of desired enhancements or changes to business processes, operations, and information process flow;
• Coordinate with other IT and business groups to meet project deliverables;
• Ensure compatibility and interoperability of planned and existing application systems;
• Prepare and deliver reports, recommendations, or alternatives that address existing and potential trouble areas in operating systems across the organization
• Act as the Subject Matter Expert for the team in the areas of Azure tools, cloud computing, IAAS, PaaS, SaaS and Cloud design patterns.
• Creating a network design in an Azure environment
• Development of the artifacts using Infrastructure as Code (IaC) in Terraform
• Detailed design and deployment of cloud services blueprints and associated delivery pipelines with a security focus
• Design and implement Azure IaaS Virtual Machines
• Build cloud foundational components encompassing network, security, identity, operations, and governance in the Azure cloud
• Manage complex architectural designs
• Define, design, implement, manage and monitor the Cloud Computing environment for Infrastructure and Platform Services (IaaS, PaaS) and Desktop as a Service as well as Azure AD, Azure SQL, Virtual Networks, Machines, Firewall, Security, Storage, AIP, ATP, ASR, CAS, EM+S, Teams, Intune, AAD Connect, Exchange Online and O365
• Engineer and maintain highly available, secure, and cost-effective solutions in Microsoft Azure
• Partners with internal teams to architect and design efficient Cloud solutions that meet business requirements
• Effectively employ expert knowledge of the Azure Cloud platform and share it with the Cloud Operations team.
• Plan and identify solutions and risks in all projects from design to deployment.
• Optimize Azure buildouts for cost/performance (VM optimization, reserved instances, etc.)
• Develop, construct, test and maintain architectures
• Align architecture with business requirements

Must-haves:

1. Project experience with Data Storage, Data Processing, or Data Security in an Azure environment
2. Enhanced Reliability clearance