Description
On-site: near Worcester/Great Malvern/Gloucester
Daily rate: up to £700 outside IR35
Duration: 6 months with possible extension
Clearance: Must be SC cleared; DV preferred
We are seeking a seasoned Senior Data Engineer (Infrastructure) to join our team. This role is essential for designing, building, and maintaining sophisticated data infrastructure systems that operate across both on-premises and Azure cloud environments. The position involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, crucial for our organisational growth and innovation.
Requirements
Develop and Manage Data Pipelines: You will design, construct, and maintain efficient and reliable data pipelines using Python, capable of supporting both streaming and batch data processing across structured, semi-structured, and unstructured data in on-premises and Azure environments.
Hybrid Cloud and Data Storage Solutions: Implement and manage data storage solutions leveraging both on-premises infrastructure and Azure, ensuring seamless data integration and accessibility across platforms.
Containerisation and Orchestration: Utilise Docker for containerisation and Kubernetes for orchestration, ensuring scalable and efficient deployment of applications across both cloud-based and on-premises environments.
Workflow Automation: Employ tools such as Apache NiFi and Apache Airflow to automate data flowsand manage complex workflows within hybrid environments.
Event Streaming Experience: Utilise event-driven technologies such as Kafka, Apache NiFi,and Apache Flink to handle real-time data streams effectively.
Securityand Compliance: Manage security setupsand access controls,involving tools like Keycloakto protectdata integrityand comply with legal standardsacross all dataplatforms.
Data Searchand Analytics: OverseeandenhanceElasticsearchsetupsfor robustdata searchingandanalytics capabilitiesin mixedinfrastructure settings.
Database Management: AdministerandoptimisePostgreSQLdatabases,enabling highperformanceandavailabilityacrossdiverse deployment scenarios.
Essential Skillsand Experience
Strong Python Skills: Expertise in Pythonfor scriptingandautomatingdataprocessesacrossvariedenvironments.
Experiencewith ETL/ELT:Demonstable experienceindevelopingandoptimisingETLorELTworkflowsparticularlyinhybrid(on-premisesandanAzure)environments.
Expertise in Hybrid CloudData Architecture :Profound knowledgeof integratingon-premisesinfrastructurewithAzurecloudservices.
ContainerisationandaOrchestrationExpertise :Solid experiencewithDockerandKubernetesin managingapplicationsacrossbothon-premisesandcloudplatforms.
Proficiency in WorkflowAutomationTools :PracticalexperiencewithApacheNiFiandanApacheAirflowinhybriddataenvironments .
Experiencein EventStreaming :ProvenabilityinmanagingandeployingeventstreamingplatformslikeKafka ,ApacheNiFi,andApacheFlink .
Data SecurityKnowledge :Experiencewithimplementingsecuritypracticesandt ools ,includingKeycloak ,acrossmultiple platforms .
SearchAndDatabaseManagementSkills :Strongbackgroundin managingElasticsearchan dPostgreSQLinenvironmentsthatspanon -premisesan dcloud infrastructures .
Your Impact
In this role,you will empowerbusinessleaders tomake informeddecisionsby delivering timely ,accurate,and actionabledata insightsfrom a robust,h ybrid infrastructure . Your expertisewill drive theseamlessintegrationofon -premis es an dcloud-baseddatasolutions ,enhancingboththeflexibilityands calabilityofourdataoperations . You willchampion the adoptionofmoderndataarchitecturesandt ooling,andplayapivotalroleincultivatinga datadriven culturewithin theorganisation ,mentoringteam members,andadvancingourengineeringpractices .
Desirable SkillsAnd Experience
Certifications inAzurean dOtherRelevantTechnologies :Certifications incloudandon -premisest echnologiesarehighlybeneficial an dwills trengthenyourapplication .
Experience inDataEngineering:Aminimumof5yearsofexperienceindataengineering ,withsignificantexposureto managing infrastructure inbothon -premis es an dcloudsettings .
This rolewill requireyoutohaveorbewillingtogothroughSecurityClearance.As partofthe onboardingprocess candidateswillbeaskedto completeaBaselinePersonnelSecurityStandard;detailsoftheevidencerequiredtoapplymaybefoundonthegovernmentwebsiteGov.UK.Ifyouareunabletomeetthisan dandassociatedcriteria ,thenyouremploymentmaybedelayed orrejected.Details ofthiswillbediscussed withyouat interview.