Description
Location: On-site near Worcester/Great Malvern/Gloucester
Daily Rate: Up to £700 outside IR35
Duration: 6 months with possible extension
Clearance: Must be SC cleared; DV preferred
Role Overview:
We are seeking a seasoned Senior Data Engineer (Infrastructure) to join our team. This role is essential for designing, building, and maintaining sophisticated data infrastructure systems that operate across both on-premises and Azure cloud environments. The position involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, crucial for our organisational growth and innovation.
Requirements:
- Develop and Manage Data Pipelines: Design, construct, and maintain efficient and reliable data pipelines using Python, capable of supporting both streaming and batch data processing across structured, semi-structured, and unstructured data in on-premises and Azure environments.
- Hybrid Cloud and Data Storage Solutions: Implement and manage data storage solutions leveraging both on-premises infrastructure and Azure, ensuring seamless data integration and accessibility across platforms.
- Containerisation and Orchestration: Utilise Docker for containerisation and Kubernetes for orchestration, ensuring scalable deployment of applications across cloud-based & on-premises environments.
- Workflow Automation: Employ tools such as Apache NiFi & Apache Airflow to automate workflows within hybrid environments.
- Event Streaming Experience :Utilise event-driven technologies such as Kafka ,Apache Nifi,&Apache Flink effectively handle real-time streams .
-Security Compliance :Manage security setups incorporating Keycloak protect integrity comply legal standards all platforms .
-Database Management Administer optimise PostgreSQL databases ensure high performance availability diverse deployment scenarios .
Essential Skills And Experience :
-Strong Python Skills Expertise scripting automating processes varied environsments .
Experience With Etl/Elt Demonstrable developing optimising workflows particularly hybrid (onprem azure)environments
Expertise In Hybrid Cloud Architecture Profound knowledge integrating infrastructures services
Containerisation Orchestration Expertise Solid experience docker kubernetes managing applications premises platforms
Proficiency Workflow Automation Tools Practical apache nifi airflow hybrid environsments
Experience Event Streaming Proven ability deploying streaming kafka apache flink etc..
Data Security Knowledge Implementing practices including keycloak multiple platfroms
Search Database Management Strong background elasticsearch postgresql spanning infrastructures
Your Impact :
In this role you will empower business leaders make informed decisions delivering timely accurate actionable insights robust infrastructure Your expertise drive seamless integration enhancing flexibility scalability operations You champion adoption modern architectures tooling play pivotal cultivating culture mentoring advancing practices
Desirable Skills And Experience :
Certifications Relevant Technologies Highly beneficial strengthen application
Minimum Years Engineering Significant exposure managing settings
Security Clearance Requirements :
This requires willing undergo As part onboarding process candidates asked complete Baseline Personnel Standard evidence required found government website Gov UK If unable meet associated criteria then employment delayed rejected Details discussed interview