Description
Important: Please Read Full Summary Before Applying
Methods is seeking 3 - 4 skilled Data Engineers with expertise in Pyspark and Python for an exciting contract opportunity. This role requires a commitment to work 4 or 5 days per week on-site at a client location in the West Midlands. Candidates must hold SC clearance or be willing to undergo the SC clearance process.
Eligibility: Sole UK Nationals Only.
Role Overview:
As a Senior Data Engineer (Infrastructure), you will play a vital role in designing, building, and maintaining complex data infrastructure systems across both on-premises and Azure cloud environments. Your contributions will facilitate scalable data operations that enhance advanced analytics and drive data-driven decision-making, key components of our organization's growth and innovation.
Key Responsibilities:
- Data Pipeline Development: Design, construct, and maintain efficient and reliable data pipelines with Python that support both streaming and batch data processing, handling structured, semi-structured, and unstructured data in on-premises and Azure settings.
- Hybrid Cloud Solutions: Implement and manage hybrid data storage solutions leveraging both on-premises infrastructure and Azure, ensuring seamless data integration and accessibility.
- Containerisation and Orchestration: Use Docker for containerisation and Kubernetes for orchestration to enable scalable and efficient application deployment across both on-premises and cloud environments.
- Workflow Automation: Utilize tools like Apache NiFi and Apache Airflow to automate data flows and manage complex workflows in hybrid environments.
Security Clearance Requirement:
This position requires you to have or be willing to obtain Security Clearance. As part of the onboarding process, candidates will need to complete a Baseline Personnel Security Standard; details on the evidence required can be found on the government website Gov.UK. Failure to meet these criteria may result in delays or rejection of employment.
Further details will be discussed during the interview process.