Outside IR35 contract jobs

Description

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:
- Databricks: Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
- AWS Cloud: Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
- IBM Skills: DB2, Datastage, Tivoli Workload Scheduler, Urban Code
- Programming Languages: Proficiency in Python, SQL.
- Data Warehousing & ETL: Experience with modern ETL frameworks and data warehousing techniques.
- DevOps & CI/CD: Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch).
- Familiarity with big data technologies like Apache Spark or Hadoop.
- Test automation skills
- ETL/ELT tools and creating common data sets across on-prem (IBM Datastage ETL) and cloud data stores

Leadership & Strategy:
Lead Data Engineering team(s) in designing developing maintaining highly scalable performant infrastructures.

Customer Data Platform Development:
Architect manage our platforms using IBM legacy platform &Databrickstechnologies e.g.S3,LambdaGlacierGlueEventBridgeRDSto support real-time batch processing needs.

Data Governance Best Practices:
Implement best practices governance security quality across platform Ensure well-documented accessible meets compliance standards.

Pipeline Automation Optimisation:
Drive automation workflows improve efficiency reliability

Team Management:
Mentor grow engineers ensuring alignment business goals delivery timelines technical standards

Cross Company Collaboration:
Work closely all levels stakeholder including scientists finance analysts MI cross-functional teams ensure seamless access integration various tools systems

Cloud Management :
Lead efforts integrate scale services optimising costs ensuring resilience Performance Monitoring Establish alerting solutions high availability systems no impact downstream consumers