Outside IR35 contract jobs

Description

Role: Data Engineer (Iceberg Experience)
Location: Milton Keynes, UK (Hybrid)
Employment Type: Contract

Job Description:
As a Data Engineer with Iceberg experience, you will play a crucial role in the design, development, and maintenance of our data infrastructure. Your work will empower data-driven decision-making and contribute to the success of our data-driven initiatives.

Key Responsibilities:
- Data Integration: Develop and maintain data pipelines to extract, transform, and load (ETL) data from various sources into AWS data stores for both batch and streaming data ingestion.
- AWS Expertise: Utilize your expertise in AWS services such as Amazon EMR , S3, AWS Glue, Amazon Redshift, AWS Lambda, and more to build and optimize data solutions.
- Data Modeling: Design and implement data models to support analytical and reporting needs, ensuring data accuracy and performance.
- Data Quality: Implement data quality and data governance best practices to maintain data integrity.
- Performance Optimization: Identify and resolve performance bottlenecks in data pipelines and storage solutions to ensure optimal performance.
- Documentation: Create and maintain comprehensive documentation for data pipelines, architecture, and best practices.
- Collaboration: Collaborate with cross-functional teams, including data scientists and analysts, to understand data requirements and deliver high-quality data solutions.
- Automation: Implement automation processes and best practices to streamline data workflows and reduce manual interventions.

Experience working with bigdata ACID file formats to build delta lake is required; particularly with Iceberg file formats and loading methods of Iceberg is essential. Good knowledge on Iceberg functionalities is necessary for using the delta features to identify changed records as well as optimization and housekeeping on Iceberg tables in the datalake.

Must have skills:
- AWS
- ETL
- EMR
- GLUE
- Spark/Scala
- Java
- Python

Good to have skills:
- Cloudera – Spark
- Hive
- Impala
- HDFS
- Informatica PowerCenter
- Informatica DQ/DG
- Snowflake Erwin

Qualifications:
-Bachelor's or master’s degree in computer science or related field.
-5 to 8 years of experience in Data Engineering including working with AWS services.
-Proficiency in AWS services like S3 , Glue , Redshift , Lambda , EMR.
-Knowledge on Cloudera based Hadoop is a plus.
-Stronger ETL development skills along with experience using integration tools.
-Knowledge of modeling techniques along with warehousing principles.
-Familiarity regarding quality principles within governance frameworks.
-Solid problem-solving capabilities alongside troubleshooting expertise.
-Outstanding communication abilities paired with teamwork aptitude; capable of collaborating across technical/nontechnical stakeholders alike.
-Awareness surrounding engineering best practices focused upon scalability/performance optimization strategies preferred .
-Prior exposure towards version control systems alongside DevOps methodologies considered advantageous .