None
Insight International (UK) Ltd
AWS Databricks Tech Lead
Description
Role: AWS Databricks Tech Lead
Location: Remote
Job Type: Contract
Key Skills:
- AWS Data / ETL work Experience -> S3, Glue, Athena
- Databricks work experience -> Data processing inside Databricks
- Data or ETL migration/Modernization experience (MUST for Architect, Optional for Lead but itโs very much preferred)
- Proficient in SQL queries
The key is to get candidates who have worked in Databricks on AWS.
Experience and Education Required:
- 10+ years of experience as Data Analyst / Data Engineer/Data Scientist with Databricks on AWS expertise in designing and implementing scalable, secure, and cost-efficient data solutions on AWS.
Job Profile:
- Hands-on data analytics experience with Databricks on AWS, Pyspark and Python.
- Must have prior experience with migrating a data asset to the cloud using a GenAI automation option.
- Experience in migrating data from on-premises to AWS.
- Expertise in developing data models, delivering data-driven insights for business solutions.
- Experience in pretraining, fine-tuning, augmenting and optimizing large language models (LLMs).
- Experience in Designing and implementing database solutions, developing PySpark applications to extract, transform, and aggregate data, generating insights.
Data Collection & Integration:
- Identify, gather, and consolidate data from diverse sources including internal databases and spreadsheets ensuring data integrity and relevance.
Data Cleaning & Transformation:
- Apply thorough data quality checks cleaning processes and transformations using Python (Pandas) and SQL to prepare datasets.
Automation & Scalability:
- Develop and maintain scripts that automate repetitive data preparation tasks.
Autonomy & Proactivity:
- Operate with minimal supervision demonstrating initiative in problem-solving prioritizing tasks continuously improving the quality and impact of your work.
Technical Skills:
- Minimum of 10 years of experience as a Data Analyst Data Engineer or related role ideally with a bachelorโs degree or higher in a relevant field.
- Strong proficiency in Python (Pandas Scikit-learn Matplotlib) SQL with experience working across various data formats sources.
- Proven ability to automate data workflows implement code-based best practices maintain documentation to ensure reproducibility scalability.