Description
Key Responsibilities:
- Design and implement end-to-end data architectures, ensuring best practices for data governance, access control, and cost efficiency using Unity Catalog and Delta Lake.
- Optimize query performance, storage solutions, and compute resource allocation within Databricks.
- Support clients in adopting the Databricks Lakehouse architecture for both structured and unstructured market data.
- Work closely with internal teams to share knowledge and contribute to ongoing learning and development initiatives.
Required Skills & Experience:
- Strong expertise in data engineering, data platforms, and analytics.
- Proficiency in Python, SQL, and/or Scala.
- Experience with at least one major cloud provider (Azure, AWS, or GCP) and familiarity with another.
- Deep knowledge of Apache Spark and distributed computing.
- Practical experience with CI/CD pipelines for production deployments.
- Understanding of MLOps and its application in data workflows.