Description
Company Description
Project Brains is a Future of Work platform that helps ambitious businesses grow by matching their needs with fractional specialists from a vetted community. This allows businesses to focus on core priorities while being supported by experts who deliver successful outcomes.
Role Description
You will design and build high-performance, scalable data processing applications in Scala, optimising ETL pipelines to handle massive datasets with Apache Spark, Kafka, and Flink. You will architect and maintain data lakes and warehouses using technologies like Databricks Delta Lake, Apache Iceberg, and Apache Hudi, ensuring efficient query performance and storage optimisation. Real-time streaming and event-driven architectures will be part of your toolkit, as you collaborate with data scientists, analysts, and DevOps to create robust, fault-tolerant solutions. You will champion data governance, security, and compliance; conduct code reviews; mentor junior developers; and drive best practices in functional programming.
Qualifications
10+ years of experience in software development with Scala
Hands-on experience with Apache Spark (batch & streaming) using Scala.
Experience in developing and maintaining data lake and warehouses using technologies like Databricks Delta Lake, Apache Iceberg, or Apache Hudi.
Experience working with open-source data formats JSON, Parquet, Avro etc.
Deep understanding of distributed computing and parallel processing.
Experience working on cloud platforms (preferably AWS) and services like S3, EMR, Glue, Redshift, BigQuery.
Proficiency in CI/CD pipelines; Docker; Kubernetes; Terraform.
Knowledge of Java; Python; or other JVM-based languages is a plus.
Experience with RESTful APIs; GraphQL; microservices architecture.
Strong understanding of functional programming concepts (immutability; monads; higher-order functions).
Experience with SQL and NoSQL databases (PostgreSQL; Cassandra; DynamoDB; HBase; MongoDB).
Experience with Kafka; Flink or other streaming frameworks.
Experience with data security;, lineage;, governance tools.
Experience working in Agile environment
Bachelor’s or Master’s degree in Computer Science Engineering or a related field.
Preferred experience:
Experience of working as a Scala developer/engineer within the technology function of a Global Investment Bank.
Experience migrating data applications from Hadoop to AWS Databricks