Description
Company Description
Project Brains is a Future of Work platform that helps ambitious businesses grow by matching their needs with fractional specialists from a vetted community. This allows businesses to focus on core priorities while being supported by experts who deliver successful outcomes.
Role Description
You will design and build high-performance, scalable data processing applications in Scala, optimising ETL pipelines to handle massive datasets with Apache Spark, Kafka, and Flink. You will architect and maintain data lakes and warehouses using technologies like Databricks Delta Lake, Apache Iceberg, and Apache Hudi, ensuring efficient query performance and storage optimisation. Real-time streaming and event-driven architectures will be part of your toolkit, as you collaborate with data scientists, analysts, and DevOps to create robust, fault-tolerant solutions. You will champion data governance, security, and compliance; conduct code reviews; mentor junior developers; and drive best practices in functional programming.
Qualifications
- Ca. 10 years of experience in software development with Scala
- Hands-on experience with Apache Spark (batch & streaming) using Scala
- Experience in developing and maintaining data lake and warehouses using technologies like Databricks Delta Lake, Apache Iceberg or Apache Hudi
- Experience working with open-source data formats JSON, Parquet, Avro etc.
- Deep understanding of distributed computing and parallel processing
- Experience working on cloud platforms (preferably AWS) and services like S3, EMR, Glue, Redshift or BigQuery
- Proficiency in CI/CD pipelines; Docker; Kubernetes; Terraform
- Knowledge of Java; Python or other JVM-based languages is a plus
- Experience with RESTful APIs; GraphQL; microservices architecture
- Strong understanding of functional programming concepts (immutability; monads; higher-order functions)
- Experience with SQL & NoSQL databases (PostgreSQL; Cassandra; DynamoDB; HBase: MongoDB)
- Experience with Kafka: Flink or other streaming frameworks
- Experience with data security: lineage: governance tools
- Experience working in Agile environment
- Bachelor’s or Master’s degree in Computer Science Engineering or a related field
Preferred experience:
Experience of working as a Scala developer/engineer within the technology function of a Global Investment Bank.
Experience migrating data applications from Hadoop to AWS Databricks