Overview:
We are working with a leading UK-based wealth and investment management firm seeking a skilled Data Engineer to support the build and evolution of their modern data platform. This role will focus on developing scalable data pipelines and contributing to a cloud-native lakehouse architecture.
Key Responsibilities:
*Design, build, and maintain robust data pipelines using Azure-based services
*Develop and optimize data workflows within Databricks
*Implement and manage data ingestion processes, including real-time streaming with Kafka
*Structure and maintain data across bronze, silver, and gold layers within a lakehouse architecture
*Write efficient, scalable SQL for data transformation and analysis
*Collaborate with data analysts, architects, and business stakeholders to deliver high-quality data solutions
*Ensure data quality, governance, and performance best practices are followed
Required Skills & Experience:
*Strong experience with Azure data platform (e.g., Data Factory, Data Lake, Synapse)
*Hands-on expertise with Databricks (PySpark or Spark)
*Solid SQL development skills
*Experience working with Kafka for streaming data pipelines
*Good understanding of lakehouse architecture, including bronze/silver/gold layering concepts
*Experience in financial services or regulated environments is advantageous
Desirable:
*Knowledge of CI/CD practices for data engineering
*Familiarity with data governance and security frameworks