Full job descriptionPlatform Architecture & Engineering responsibilities:
*Design and implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements
*Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers
*Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB’s))
*Establish secure, compliant environments including networking (VNet/VPC, Private Link), identity (IAM/Entra ID), data governance (Unity Catalog), and access controls
*Define environment strategies (dev/test/prod), CI/CD pipelines, and release processes for Databricks deployments
*Implement monitoring, logging, cost optimisation, and performance tuning across the platform
*Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads
Client Delivery & Enablement responsibilities:
*Work directly with clients to translate business and technical requirements into scalable platform designs
*Lead technical workshops, architecture sessions, and whiteboarding engagements with client stakeholders
*Support rapid prototyping and proof-of-concept builds within Databricks to demonstrate platform capabilities and accelerate client adoption
*Provide best practice guidance on Lakehouse architecture, data modelling, workload optimisation, and cost management
*Produce high-quality technical documentation including architecture diagrams, architecture decision records (ADRs), runbooks, and deployment guides
*Enable client teams through structured knowledge transfer, training, and platform handover
*Collaborate with data engineers, data scientists, and product teams to ensure successful delivery outcomes
Governance & Security:
*Implement Unity Catalog for centralised data governance, including access control (RBAC/ABAC), data lineage, audit logging, and compliance enforcement
*Apply security best practices across platform design: network isolation, secret management, encryption at rest and in transit, and identity federation
*Ensure platform designs meet client regulatory and compliance requirements (e.g. GDPR, ISO 27001, sector-specific standards)
What success looks like in the role:
*Delivery of robust, secure, and scalable Databricks platforms that meet client performance and cost expectations
*Clear, well-architected solutions that balance flexibility, governance, and operational efficiency
*Strong client relationships built on trust, technical credibility, and effective communication
*Accelerated client adoption of the Lakehouse platform through well-designed enablement and documentation
*Reduced deployment time through reusable infrastructure patterns and automation
*Proactive identification of risks, trade-offs, and optimisation opportunities across platform design and delivery
*Contribution to the organisation’s growing body of reusable platform accelerators, reference architectures, and internal knowledge
Competencies and Behaviours:
*3+ years experience in data platform engineering, cloud engineering, or similar roles
*Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows
*Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/VPC, IAM)
*Experience with infrastructure-as-code tools (e.g. Terraform preferred) and CI/CD pipelines (e.g. Azure DevOps, GitHub Actions)
*Solid understanding of data architecture concepts including Lakehouse medallion architecture and dimensional modelling
*Familiarity with security and governance frameworks (e.g. RBAC, ABAC, data masking, audit, compliance standards)
*Excellent communication skills with the ability to explain complex technical concepts to non-technical stakeholders
*Comfortable working in a client-facing consultancy environment with multiple concurrent engagements
*Proactive, self-driven, and able to take ownership of end-to-end platform delivery
*Willingness to travel within the UK as required
*Right to work in the UK