Data Architect

Databricks Architect

• Location: London, UK
• Work Mode: Hybrid – 3 days onsite per week
• Contract Duration: 6-12 months
• Experience Required: 8–10+ years
• Job Type: Contract

We are looking for an experienced Databricks Architect to join our client's team in London. The ideal candidate will have extensive hands-on experience in Azure Databricks, data platform architecture, streaming workload migrations, and enterprise-level data engineering solutions.

The successful candidate will be responsible for designing and executing Databricks migration strategies, ensuring data integrity, optimizing performance, and providing technical leadership throughout the project lifecycle.

Key Responsibilities

• Design, architect, and implement scalable solutions using Azure Databricks, Unity Catalog, and Delta Lake.
• Independently design and validate managed table migration strategies.
• Design and execute streaming workload migrations using Structured Streaming and Delta Live Tables (DLT)/Lakeflow.
• Manage checkpoint continuity to ensure seamless migration without data loss.
• Plan and execute historical data backfills, reconciliation, and rollback strategies.
• Implement Databricks Asset Bundles (DAB) for automated deployments.
• Design, develop, and maintain CI/CD pipelines for Databricks environments.
• Optimize Databricks workloads, Spark jobs, and data pipelines for performance and efficiency.
• Plan and execute production cutovers with minimal business disruption.
• Implement Azure security best practices, including access controls, identity management, and data governance.
• Provide technical leadership, review solution designs, and guide engineering teams.
• Troubleshoot complex technical issues and ensure adherence to Databricks architecture best practices.

Required Skills and Experience

• 8–10+ years of overall IT experience, with strong experience in data engineering and architecture.
• Extensive hands-on expertise in Azure Databricks, Unity Catalog, and Delta Lake.
• Strong experience with Structured Streaming, Delta Live Tables (DLT), and Lakeflow.
• Proven experience in Databricks platform migrations, particularly managed tables and streaming workloads.
• Expertise in checkpoint continuity, historical backfill, data reconciliation, and rollback planning.
• Strong knowledge of Databricks Asset Bundles and modern deployment practices.
• Experience implementing CI/CD pipelines using Azure DevOps, GitHub Actions, or equivalent tools.
• Strong understanding of Azure security, governance, and access management.
• Experience in Databricks performance tuning and workload optimization.
• Proven experience managing production cutovers and migration activities.
• Strong architectural decision-making, problem-solving, and technical leadership skills.