Senior Data Engineer
Licensed sponsor United Kingdom Contract Posted 5 hours ago
Senior Data Engineer – Azure Databricks
- Location: London, UK
- Work Mode: Hybrid – 3 days onsite per week
- Contract Duration: 6-12 Months
About the Role
We are seeking an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks to design, develop, migrate, and optimize scalable data pipelines on a modern cloud data platform.
The ideal candidate will have extensive experience in Python, SQL, PySpark, Spark, Delta Lake, DBT, and Azure, with proven expertise in building production-grade data pipelines, implementing data quality frameworks, and delivering high-performance data solutions.
Key Responsibilities
- Design, develop, and migrate scalable data pipelines using Azure Databricks.
- Build and maintain end-to-end ETL/ELT pipelines, from raw data ingestion to curated data layers.
- Develop and optimize large-scale data processing workflows using Python, PySpark, Spark, and SQL.
- Implement data transformations and modelling using DBT.
- Design and maintain Delta Lake architectures for reliable and high-performance data storage.
- Develop dimensional, curated, and semantic data models to support analytics and reporting.
- Implement data validation, automated quality checks, monitoring, and pipeline reliability frameworks.
- Architect and deploy data engineering solutions using Microsoft Azure cloud services.
- Troubleshoot and optimize pipeline performance, scalability, and efficiency.
- Collaborate with technical and business stakeholders to translate requirements into robust data engineering solutions.
Required Skills and Experience
- Strong hands-on experience with Azure Databricks in production environments.
- Excellent programming skills in Python and SQL.
- Extensive experience with PySpark and Apache Spark.
- Hands-on experience with Delta Lake architecture and data processing.
- Practical experience with DBT (Data Build Tool).
- Strong understanding of data modelling, ETL/ELT, and data warehousing concepts.
- Experience designing, developing, and migrating end-to-end data pipelines.
- Strong experience with Microsoft Azure cloud services.
- Knowledge of data quality frameworks, validation, monitoring, and performance optimization.
- Ability to independently deliver scalable, reliable, production-grade data engineering solutions.
