Data Engineer
Licensed sponsor London Contract Posted 2 hours ago
RED is currently looking for a Data Engineer with experience migrating to Databricks within a financially regulated environment. This is a hybrid, 3-month contract with chances to extend.
Contract Information
- Location: Hybrid in London
- Contract: Inside/Outside TBD
- Duration: 3 Months with a chance to extend
- Language: English
Key Responsibilities
- Design, develop, and implement scalable data pipelines to migrate historical quoted data into Databricks.
- Support the development and implementation of the Databricks migration framework.
- Build and optimise ETL/ELT pipelines using Python and PySpark.
- Extract, transform, and load large volumes of historical financial data from existing systems.
- Ensure data quality, accuracy, and consistency throughout the migration process.
- Develop data validation and reconciliation processes to verify the completeness and integrity of migrated datasets.
- Work closely with Data Architects, Engineers, and business stakeholders to ensure successful delivery.
- Troubleshoot migration issues, resolve performance bottlenecks, and optimise data processing.
- Maintain technical documentation covering pipeline development, migration processes, and data transformations.
- Ensure all data engineering activities align with financial services regulatory, security, and governance requirements.
Essential Skills & Experience
- 5+ years of commercial experience working as a Data Engineer.
- Strong hands-on experience with Databricks, including developing and maintaining data pipelines.
- Advanced proficiency in Python and PySpark for large-scale data processing and transformation.
- Proven experience delivering data migration projects, ideally involving legacy systems migrating to cloud-based data platforms.
- Previous experience working within financial services or another highly regulated financial environment.
- Strong SQL skills and experience working with large, complex datasets.
- Experience implementing data quality checks, validation, and reconciliation processes.
- Understanding of data governance, security, and regulatory requirements within financial services.
- Ability to work independently and deliver against agreed project milestones.
If you're interested, please apply with your updated CV.
