Data Engineer - Commodities
The advert
As posted by Millennium Capital ManagementData Engineer - Commodities
About Millennium
Millennium is a global, diversified alternative investment firm, founded in 1989. Defined by evolution, innovation and focus, Millennium’s mission is to deliver results for our investors.
Our people are empowered with both independence and support: the autonomy to pursue ideas with conviction and the backing of a global network committed to collaboration, disciplined risk management and continuous learning. With opportunities to deepen expertise and accelerate development, talent at Millennium is equipped to adapt, evolve and build lasting impact over time. Discover how transformative growth accelerates impact.
Meet the Team
The Commodities Technology team builds and operates a data platform that aggregates and curates critical commodities data, including weather, supply and demand, storage, transportation, and other fundamental and alternative datasets. The team’s curated content layer supports Portfolio Managers and researchers in understanding markets and constructing trades. As part of Millennium’s Information Technology organization, the team develops flexible, scalable technology and proprietary systems that support advanced analytical and trading capabilities.
What You'll Do
- Design, implement, and maintain end-to-end ETL workflows in Python and SQL to ingest and transform commodities data from multiple vendor and internal sources.
- Build and maintain standardized data models, schemas, and metadata that make commodities datasets easy to understand, discover, and reuse.
- Schedule, monitor, and manage data pipelines using Airflow or similar workflow orchestration tools to ensure reliable, timely data delivery.
- Implement validation, reconciliation, and anomaly-detection controls to maintain data completeness, accuracy, and consistency.
- Apply AI to automate schema inference across structured and semi-structured data sources, manage schema drift, and accelerate scalable ingestion development.
- Use AI-driven data-quality, observability, and documentation capabilities to detect anomalies, monitor data health, and produce clear lineage and technical documentation.
- Maintain high-quality, production-ready code and repeatable deployments through Git, GitHub Actions, PyTest, and automated testing practices.
- Partner with commodities Portfolio Managers, researchers, data analysts, and data strategists to refine datasets, definitions, and documentation based on evolving business needs.
What You Bring
- 5+ years of experience in data engineering, analytics engineering, or a similar role building and maintaining ETL pipelines.
- Strong Python and SQL skills, including experience working with large datasets and complex data transformations.
- Hands-on experience with Airflow or comparable workflow orchestration tools.
- Experience with Git, CI/CD pipelines such as GitHub Actions, and automated testing frameworks such as PyTest.
- Strong attention to detail, data quality, and documentation, with the ability to identify edge cases and protect data integrity.
- Ability to work independently, communicate effectively with technical and non-technical stakeholders, and manage multiple concurrent initiatives.
- Knowledge of commodities markets and data, including weather, supply and demand, storage, freight, or flows, is preferred.
- Experience with data warehousing, columnar storage formats, analytic databases, data catalog or governance tools, or financial services, trading, or research-driven environments is preferred.
Sponsor check
Millennium Capital Management holds a Home Office sponsor licence (A-rated · Skilled Worker route), checked against the register published 19 Nov 2026. We read this advert: no sponsorship restrictions. Home Office FOI figures: 302 visa sponsorships in 2021–24.
How we check sponsorshipSponsor check
- Sponsor register
Millennium Capital Management is on the Home Office register of licensed sponsors: A-rated, licensed since at least 2021.
Home Office register · published 19 Nov 2026
- Visa sponsorships
302 visa sponsorships in 2021–24 (Home Office FOI 2025/12841).
- From the advert
The advert doesn't mention sponsorship.
Read from the advert below
Facts from the Home Office register of licensed sponsors and Home Office FOI figures. We don't judge employers; check the advert and the register yourself.
Our estimate from the job title
- Likely occupation
- SOC 2134 · Programmers and software development professionals
- Going rate
- £54,700 a year (£28.05/h)estimate
- Visa route
- Likely eligible for Skilled Worker sponsorship
- Visa salary for this occupation
- £54,700 · new entrants £38,300estimate
- Typical UK pay
- £56,914 median full-time
The employer picks the real occupation code on the Certificate of Sponsorship. If it picks a different one, the visa salary changes. How we read this
Millennium Capital Management
- Sponsor licence
- A-rated · Worker
- Visa sponsorships
- 302 · 2021–24
- Visa routes
- Global Business Mobility: Senior or Specialist Worker, Skilled Worker
- Location
- London
Millennium Capital Management is a licensed UK visa sponsor (A-rated). Home Office figures released under the Freedom of Information Act show Millennium Capital Management assigned 302 Certificates of Sponsorship in 2021–24 and 81 in 2025 for Skilled Worker and Global Business Mobility visas.
Similar sponsor-licensed roles
The closest matches at other employers with active sponsor licences.
Engineering Manager - Commodities Kpler · London Sponsorable from abroad IG Data Engineer IG Group Holdings · London Sponsorable from abroad
Reliability Engineer Two Sigma · London Sponsorable from abroad
Senior Director, Engineering - Energy & Commodities (Machine Learning, AI & Data) Sapient · London Sponsorable from abroad BB Senior Engineer - Equities Data Platform Barclays Bank UK · London Sponsorable from abroad
Data Engineer MULBERRY (DESIGN) · London Sponsorable from abroad MG Data Engineer, Data Platform Macquarie Group Services Australia Pty Ltd (UK Branch) · London Sponsorable from abroad
Python Software Engineer - Data Infrastructure MAN · London Sponsorable from abroad New roles like this
Added in the last 48 hours by other sponsor-licensed employers.
Senior Data Engineer Hack the Box · London Sponsorable from abroad
Senior Software Engineer - Data Engineering Moody's Investors Service · Manchester Sponsorable from abroad IB Data Engineer Investec Bank · United Kingdom Sponsorable from abroad HS Data Engineer HCC Service Company Inc · London Sponsorable from abroad
Principal Data Engineer FUNDING CIRCLE · London Sponsorable from abroad UM Database Administrator Universal Music Operations Ltd · United Kingdom Sponsorable from abroad Roles like this at sponsors
Licensed UK visa sponsors
Engineering Manager - Commodities Kpler · London Sponsorable from abroad IG Data Engineer IG Group Holdings · London Sponsorable from abroad
Reliability Engineer Two Sigma · London Sponsorable from abroad