Senior Data Engineer - Python, Databricks

Job Responsibilities

•  Develop workflows and ELT data pipelines using Python, Spark/PySpark, and
• Databricks
•  Onboard enterprise datasets into Palmos, including ingestion, transformation, and
• validation of data assets
•  Build, test, and maintain scalable data pipelines and data architectures that support
• enterprise controls and analytics use cases
•  Build buisness controls algorithms for communications data to identify anomalies
•  Build data completeness and integrity controls for the pipelines
•  Apply data engineering best practices for performance optimization, reliability, and
• maintainability
•  Use SQL extensively and work with both relational and NoSQL data stores
•  Partner with producers to understand data requirements and translate them into
• production-ready solutions
•  Apply SDLC practices including CI/CD, testing, and operational monitoring to ensure
• pipeline stability
•  Contribute to reusable frameworks and standards to accelerate onboarding and
• pipeline delivery
•  Identify data issues, anomalies, and optimization opportunities to improve data
• quality and performance
•  Leverage enterprise-authorized AI coding assist tools within the work environment to
• improve code quality, delivery speed, and productivity across complex deliverables
• (e.g., code generation/refactoring, unit test creation, documentation), while validating
• outputs through peer review, automated testing, and secure coding standards;
• contribute learnings and reusable patterns to improve broader team effectiveness

Required Qualifications, Capabilities, and Skills

•  Hands-on experience with Databricks, Spark/PySpark, Python, and SQL
•  Experience developing and maintaining data pipelines and data processing systems
•  Understanding of the data lifecycle, including ingestion, transformation, storage, and
• consumption
•  Knowledge of cloud platforms (AWS) and distributed data processing
•  Experience with SDLC practices including CI/CD, testing, and deployment
•  Strong problem-solving skills and ability to troubleshoot data and pipeline issues
•  Ability to collaborate effectively within agile teams and across stakeholders
•  Hands-on experience using enterprise-authorized AI-assisted software development
• tools within the work environment (e.g., for coding, test creation, troubleshooting, or
• documentation) with demonstrated ability to critically evaluate, validate, and refine AI-
• generated outputs for correctness, performance, and security
•  Understanding of responsible AI use in engineering workflows, including data
• sensitivity considerations, secure handling of inputs/outputs, and adherence to
• resiliency and security expectations; ability to guide peers on safe and effective
• usage within team practices

Preferred Qualifications, Capabilities, and Skills

•  Experience with Databricks lakehouse, Databricks Genie, Delta Lake, and medallion
• architecture
•  Familiarity with enterprise data platforms and data mesh principles
•  Exposure to data quality, observability, and metadata management tools
•  Experience supporting analytics, reporting, or AI/ML workloads
•  Experience working on regulatory controls