Senior Data Engineer

Role purpose:

The Senior Data Engineer plays a pivotal role within the Data team of the Digital and Information function, assuming a leadership position in ensuring the availability, integrity, quality, and accuracy of data. With a wealth of experience, the Senior Data Engineer leads the design, development, and maintenance of data pipelines and workflows, driving the organisation's data processing capabilities and enabling the seamless delivery of BI reporting and analytics.

In addition to core responsibilities, the Senior Data Engineer takes a proactive role in advancing our Azure-based data platform, leveraging cutting-edge technologies like Azure Data Factory and Azure Databricks. This role presents exciting opportunities to spearhead digital transformation projects and innovate within the data landscape.

Operating within Agile principles and methodologies, the Senior Data Engineer collaborates closely with fellow data engineers and key stakeholders, assuming a leading role in shaping and managing the portfolio of reporting and data requirements. Leveraging their extensive expertise, the Senior Data Engineer contributes significantly to refining the overall data strategy, ensuring alignment with organisational objectives and driving continuous improvement initiatives.

Key Responsibilities:

• Ownership and delivery of the design, development and maintenance of highly scalable and robust data pipelines and workflows utilising Azure Data Factory and Databricks.
• Lead data transformation and integration efforts to augment data from diverse sources, ensuring data governance and regulatory compliance throughout the process.
• Establish and enforce data quality standards through the implementation of rigorous data validation processes, ensuring the utmost accuracy, completeness, and consistency of the data ecosystem.
• Serve as a key liaison between cross-functional teams, including data scientists, analysts, and business stakeholders, to comprehensively understand data requirements, conduct User Acceptance Testing (UAT), and deliver tailored solutions that align with business objectives.
• Proactively monitor pipeline performance, swiftly identifying and resolving any potential issues to uphold seamless data processing operations and minimize downtime.
• Drive comprehensive documentation of data pipelines, workflows, and procedures to foster knowledge sharing and ensure ongoing system maintainability and scalability.
• Demonstrate proficiency in managing and resolving 2nd line data and reporting support requests through designated ticketing systems, mitigating disruptions to business operations with timely and effective resolutions.
• Actively engage in daily stand-ups, weekly stakeholder meetings, and retrospectives, providing valuable insights and contributing to continuous improvement initiatives across the team and broader organization.

Technical / Professional Qualifications / Requirements:

• Hands-on advanced experience in designing, developing, and managing data pipelines using Azure Data Factory.
• Strong working knowledge of Databricks for data processing, analytics and Machine learning.
• Advanced experience with data modelling techniques and data warehousing concepts.
• Advanced proficiency in SQL for data querying and manipulation and good knowledge of Python.
• Ability to work effectively in a fast-paced, dynamic environment and manage multiple priorities, following the Agile way of working
• Ability to lead and analyse complex data problems and propose effective solutions.
• Excellent communication skills with the ability to collaborate effectively with cross-functional teams.

Benefits Include:

* 25 days AL plus bank holidays
* 1 Wellbeing day
* Private Medical
* Pension - 8% matched pension
* Life Assurance x4 basic salary
* Income protection
* Dental plan - voluntary benefit