Data Engineering is an emerging discipline which can help facilitate the utilization of data as an asset within the business.

The development of robust and reusable data engineering solutions is required to support the long-term goals of the organization.

As part of the Digital Services Team, the Senior Data Engineer will contribute to the specification, building and maintenance of data engineering platforms.

The Data Engineer must be able to effectively work with key business users and IS colleagues within the organization to provide a high-quality data engineering solution, in addition provisioning Subject Matter Expert (SME) best practice guidance across the domain.

The Data Engineer must also be able to effectively work with centralized teams to ensure the solutions delivered within the affiliate align to company standards and procedures.

Duties and Responsibilities

Functional Expertise:

• Providing functional expertise and guidance around leveraging the capabilities of data engineering platforms to business stakeholders and data engineering community.
• Liaising between business key users and the data engineering function - for investigating, analysing, and implementing cloud-based data engineering solutions (including documenting, data mapping, analytical processing and source system modifications).
• Taking the lead role in the data engineering community ensuring that best practice is followed as per the technical framework.

Solution Development:

• Producing end-to-end solutions for new and existing data engineering projects including delivery of configuration enhancements from concept to solution and the designing and implementing of data models.
• Developing data manipulation services (e.g., machine learning, stored procedures, USQL, TSQL, ETL, Python).
• Run proof of concepts and investigate opportunities for use of innovative cloud solutions.

Solution Support:

• Managing complex incident resolution on productionised data engineering solutions (root-cause analysis, problem statements).
• Responsible for ensuring the production services are optimally configured and aligned to best practices to protect production data.
• Supporting the development of automated business intelligence reports and data science projects in conjunction with business stakeholders including digital and data, technical data and innovation and self-service business intelligence.
• Ensuring support of productionised digital products.
• Working in close collaboration with IS Platform Teams to define and develop the industrialised data engineering support processes.
• Ensuring that system integrity is maintained by following and ensuring compliance with agreed standards and procedures.

Data Strategy:

• Assisting in the definition of the data engineering strategy, solutions, and roadmaps, ensuring they meet both business and performance goals in line with the organizations IS strategy and data governance guidelines.
• Identifying new data engineering techniques, tools, or best practices, documenting them in the technical framework and assisting in integrating into the wider data engineering community.
• Participating in monthly security reviews to ensure compliance for all cloud applications.
• Participating in quarterly software component reviews to avoid obsolescence issues across cloud components.
• Participating in monthly cost optimization reviews to ensure applications run costs are kept as low as possible.

Required Skills and Experiences

• A numerate honours degree (e.g., Computer Science, Mathematics, Engineering) with demonstrable industry experience.
• Ability to develop functional specifications for business requirements.
• Experience writing robust, reusable code to create production-strength data pipelines.
• Extensive knowledge of scripting languages, preferably Python.
• The ability and motivation to learn new programming languages and techniques when necessary.
• Strong SQL knowledge.
• Knowledge of distributed computing, data and application architectures, basic networking, security, and infrastructure.
• Proven production experience and best practice knowledge of cloud platforms, features, and capabilities (Microsoft Azure components).
• Proven production experience and best practice knowledge of Azure Data Factory.
• Knowledge of best practice principles behind cloud-based data ingestion, storage and associated data transformations.
• Demonstrable experience of modern ETL technologies and packages, including upstream enterprise platforms.
• Experience integrating cloud services to work with on premise data sources and clients.
• Excellent interpersonal, communication skills, written and verbal combined with effective technical presentation skills.
• Organising, methodical and able to multi-task whilst working under tight deadlines.

Desired Skills and Experiences

• A thorough understanding of the upstream industry and knowledge of upstream companies’ main strategic drivers.
• Knowledge of multi-tier, high availability, cloud and on-premises data engineering solutions in a complex multi-site enterprise environment.
• Experience of working within a Power BI based reporting environment.
• Experience of working within a dynamic agile based team.

TMM Recruitment INDIT

Email:it_team@tmmrecruitment.com