Role overview

Description:

Designing, developing, and optimizing Azure Data Factory and Databricks pipelines from Oracle databases to Lakehouses
Designing and optimizing data models using Medallion Architecture principles
Designing, developing and optimizing data connections from Databricks Medallion Architecture to on-premises data sources for downstream consumers.
Designing and optimizing relational data models
Translating Informatica ETL to Azure Data Factory and Databricks ELT

Knowledge Transfer:

Transfer From DataOps/Cloud Data Engineer to Designated CSC Resource

When Knowledge Will Be Transferred:

Knowledge transfer must be completed one week prior to the end of the project or one week prior to the consultant leaving the ministry.

What Knowledge Will Be Transferred:

All deliverables, including design/supporting/release/training documents must be checked into designated version control repositories (for example, git repositories or SharePoint). All final documents and working drafts related to project requirements or solution design must be stored on designated project repositories (for example, SharePoint site, HPQC, TFS, etc.)
Project manager and designated ministry staff must be regularly informed in writing (by email) of where documentation has been stored and must be provided a minimum of one walk-through of all documentation as part of the final knowledge transfer activities.

How Knowledge Will Be Transferred:

Knowledge will be transferred through 1 on 1 sessions, emails, document updates and document review with the team.

Requirements

Experience and Skill Set Requirements:

Must Haves:

Experience developing ETL processes with Informatica, Azure Data Factory and Databricks (Python, Lakeflow, and SQL Optimization)
Experience with solution development with Databricks in Medallion Architecture.
Experience with Cloud data engineer must be familiar and experienced with different programming languages (Python, SQL, t-SQL, PL/SQL, Informatica, ADF, SSIS. Microsoft Fabric) and be able to integrate with many different platforms to create data pipelines, jobs, automate tasks, and write scripts.
Experience with Data pipeline and workflow development, orchestration, deployment, and automation and specializing in programming and pipelines to create and managing the dataflow, Delta Lake indexing, parallelism and movement of data.

Skill Set Requirements:

Technical Experience:

Experience developing ETL processes with Informatica, Azure Data Factory and Databricks (Python, Lakeflow, and SQL Optimization)
Experience with solution development with Databricks in Medallion Architecture.
Experience with Cloud data platforms, data management and data exchange tools and technologies
Experience with Commercial and opensource data and database development and management and specializing in data storage setting (Delta Lake) and managing cloud Data As a Service (DaaS). application Database as a Service (DBaaS), Data Warehouses as a Service (DWaaS), and other storage platforms (both in the cloud and on-premise).
Experience with Data pipeline and workflow development, orchestration, deployment, and automation and specializing in programming and pipelines to create and managing the dataflow, Delta Lake indexing, parallelism and movement of data.
Experience with Cloud data engineer must be familiar and experienced with different programming languages (Python, SQL, t-SQL, PL/SQL, Informatica, ADF, SSIS. Microsoft Fabric) and be able to integrate with many different platforms to create data pipelines, jobs, automate tasks, and write scripts.
Experience in Continuous Integration/Continuous Development/Deployment (CI/CD) and Data provisioning Automation
Experience with digital product, data analysis, data exchange, data integration APIs, data provisioning, and data security
Extensive experience in designing/developing and implementing data conversation and migration of VLD (Very large Data) of Online analytical processing (OLAP) and online transaction processing (OLTP) environments to Cloud Software-as-a-service (SaaS), Platform-as-a-service (PaaS) and Infrastructure-as-a-service (IaaS) environments.
Experience in design, development and implementation of fact/dimension model, data mapping, data warehouse, data lakehouse for enterprise.
Demonstrated experience developing star schema multi-dimensional models and documenting detailed design models.
Experience managing Cloud Data services for project delivery, including storage, repositories, Data Lakehouse, key vault, virtual machine, Azure Storage Account, SIHR, parquet files etc.
Experience with structured, semi-structured, unstructured data collection, ingestion, provisioning and exchange technological development of enterprise data warehouse and data lake and data lakehouse solutions and operational support.
Experience with DataOPS performance monitoring and tuning
Implementing and enforcing data quality checks and principles, including data validation, profiling, cleansing, and monitoring across pipelines
Applying data governance practices and information architecture standards, including development of conceptual, logical, and physical data models
Integrating and managing Microsoft Entra ID for secure authentication, authorization, and role-based access control across data platforms
Implementing data anonymization and masking techniques to protect sensitive and regulated data in compliance with privacy requirements
Developing and maintaining data lineage reports to provide end-to-end visibility and traceability of data movement and transformations
Databricks and/or Microsoft Fabric certifications are considered an asset

Core Skills:

Excellent analytical, problem-solving and decision-making skills; verbal and written communication skills; presentation skills; interpersonal and negotiation skills
Demonstrates development and solution design experience with technology, data, databases, statistics, applications, and networking techniques, tools and practices that enables the candidate to lead the design and development of business intelligence projects.
Experience in DataOPS best practices, and Agile project development and deployment
Demonstrates experience in working with and analyzing large and diverse data sets, data visualization, KPI development and data profiling to identify relationships and themes.
Demonstrates experience manipulating and analyzing complex, high-volume data from structured and unstructured sources.
Demonstrates experience creating detailed documentation for the tools that are developed, such as business requirements, system requirements and technical requirements.
Demonstrates experience with advanced data visualization tools and techniques, designing high quality interfaces to present information in a meaningful way.
Troubleshooting of production issues and creating fixes for any unplanned deployments.
Defect investigation, resolution and assignment to team members using defect/issue.
Demonstrated experience in coordinating and planning for the modernization of large, complex multi-platform, multitier information technology systems.
Experience with Azure DevOps.
Experience with AODA/WCAG compliance.
A team player with a track record for meeting deadlines

Project Experience and Techniques:

Demonstrates development and design experience with SDLC processes, Agile and Waterfall methodologies.
Demonstrates experience conducting system testing and unit testing, and conducting SIT, SAT and UAT Testing.
Demonstrates experience communicating regarding day-to-day tasks and issue tracking, reporting and facilitating resolution of issues and risks for all project activities.
Demonstrates experience gathering and consolidating input into changes required to business and/or system requirements, change requests, project artefacts, as well as leading requirement gathering sessions.

General Skills:

Demonstrates experience problem solving and decision making.
Demonstrates experience communicating clearly in both written and verbal formats.
Demonstrates experience working with and leading multiple teams representing various areas in delivering the project.
Demonstrates ability to work collaboratively, with the ability to coordinate multiple projects with competing priorities and a track record for meeting strict deadlines.
Demonstrates experience working collaboratively with other team members and/or groups and leading a team to ensure optimal solution integration.
Demonstrates experience leading a team and with providing technical advice and guidance.
Demonstrates experience working with both the business users and IT development teams to ensure business requirements are properly reflected in the system design and technical specifications.

Organization Experience:

Previous public sector experience in an organization of equivalent size.