Role overview

Job Summary:

The Senior DataOps/Cloud Data Engineer is responsible for designing, developing, and optimizing scalable data pipelines and cloud-based data platforms. This role focuses on implementing DataOps best practices, automating workflows, and supporting enterprise data solutions including data lakes, warehouses, and lakehouses in cloud environments. The engineer will collaborate with cross-functional teams to enable data integration, quality, governance, and migration initiatives.
Responsibilities:

Design and develop end-to-end data pipelines from source systems to end users.
Optimize and automate data workflows and pipeline performance.
Implement DataOps practices including CI/CD, deployment automation, and monitoring.
Develop and maintain cloud data platforms and services including data lakes, warehouses, and lakehouses.
Automate metadata collection, data quality, and lineage enablement.
Support cloud modernization and data migration projects.
Collaborate with architects, data modelers, and governance teams to operationalize data standards and metadata.
Troubleshoot complex data engineering and integration issues.
Ensure data security, governance, and operational best practices are followed.

Required Skills:

Proven experience with Databricks, Delta Lake, and Databricks SQL.
Strong knowledge of Medallion Architecture.
Experience with Azure Data Services and Azure Data Factory.
Proficiency with Informatica or equivalent data integration platforms.
Operationalizing common data models and metadata repositories.
Expertise in designing and supporting cloud-native data platforms and scalable ETL/ELT pipelines.
Experience implementing DataOps including deployment automation, testing, and operational support.
Strong analytical, problem-solving, and communication skills.

Desired Skills:

Experience with Microsoft Fabric and Microsoft Purview.
Knowledge of graph technologies and semantic repositories.
Familiarity with AODA and related accessibility standards.
Experience working in public sector or operational environments.
Ability to adapt to tight deadlines and shifting project priorities.

Rated Criteria:

Cloud Data Platform Design and Implementation - 25 Points
Data Pipeline Development and Optimization - 20 Points
DataOps and CI/CD Automation - 15 Points
Metadata, Data Quality, and Lineage Enablement - 15 Points
Experience with Databricks and Delta Lake Technologies - 15 Points
Integration and Migration Expertise - 10 Points

Total: 100 Points Deliverables:

Robust data pipelines and integration services supporting enterprise data needs.
Automated metadata collection and synchronization processes.
Cloud data platform implementation and configuration documentation.
Enabled data quality and lineage capabilities within cloud environments.
CI/CD pipelines and DataOps automation frameworks.
Data migration and interoperability services to cloud platforms.
Cloud-hosted reusable data products and services.
Metadata and governance platform integrations supporting enterprise standards.

Must Haves:

Experience with Databricks, Delta Lake, and Databricks SQL.
Knowledge of Medallion Architecture.
Hands-on experience with Azure Data Services and Azure Data Factory.
Proficiency with Informatica or equivalent integration tools.
Ability to operationalize common data models and metadata repositories.

Public Sector Experience:

Preferred