Data Engineer

Project description

We've been engaged by an Investment Banking client in the FM potfolio . PTS app is an inhouse build system which is silver source of all settlement information originated from GPTM. PTS app is in CDP (Cloudera data platform) platform which bag data ecosystem. It will be migrated to a different datacentre and CDP will be decommissioned . Hence PTS application, we need to rewrite code approx. 40% to rebuild from scratch like develop data ingestion framework, extraction frame work and migration of existing components to new data centre from west to East.

Responsibilities

• Design, develop and maintain scalable data pipelines and ETL workflows using Azure Data Factory, Databricks and Microsoft Fabric. Develop data processing solutions using Apache Spark and Databricks SQL. Perform data ingestion and integration across on-premise and cloud data sources. Lead or support migration of analytics and data workflows from legacy platforms such as Dataiku to Databricks. Develop and optimize complex SQL queries, stored procedures, views and database structures. Perform SQL query optimization, indexing and performance tuning to improve data retrieval and application performance. Implement and maintain CI/CD pipelines using Azure DevOps and GitHub. Apply data security practices including RBAC, encryption and audit logging. Implement data governance, data lineage and data quality practices across data engineering workflows. Develop and maintain Power BI dashboards, reports and data models where required. Work closely with data scientists, business analysts, QA and application development teams.

SKILLS

Must have

• 4+ years of experience in Data Engineering / Azure Data Engineering Strong experience with Azure Data Factory (ADF) Hands-on experience with Databricks Strong knowledge of Apache Spark Strong SQL development and SQL performance tuning Experience with ETL / Data Pipelines Experience with Microsoft Fabric Experience with Azure DevOps and GitHub Knowledge of CI/CD Experience with Data Governance and Data Lineage Knowledge of RBAC and enterprise data security Good understanding of cloud-based data architecture

Nice to have

• Experience of Agile/Scrum development methodologies is a plus • Be nice, respectful, able to work in a team • Willingness to learn