The Data Engineer will design, develop, and execute scalable data ingestion and transformation pipelines to move data into Fabric Lakehouse / Warehouse environments, ensuring high data quality, consistency, and performance.

The role involves close collaboration with Data Architects, Integration teams, Business Analysts, and Functional SMEs to implement source-to-target mappings, transformation logic, and reconciliation processes throughout the migration lifecycle.

Ability to collaborate closely with Business Analysts, Data Architects, and Functional SMEs to understand source system data structures and business rules.
Analyse source system schemas, tables, relationships, and data patterns to identify relevant data elements required for migration.
Develop and maintain Source-to-Target Mapping (STM) documents
Identify data gaps, inconsistencies, and transformation requirements during the mapping process
Design and develop data ingestion pipelines using Microsoft Fabric Data Pipelines, Azure Data Factory, or Synapse pipelines.
Build data pipelines to extract data from SaaS platforms, legacy databases, files, APIs, and enterprise applications.
Load data into Microsoft Fabric OneLake (Lakehouse or Warehouse) environments.
Implement pipelines for initial bulk migrations, incremental loads, delta processing, and cutover migrations
Develop transformation logic using Fabric Notebooks (PySpark / Spark SQL), Dataflows Gen2, and SQL transformations in Fabric Warehouse
Implement business transformation rules defined in source-to-target mappings.
Standardize and cleanse data prior to loading into target systems.
Design transformation workflows following Bronze / Silver / Gold architecture patterns
Implement ingestion into Fabric Lakehouse tables (Delta format).
Manage partitioning, indexing, and file compaction strategies
Implement validation checks including record counts, field-level reconciliation, completeness checks, and consistency validation
Build automated data quality validation frameworks.
Monitor pipeline execution and troubleshoot issues.
Ensure migration runs meet defined performance and timing requirements.
Optimize Spark jobs and SQL queries in Microsoft Fabric.
Implement parallel processing and batch strategies for large datasets.
Implement CI/CD pipelines using Azure DevOps and Fabric deployment pipelines.
Maintain version control for notebooks, pipelines, SQL scripts, and transformation logic
Implement security using Microsoft Entra ID, RBAC, and Fabric workspace security.

Core Data Engineering Skills

Strong SQL development skills
Experience building ETL/ELT pipelines
Experience working with large-scale datasets
Understanding of data modelling and data architecture

Microsoft Fabric Skills

Fabric Lakehouse
Fabric Data Pipelines
Fabric Notebooks
Dataflows Gen2
Fabric Warehouse
One Lake architecture
Delta Lake format

Azure Platform Skills

Azure Data Factory
Azure Storage / ADLS Gen2
Azure DevOps
Azure Key Vault
Microsoft Entra ID

Programming Skills

SQL
Python (PySpark preferred)
Spark SQL

Salary negotiable according to ability.
Health insurance, social insurance, unemployment insurance, community insurance according to the provisions of law.
Consider salary increase depending on candidate's ability.
Bonus for business performance based on business situation and employee's contribution capacity.
Work in a dynamic environment, develop yourself quickly.
Be trained in both hard skills and soft skills in professional careers.
Team building and annual trip.
Participate in the SoftWorld Care program for all employees.