Dec 2021 - Present
Noida, Uttar Pradesh, India
Currently working as Lead engineer in developing ETL Pipelines using Azure Data Factory, have been deloping multiple data pipelines comprising different data sources for example Oracle, SQL Servers, Sharepoint List, API's.
Designing the data pipelines in a way so that they can produce reuseability, and also giving multiple options that involves activiating a particular table for data load, setting up flag for full/incremental load on the basis of Business Needs, Giving option to load for a particular site level.
Have been using Data lake, Azure SQL Database, Azure Databricks, Azure Synapse Analytics as intermediate layer for data storage and end products involving Facts and Dimensions that can be further consumed by Azure Analaysis Services.
Worked on writing SQL Logic that involves pipeline re-run process so that pipeline can be re-run from the same watermark point as it was failed to avoid data loss.
Also worked on Logic App to ingest the data from SQL Database into Sharepoint List on the basis of Row Active flag.