AKAbdullah Khan

Enterprise • Warburg Pincus • 2023

D365 High-Throughput ETL Platform

Designed ingestion for 100+ D365 F&O tables with strict delivery windows into Synapse/Delta Lake.

Challenge

Meeting tight data freshness SLAs while coordinating many I/O-bound jobs and event-driven file flows.

Solution

Developed multi-threaded orchestration, integrated Event Grid + Azure Functions, and used Redis for state-aware pipeline control.

Outcomes

  • Delivered 10-minute SLA target
  • Scaled ingestion across 100+ tables
  • Improved state management reliability

Technology Stack

  • Azure Synapse
  • PySpark
  • Azure Functions
  • Event Grid
  • Redis
  • Delta Lake

Ready To Build

Want this delivery model for your team?

I can map the same principles to your architecture, timeline, and constraints.

Talk About Your ContextBack to Projects