AKAbdullah Khan

Enterprise • National Bank of Belgium • 2022-2024

IDF Ingestion Delivery Framework

Primary engineer for a centralized ingestion framework enabling the bank's cloud lakehouse teams to onboard sources faster with stronger governance.

Challenge

Multiple teams needed reusable ingestion with file detection, SCD2 support, and consistent orchestration across varying source formats.

Solution

Built PySpark + SparkSQL ingestion modules, metadata-driven Airflow DAG patterns, and Delta Lake optimization strategies (partitioning and Z-Order).

Outcomes

  • Reduced onboarding effort through metadata-driven orchestration
  • Improved performance on high-volume datasets
  • Introduced stronger engineering quality via SOLID patterns + pytest

Technology Stack

  • Azure
  • Databricks
  • PySpark
  • SparkSQL
  • Delta Lake
  • Airflow
  • Unity Catalog

Ready To Build

Want this delivery model for your team?

I can map the same principles to your architecture, timeline, and constraints.

Talk About Your ContextBack to Projects