Deepak Kumar | Data Engineer — GCP, PySpark, BigQuery

stage: about

From raw data to decisions people trust

I'm a Data Engineer currently working across batch and real-time pipelines — most recently at DB Schenker, where I spent two years building ETL/ELT workflows on GCP (BigQuery, Cloud Storage, Cloud Composer) that process millions of records without anyone needing to babysit them.

My focus is the unglamorous part of data engineering that actually matters: pipelines that are reliable, queries that are fast, and dashboards that load before the meeting starts. I've cut ETL processing time by 30%+, built Power BI dashboards that shaved 25% off reporting turnaround, and shipped a courier-tracking system to 8+ locations that's still running today.

Outside of pipelines: I'm comfortable in PySpark for big-data transforms, SQL for the analysis everyone actually reads, and increasingly, dbt and CI/CD for making sure none of it breaks silently.

Based inGurugram, Haryana, India

Current roleData Engineer, DB Schenker

Core cloudGoogle Cloud Platform

EducationB.Tech, Computer Science

Open toData Engineer / Analytics Engineer roles

Let's talk

GCP Engineer

stage: tech stack

Tools that move data, not just tickets

Cloud & GCP

Google Cloud Platform
BigQuery
Dataproc
Dataflow
Cloud Composer
Cloud Storage
Pub/Sub
Cloud Functions
IAM

Big Data & Pipelines

Apache Spark
PySpark
Apache Airflow
ETL / ELT
Medallion Architecture
dbt
Hadoop (HDFS)
Hive
Sqoop
DAG orchestration
Shell scripting

Languages

Python
SQL
MySQL
Oracle SQL
Java
C / C++

Analytics & BI

Power BI
Data Modeling
Advanced Excel
Git / GitHub
VS Code
RESTful APIs

Adjacent Systems

Power Apps
Power Automate
EDI
WMS
Supply Chain (NGW)
SharePoint

stage: experience

Where the pipelines actually ran

Data Engineer – Executive (Data Integration)

Jul 2023 – Oct 2025

DB Schenker · Gurugram, Haryana

Designed scalable batch and real-time pipelines in Python and SQL, processing millions of records with optimized query performance across distributed systems.
Built and optimized ETL/ELT workflows on GCP (BigQuery, Cloud Storage) — improving data processing efficiency by 30%+.
Processed large-scale datasets using PySpark, improving transformation speed for big-data handling.
Designed batch + streaming ingestion pipelines for near-real-time data availability.
Built scalable data models and Power BI dashboards, cutting reporting time by 25%.
Built CI/CD pipelines for data workflows, enabling faster, more reliable deployments.
Independently developed a Courier Tracking system (Power Apps + Power Automate + SharePoint), deployed across 8+ locations with a live dashboard.

BigQuerydbtDataproc AirflowPower BI GitHub CI/CDRESTful APIsPySparkGCP

Graduate Engineer Trainee

Aug 2022 – Jun 2023

Manipal Business Solutions · Gurugram, Haryana

Optimized SQL queries, ETL workflows, and database schemas (procedures, indexing) to improve performance and reporting efficiency.
Integrated backend workflows with frontend UI systems, collaborating cross-functionally on end-to-end data solutions.

GCPPySparkSpark BigQuerySQL TerraformETLDatabase Design

stage: projects

Systems built at production scale

Each project includes the architecture decision, the scale it ran at, and why specific tools were chosen — not just what was built.

Personal Project May 2025

GitHub

Flipkart Insight Engine

Scalable e-commerce analytics pipeline · Apache Spark · Databricks · Python · NLP

Why these tools?

Dataset exceeded single-machine memory limits — chose Apache Spark on Databricks over pandas because distributed processing cut transformation time from ~40min to under 6min. Databricks managed cluster autoscaling so compute cost scaled with actual load, not peak.

Architecture · Flipkart Insight Engine

Raw Product
Dataset

CSV / Parquet

PySpark
Preprocessing

Databricks Cluster

Analytics
Engine

Sentiment · Correlation · Time-series

Insight
Reports

Stakeholder Dashboards

ScaleMulti-GB product datasets

Perf gain~40min → <6min transform

PatternBatch · distributed

6xfaster data transformation vs single-node pandas

3analysis types: sentiment, correlation, time-series

100%automated — zero manual preprocessing steps

Chose Spark over pandas because dataset size exceeded single-node memory; distributed processing on Databricks reduced job time by ~85%
Built NLP sentiment pipeline on product reviews — extracted positive/negative signal to surface underperforming SKUs for category teams
LEAD/LAG window functions on time-series data revealed seasonal demand spikes 2 weeks before they peaked in sales figures
Data normalization layer ensured consistent schema across 6 product categories before analytics ran — no silent null propagation

DB Schenker · Production 2023 – 2025

Real-Time Logistics Data Platform

30GB/day · 6 sources · GCP · BigQuery · Dataflow · Pub/Sub · Cloud Composer · PySpark

Why these tools?

Logistics data arrives from 6 heterogeneous sources (EDI feeds, APIs, WMS events) at different cadences — some batch, some streaming. Chose Pub/Sub + Dataflow for streaming ingestion because it decouples producers from consumers and handles backpressure automatically. BigQuery over a traditional warehouse because columnar storage on 30GB/day queries ran 4× faster without manual indexing. Cloud Composer (Airflow) for orchestration because DAG-based scheduling gave full visibility into dependency chains — critical when upstream delays cascade.

Architecture · Real-Time Logistics Data Platform

INGEST

EDI Feeds

REST APIs

WMS Events

STREAM

Pub/Sub

Event queue

Dataflow

Apache Beam

TRANSFORM

Dataproc
PySpark

Heavy transforms

Cloud Composer
Airflow DAGs

Orchestration

STORE

BigQuery

Bronze→Silver→Gold

Cloud Storage

Raw archive

SERVE

Power BI

Analysts

Daily volume30 GB / day

Sources6 heterogeneous

LatencyNear real-time streaming

PatternLambda — batch + stream

30GBprocessed daily across 6 source systems

30%+ETL efficiency improvement on GCP

25%reduction in reporting turnaround time

<5minlatency on streaming ingestion via Pub/Sub

Pub/Sub chosen over Kafka because managed infrastructure on GCP eliminated broker management overhead — team of 2 couldn't justify ops burden of self-hosted Kafka at this scale
Dataflow (Apache Beam) for streaming because unified batch/stream SDK meant same transform code runs in both modes — reduced codebase duplication by ~60%
Dataproc for heavy PySpark jobs instead of running on Composer workers — isolated compute per job, no resource contention with orchestration layer
Medallion architecture in BigQuery (Bronze → Silver → Gold) — Bronze for raw immutable ingestion, Silver for cleaned/joined, Gold for aggregated business metrics. Enabled point-in-time reprocessing without data loss
Built CI/CD for DAG deployments — bad DAG pushes no longer blocked the entire pipeline; automated tests caught schema mismatches before production
Power BI dashboards on Gold layer served 4 business teams; reduced manual reporting from 3hrs/week to automated refresh

DB Schenker · Production 2024

Courier Tracking Ops Platform

Real-time ops dashboard · Power Apps · Power Automate · SharePoint · Deployed 8+ locations

Why these tools?

Operations team needed real-time package visibility but had no engineering budget for a custom app. Chose Power Apps + Power Automate + SharePoint Lists as the data store because: zero infra cost, native integration with existing Microsoft 365 stack, and non-technical ops staff could update records without SQL access. Tradeoff: SharePoint Lists cap at 5,000 rows per view — solved with indexed columns and filtered views per location.

Architecture · Courier Tracking Platform

Courier
Scan Events

8+ locations

Power
Automate

Event triggers

SharePoint
Lists

Indexed store

Power Apps
Dashboard

Live status

Locations8+ depots

Infra cost₹0 (M365 stack)

Constraint solvedSP 5k row limit via indexed cols

8+locations live, deployed independently

0engineering infra cost using existing M365

Real-timepackage status visible to ops without SQL access

Sole engineer — scoped, built, and deployed end-to-end without a product manager or dedicated budget
Hit SharePoint 5,000-row view limit during rollout — resolved by adding indexed columns on location + status fields and splitting views per depot, restoring query performance
Power Automate flows trigger on scan events and update SharePoint Lists within seconds — ops team sees package movement in near-real time
Rolled out across 8 depot locations with custom filtered views per location — no single ops user sees another location's data

Personal Project

GitHub

Employee Attrition SQL Analysis

Advanced SQL analytics · MySQL · Window Functions · Statistical pattern detection

Why SQL-only, no Python?

Deliberately built this entirely in SQL to demonstrate that complex analytics — window functions, ranking, trend detection — don't always need a Python layer. At scale companies, pushing computation to the database engine (rather than pulling data into pandas) is faster and cheaper. This project was a proof of that principle.

5advanced SQL techniques: LEAD, LAG, DENSE_RANK, subqueries, CTEs

3attrition risk factors identified with statistical backing

Used LEAD/LAG window functions to track income changes over time — found employees with stagnant salary growth for 2+ years had 3× higher attrition risk
DENSE_RANK() over tenure buckets identified that the 2–4 year career stage had the highest attrition rate across all departments
Department × gender cross-tabulation surfaced that attrition in Sales was driven by gender pay gap, not workload — a finding SQL analytics delivered without any ML
All analysis runs in a single SQL script — no ETL, no dependencies, fully reproducible by anyone with MySQL access