Kharanshu Papolli · LinkedIn · Portfolio
I took the public Instacart Market Basket dataset (3.42M orders, 206K users, 33.8M line items) and worked through it the way I'd approach a real product analytics problem — data quality first, then user segmentation, then a specific A/B test proposal that comes out of the findings rather than being tacked on.
The short version of what I found: Instacart's retention gap isn't at acquisition, it's cadence. A weekly shopper places 27 median lifetime orders. A monthly shopper places 6. That 4.5x difference is the biggest single lever in the data.
If you want the full narrative, read docs/executive_memo.md — it's written as an internal memo, not a tutorial. If you want the A/B test writeup, that's in docs/ab_test_case_study.md. If you want to verify the numbers, every claim in both documents is backed by a specific line in one of the analysis scripts.
The scripts run in order and each one writes parquet outputs the next one reads. Nine scripts total, about three minutes end-to-end on a laptop.
00_data_quality_audit.py runs before anything else. Nine checks — row counts, nulls, duplicate keys, referential integrity, value ranges, test-set leakage, sequence gaps, and logical consistency of the reordered flag. The dataset passes all of them, but there are five Kaggle-imposed quirks that shape how the findings should be read (see below).
01_load_and_explore.py loads the raw CSVs with optimized dtypes, builds a cumulative day offset per user (since Instacart has no absolute timestamps), and saves parquet files. Loading 33M line items into 2GB of RAM took some care with dtypes and dict-based joins instead of merges.
02_user_funnel.py builds the engagement funnel. Because Kaggle excluded users with fewer than 4 orders, the funnel starts at "reached 1+ orders" instead of "signup." From there: 88% reach 5+ orders, 54% reach 10+, 26% reach 20+, 6% reach 50+.
03_cohort_retention.py computes weekly retention curves and splits users into cohorts by first-basket size. The retention curve is bimodal (weekly vs biweekly shoppers create oscillation around 40–55%). First-basket size doesn't predict retention — spread across cohorts is 0.9 percentage points, which is inside the noise. That's a killed hypothesis, worth documenting.
04_reorder_analysis.py goes deep on reorder patterns. Overall reorder rate is 59% of line items. Dairy eggs leads at 67%, beverages and produce at 65%. Habit escalates monotonically with tenure: 22% at orders 1-3, 83% at order 51+. On a 15K-user sample, consecutive baskets share a median of 24% of items — the gap between "will ever reorder" (59%) and "reordered last time" (24%) is what motivates the Smart Reorder feature.
05_operations.py covers hourly, day-of-week, and department mix patterns. Sunday and Monday together carry 34.7% of weekly volume. Peak hour is 10 AM. Basket size is stable at around 10 items across all hours.
06_ab_test_smart_reorder.py designs and simulates an A/B test for the Smart Reorder feature. Real baseline from the Engaged tier (2.474 orders/30d, std 1.366), power analysis for 1,914 users per arm at 5% MDE, primary metric plus two guardrails, ship/no-ship decision framework. Under a plausible +7% true lift, the simulation gives +6.2% primary metric with guardrails intact.
07_product_and_basket_analysis.py does three things. First, splits products into a Volume × Reorder Rate 2×2 (Hero, Workhorse, Sticky Niche, Long Tail) — 1,505 Hero products drive 43% of total volume. Second, tracks basket size across order numbers and finds it stabilizes at order #4 — the "usual basket" concept forms fast. Third, compares first-order behavior of users who stalled at 4-5 orders vs users who reached 50+; Power users' first orders skew +3.4pp produce and +2.5pp dairy.
08_signals_and_personas.py is the one that ties everything together. Defines three named signals per user (Habit Strength, Cadence Rhythm, Basket Convergence), buckets users into four cadence personas, and produces the 4.5x retention differential that drives the memo. Also tests the basket-convergence hypothesis and shows it doesn't hold.
Two findings stood out because they contradicted what I expected going in.
Basket convergence doesn't predict retention. I expected users whose baskets narrowed over time to be the habitual ones and to retain longer. That's a common growth hypothesis. When I ran it, convergence quartiles all showed ~14 median lifetime orders. Once you control for cadence, convergence explains nothing. Cadence explains everything.
First-basket size doesn't predict retention either. Users with 1–3 items in their first order retain almost identically to users with 16+ items. Spread of 0.9pp across four cohorts. The intuition that "getting a big first basket locks them in" isn't supported by this data.
Both of these are more useful findings than confirmations would have been — they save a product team from spending time on levers that don't move.
Comes from the cadence finding. Target Biweekly Shoppers (63K users, 14 median lifetime orders) with a Cadence Nudge — a push notification triggered when their days-since-last-order exceeds their personal median by 3 or more days, with the Smart Reorder tile pre-loaded. Full writeup in docs/ab_test_case_study.md.
instacart_project/
├── README.md
├── requirements.txt
├── .gitignore
├── analysis/
│ ├── 00_data_quality_audit.py run this first
│ ├── 01_load_and_explore.py
│ ├── 02_user_funnel.py
│ ├── 03_cohort_retention.py
│ ├── 04_reorder_analysis.py
│ ├── 05_operations.py
│ ├── 06_ab_test_smart_reorder.py
│ ├── 07_product_and_basket_analysis.py
│ └── 08_signals_and_personas.py
├── dashboard/
│ ├── build_static_preview.py
│ └── dashboard_preview.png
└── docs/
├── executive_memo.md
└── ab_test_case_study.md
Get the data from Kaggle (https://www.kaggle.com/c/instacart-market-basket-analysis/data), drop the CSVs into data/raw/, then:
pip install -r requirements.txt
python analysis/00_data_quality_audit.py
python analysis/01_load_and_explore.py
python analysis/02_user_funnel.py
python analysis/03_cohort_retention.py
python analysis/04_reorder_analysis.py
python analysis/05_operations.py
python analysis/06_ab_test_smart_reorder.py
python analysis/07_product_and_basket_analysis.py
python analysis/08_signals_and_personas.py
python dashboard/build_static_preview.pyThe quality audit turned up five things that affect how you should read any of this. They're Kaggle's decisions, not bugs — but they bound the findings.
Users with fewer than 4 lifetime orders were excluded before the data was released, so I can't observe true one-and-done churn. The funnel starts at "1+ orders" for this reason.
days_since_prior_order is capped at 30. Any actual gap of 30+ days shows as exactly 30.0, which probably compresses the median inter-order estimate slightly.
There are no absolute timestamps, only relative day offsets per user. That means retention is "weeks since a user's first order" rather than calendar weeks. Can't correlate with campaigns or seasons.
No prices, no delivery times. The GMV projection uses ₹400 as a placeholder AOV. Order counts and reorder rates are real; monetary values are estimates.
order_number caps at 100. About 1,374 users are at the cap, so their true lifetime count could be higher.
Everything in this repo points to a specific line of code. If you ask "how did you get 59% reorder rate?" the answer is items['reordered'].mean() on line 60 of the loader. The 4.5x cadence differential is the median comparison in 08_signals_and_personas.py.
The A/B test is explicitly a design plus simulation — not a live experiment. Real baseline mean and std from the data, plausible true lift, Welch's t-test on the simulated arms. That's the sanity check you'd do before actually running the test at Blinkit or Zomato: "if we launch this, will we have the power to detect the effect we care about?"