This project in Jupyter Notebook implements a data cleaning and exploratory analysis pipeline for the Life Expectancy dataset. The workflow is designed to transform raw health data into a model-ready format.
In addition, it also contains the Jupyter notebooks and exercises for Solving Mathematical, Physical and Chemical Problems. The notebooks cover scientific programming, numerical computing, data analysis, visualization, and scientific data processing using Python libraries such as NumPy, SciPy, Pandas, Matplotlib, and Seaborn.
Before training any ML model, raw data is almost never "ready": it may contain missing values, outliers, duplicates, and categorical variables that have not yet been encoded. This document summarizes, step by step, the pipeline followed in the report to clean and prepare the dataset.
flowchart TD
A["1. Environment Setup<br/>venv + pandas/numpy/seaborn/matplotlib"] --> B["2. Dataset Loading<br/>pd.read_csv"]
B --> C["3. Sanity Check<br/>shape, info, data types"]
C --> D["4. Missing Values<br/>isnull().sum"]
D --> E["5. Duplicates and Garbage Values<br/>duplicated, value_counts"]
E --> F["6. EDA<br/>describe, histograms, boxplots,<br/>scatterplots, correlation"]
F --> G["7. Missing Value Treatment<br/>Median + KNNImputer"]
G --> H["8. Outlier Treatment<br/>IQR Method + Capping"]
H --> I["9. Duplicates<br/>drop_duplicates"]
I --> J["10. Encoding<br/>One-Hot Encoding<br/>get_dummies"]
J --> K["Dataset is numeric,<br/>complete, and duplicate-free<br/>ready for scaling and modeling"]
| # | Stage | Objective | Key Function / Method |
|---|---|---|---|
| 1 | Environment Setup | Isolate project dependencies | python3 -m venv env |
| 2 | Dataset Loading | Load the CSV into a DataFrame | pd.read_csv() |
| 3 | Sanity Check | Check dimensions and column data types | df.shape, df.info() |
| 4 | Missing Values | Quantify the percentage of missing values per column | df.isnull().sum() |
| 5 | Duplicates / Garbage | Detect repeated rows or inconsistent labels | df.duplicated().sum(), value_counts() |
| 6 | EDA | Understand distributions, outliers, and relationships with the target | describe(), histplot, boxplot, heatmap |
| 7 | Imputation | Fill in missing values | Median (skewed variables) + KNNImputer (remaining variables) |
| 8 | Outliers | Limit extreme values without removing rows | IQR method + capping (winsorizing) |
| 9 | Duplicates (Confirmation) | Remove duplicates if any exist | drop_duplicates() |
| 10 | Encoding | Convert categorical variables into numerical values | pd.get_dummies(drop_first=True) |