/Python & Data Analytics/Phase 5

Phase 5Days 49-60

Data Cleaning, EDA & Visualization

Turn messy tables into trustworthy patterns without hiding uncertainty

Phase Goal

Profile, clean, reshape, visualize, and document real data while protecting meaning and grain.

Progress

Notes

Open the written lectures for this course before checking off the phase topics.

Projects

Public Library Programme Equity Study

Day 49: Data Profiling as Investigation

Build the first-principles mental model for Schema, grain, keys, counts, and lineage; name the question it answers and the unit of analysisPredict and trace a tiny example of Schema, grain, keys, counts, and lineage before inspecting the resultApply Schema, grain, keys, counts, and lineage to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Schema, grain, keys, counts, and lineage; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Missingness, uniqueness, ranges, and categories; name the question it answers and the unit of analysisPredict and trace a tiny example of Missingness, uniqueness, ranges, and categories before inspecting the resultApply Missingness, uniqueness, ranges, and categories to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Missingness, uniqueness, ranges, and categories; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Cross-field consistency; name the question it answers and the unit of analysisPredict and trace a tiny example of Cross-field consistency before inspecting the resultApply Cross-field consistency to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Cross-field consistency; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Risk-prioritized profiling report; name the question it answers and the unit of analysisPredict and trace a tiny example of Risk-prioritized profiling report before inspecting the resultApply Risk-prioritized profiling report to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Risk-prioritized profiling report; diagnose the visible failure and perform the corrective check

Retrieve one recent mechanism and one older prerequisite before consulting notesComplete a guided example, then reconstruct its critical path from a blank artifactSolve a changed task that requires choosing among already-taught toolsRecord the decision changed, failure tested, help used, and adaptive 1/3/7/14/30-day review cue

Day 50: Missing Data Mechanisms

Build the first-principles mental model for Absent, unknown, not-applicable, and suppressed values; name the question it answers and the unit of analysisPredict and trace a tiny example of Absent, unknown, not-applicable, and suppressed values before inspecting the resultApply Absent, unknown, not-applicable, and suppressed values to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Absent, unknown, not-applicable, and suppressed values; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Missingness patterns; name the question it answers and the unit of analysisPredict and trace a tiny example of Missingness patterns before inspecting the resultApply Missingness patterns to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Missingness patterns; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Deletion, explicit category, fill, and sensitivity; name the question it answers and the unit of analysisPredict and trace a tiny example of Deletion, explicit category, fill, and sensitivity before inspecting the resultApply Deletion, explicit category, fill, and sensitivity to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Deletion, explicit category, fill, and sensitivity; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Blanket fill and invented certainty; name the question it answers and the unit of analysisPredict and trace a tiny example of Blanket fill and invented certainty before inspecting the resultApply Blanket fill and invented certainty to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Blanket fill and invented certainty; diagnose the visible failure and perform the corrective check

Day 51: Duplicates & Entity Resolution

Build the first-principles mental model for Row duplicates versus entity duplicates; name the question it answers and the unit of analysisPredict and trace a tiny example of Row duplicates versus entity duplicates before inspecting the resultApply Row duplicates versus entity duplicates to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Row duplicates versus entity duplicates; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Candidate keys and survivorship; name the question it answers and the unit of analysisPredict and trace a tiny example of Candidate keys and survivorship before inspecting the resultApply Candidate keys and survivorship to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Candidate keys and survivorship; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Normalized match fields; name the question it answers and the unit of analysisPredict and trace a tiny example of Normalized match fields before inspecting the resultApply Normalized match fields to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Normalized match fields; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for False merges, false splits, and audits; name the question it answers and the unit of analysisPredict and trace a tiny example of False merges, false splits, and audits before inspecting the resultApply False merges, false splits, and audits to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind False merges, false splits, and audits; diagnose the visible failure and perform the corrective check

Day 52: Outliers & Valid Extremes

Build the first-principles mental model for Range rules and domain limits; name the question it answers and the unit of analysisPredict and trace a tiny example of Range rules and domain limits before inspecting the resultApply Range rules and domain limits to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Range rules and domain limits; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for IQR and robust segment checks; name the question it answers and the unit of analysisPredict and trace a tiny example of IQR and robust segment checks before inspecting the resultApply IQR and robust segment checks to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind IQR and robust segment checks; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Correct, cap, exclude, or retain; name the question it answers and the unit of analysisPredict and trace a tiny example of Correct, cap, exclude, or retain before inspecting the resultApply Correct, cap, exclude, or retain to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Correct, cap, exclude, or retain; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Deleting inconvenient truth; name the question it answers and the unit of analysisPredict and trace a tiny example of Deleting inconvenient truth before inspecting the resultApply Deleting inconvenient truth to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Deleting inconvenient truth; diagnose the visible failure and perform the corrective check

Day 53: Tidy Data & Reshaping

Build the first-principles mental model for Observation, variable, and value; name the question it answers and the unit of analysisPredict and trace a tiny example of Observation, variable, and value before inspecting the resultApply Observation, variable, and value to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Observation, variable, and value; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for melt from wide to long; name the question it answers and the unit of analysisPredict and trace a tiny example of melt from wide to long before inspecting the resultApply melt from wide to long to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind melt from wide to long; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for pivot and pivot_table; name the question it answers and the unit of analysisPredict and trace a tiny example of pivot and pivot_table before inspecting the resultApply pivot and pivot_table to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind pivot and pivot_table; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Duplicate keys and lost identifiers; name the question it answers and the unit of analysisPredict and trace a tiny example of Duplicate keys and lost identifiers before inspecting the resultApply Duplicate keys and lost identifiers to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Duplicate keys and lost identifiers; diagnose the visible failure and perform the corrective check

Day 54: Categorical & Text Cleaning

Build the first-principles mental model for Whitespace, case, Unicode, and mappings; name the question it answers and the unit of analysisPredict and trace a tiny example of Whitespace, case, Unicode, and mappings before inspecting the resultApply Whitespace, case, Unicode, and mappings to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Whitespace, case, Unicode, and mappings; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Ordered categories and unknown buckets; name the question it answers and the unit of analysisPredict and trace a tiny example of Ordered categories and unknown buckets before inspecting the resultApply Ordered categories and unknown buckets to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Ordered categories and unknown buckets; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for String extraction; name the question it answers and the unit of analysisPredict and trace a tiny example of String extraction before inspecting the resultApply String extraction to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind String extraction; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Collapsed meanings and reversibility; name the question it answers and the unit of analysisPredict and trace a tiny example of Collapsed meanings and reversibility before inspecting the resultApply Collapsed meanings and reversibility to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Collapsed meanings and reversibility; diagnose the visible failure and perform the corrective check

Day 55: Dates, Periods & Time-Aware EDA

Build the first-principles mental model for Datetime parsing and ambiguity; name the question it answers and the unit of analysisPredict and trace a tiny example of Datetime parsing and ambiguity before inspecting the resultApply Datetime parsing and ambiguity to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Datetime parsing and ambiguity; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Components, periods, and durations; name the question it answers and the unit of analysisPredict and trace a tiny example of Components, periods, and durations before inspecting the resultApply Components, periods, and durations to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Components, periods, and durations; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Resample versus calendar grouping; name the question it answers and the unit of analysisPredict and trace a tiny example of Resample versus calendar grouping before inspecting the resultApply Resample versus calendar grouping to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Resample versus calendar grouping; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Incomplete periods and timezone mistakes; name the question it answers and the unit of analysisPredict and trace a tiny example of Incomplete periods and timezone mistakes before inspecting the resultApply Incomplete periods and timezone mistakes to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Incomplete periods and timezone mistakes; diagnose the visible failure and perform the corrective check

Day 56: Distribution Charts

Build the first-principles mental model for Histogram bins and density; name the question it answers and the unit of analysisPredict and trace a tiny example of Histogram bins and density before inspecting the resultApply Histogram bins and density to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Histogram bins and density; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Box, violin, and ECDF trade-offs; name the question it answers and the unit of analysisPredict and trace a tiny example of Box, violin, and ECDF trade-offs before inspecting the resultApply Box, violin, and ECDF trade-offs to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Box, violin, and ECDF trade-offs; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Segment comparison on common scales; name the question it answers and the unit of analysisPredict and trace a tiny example of Segment comparison on common scales before inspecting the resultApply Segment comparison on common scales to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Segment comparison on common scales; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Small groups and hidden multimodality; name the question it answers and the unit of analysisPredict and trace a tiny example of Small groups and hidden multimodality before inspecting the resultApply Small groups and hidden multimodality to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Small groups and hidden multimodality; diagnose the visible failure and perform the corrective check

Day 57: Comparison & Relationship Charts

Build the first-principles mental model for Ordered bars, dots, and references; name the question it answers and the unit of analysisPredict and trace a tiny example of Ordered bars, dots, and references before inspecting the resultApply Ordered bars, dots, and references to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Ordered bars, dots, and references; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Scatterplots, transparency, and facets; name the question it answers and the unit of analysisPredict and trace a tiny example of Scatterplots, transparency, and facets before inspecting the resultApply Scatterplots, transparency, and facets to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Scatterplots, transparency, and facets; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Correlation versus cause; name the question it answers and the unit of analysisPredict and trace a tiny example of Correlation versus cause before inspecting the resultApply Correlation versus cause to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Correlation versus cause; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Dual axes, overplotting, and Simpson’s paradox; name the question it answers and the unit of analysisPredict and trace a tiny example of Dual axes, overplotting, and Simpson’s paradox before inspecting the resultApply Dual axes, overplotting, and Simpson’s paradox to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Dual axes, overplotting, and Simpson’s paradox; diagnose the visible failure and perform the corrective check

Day 58: Seaborn & Matplotlib Together

Build the first-principles mental model for Pandas → Seaborn → Matplotlib stack; name the question it answers and the unit of analysisPredict and trace a tiny example of Pandas → Seaborn → Matplotlib stack before inspecting the resultApply Pandas → Seaborn → Matplotlib stack to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Pandas → Seaborn → Matplotlib stack; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Figures, axes, themes, labels, and annotation; name the question it answers and the unit of analysisPredict and trace a tiny example of Figures, axes, themes, labels, and annotation before inspecting the resultApply Figures, axes, themes, labels, and annotation to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Figures, axes, themes, labels, and annotation; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Small multiples; name the question it answers and the unit of analysisPredict and trace a tiny example of Small multiples before inspecting the resultApply Small multiples to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Small multiples; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Color accessibility and legend overload; name the question it answers and the unit of analysisPredict and trace a tiny example of Color accessibility and legend overload before inspecting the resultApply Color accessibility and legend overload to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Color accessibility and legend overload; diagnose the visible failure and perform the corrective check

Day 59: EDA Narrative & Reproducibility

Build the first-principles mental model for Question and hypothesis logs; name the question it answers and the unit of analysisPredict and trace a tiny example of Question and hypothesis logs before inspecting the resultApply Question and hypothesis logs to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Question and hypothesis logs; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Facts, interpretation, and recommendation; name the question it answers and the unit of analysisPredict and trace a tiny example of Facts, interpretation, and recommendation before inspecting the resultApply Facts, interpretation, and recommendation to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Facts, interpretation, and recommendation; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Restart-run-all and reusable code; name the question it answers and the unit of analysisPredict and trace a tiny example of Restart-run-all and reusable code before inspecting the resultApply Restart-run-all and reusable code to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Restart-run-all and reusable code; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Cherry-picking and causal overreach; name the question it answers and the unit of analysisPredict and trace a tiny example of Cherry-picking and causal overreach before inspecting the resultApply Cherry-picking and causal overreach to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Cherry-picking and causal overreach; diagnose the visible failure and perform the corrective check

Day 60: Phase Project — Library Equity Study

Build the first-principles mental model for Programme, branch, capacity, and neighbourhood grains; name the question it answers and the unit of analysisPredict and trace a tiny example of Programme, branch, capacity, and neighbourhood grains before inspecting the resultApply Programme, branch, capacity, and neighbourhood grains to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Programme, branch, capacity, and neighbourhood grains; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Cleaning log with before/after evidence; name the question it answers and the unit of analysisPredict and trace a tiny example of Cleaning log with before/after evidence before inspecting the resultApply Cleaning log with before/after evidence to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Cleaning log with before/after evidence; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for EDA from overview to segment and anomaly; name the question it answers and the unit of analysisPredict and trace a tiny example of EDA from overview to segment and anomaly before inspecting the resultApply EDA from overview to segment and anomaly to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind EDA from overview to segment and anomaly; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Funding recommendation and alternative explanation; name the question it answers and the unit of analysisPredict and trace a tiny example of Funding recommendation and alternative explanation before inspecting the resultApply Funding recommendation and alternative explanation to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Funding recommendation and alternative explanation; diagnose the visible failure and perform the corrective check

Deliver Public Library Programme Equity Study for a library programming director; support the decision: where should evening-programme funding be allocated next quarter?Reconstruct the critical workflow without the example open and disclose help usedFind and repair a planted data, logic, or communication failure with evidenceRespond to a changed requirement without being told which technique to useState limitations, non-goals, uncertainty, and what evidence would change the recommendation

Capstone

Public Library Programme Equity Study

Build for a library programming director. The result must support this decision: where should evening-programme funding be allocated next quarter? Include reproducible inputs, validation evidence, one tested failure, a changed requirement, and an explicit non-goal.

Reproducible source-to-output workflow
Documented quality checks and assumptions
Decision-ready output for a named user
Independent reconstruction and transfer evidence

Phase Complete!

After this phase, you'll be able to:

Data Profiling as Investigation
Missing Data Mechanisms
Duplicates & Entity Resolution
Outliers & Valid Extremes
Tidy Data & Reshaping
Categorical & Text Cleaning
Dates, Periods & Time-Aware EDA
Distribution Charts

Independently deliver Public Library Programme Equity Study, diagnose a planted failure, explain the decision logic, and transfer the workflow to a changed requirement.

Back to all phases

Phase Goal

Day 49: Data Profiling as Investigation

Schema, grain, keys, counts, and lineage

Missingness, uniqueness, ranges, and categories

Cross-field consistency

Risk-prioritized profiling report

Practice, retrieval, and analyst judgment

Day 50: Missing Data Mechanisms

Absent, unknown, not-applicable, and suppressed values

Missingness patterns

Deletion, explicit category, fill, and sensitivity

Blanket fill and invented certainty

Practice, retrieval, and analyst judgment

Day 51: Duplicates & Entity Resolution

Row duplicates versus entity duplicates

Candidate keys and survivorship

Normalized match fields

False merges, false splits, and audits

Practice, retrieval, and analyst judgment

Day 52: Outliers & Valid Extremes

Range rules and domain limits

IQR and robust segment checks

Correct, cap, exclude, or retain

Deleting inconvenient truth

Practice, retrieval, and analyst judgment

Day 53: Tidy Data & Reshaping

Observation, variable, and value

melt from wide to long

pivot and pivot_table

Duplicate keys and lost identifiers

Practice, retrieval, and analyst judgment

Day 54: Categorical & Text Cleaning

Whitespace, case, Unicode, and mappings

Ordered categories and unknown buckets

String extraction

Collapsed meanings and reversibility

Practice, retrieval, and analyst judgment

Day 55: Dates, Periods & Time-Aware EDA

Datetime parsing and ambiguity

Components, periods, and durations

Resample versus calendar grouping

Incomplete periods and timezone mistakes

Practice, retrieval, and analyst judgment

Day 56: Distribution Charts

Histogram bins and density

Box, violin, and ECDF trade-offs

Segment comparison on common scales

Small groups and hidden multimodality

Practice, retrieval, and analyst judgment

Day 57: Comparison & Relationship Charts

Ordered bars, dots, and references

Scatterplots, transparency, and facets

Correlation versus cause

Dual axes, overplotting, and Simpson’s paradox

Practice, retrieval, and analyst judgment

Day 58: Seaborn & Matplotlib Together

Pandas → Seaborn → Matplotlib stack

Figures, axes, themes, labels, and annotation

Small multiples

Color accessibility and legend overload

Practice, retrieval, and analyst judgment

Day 59: EDA Narrative & Reproducibility

Question and hypothesis logs

Facts, interpretation, and recommendation

Restart-run-all and reusable code

Cherry-picking and causal overreach

Practice, retrieval, and analyst judgment

Day 60: Phase Project — Library Equity Study

Programme, branch, capacity, and neighbourhood grains

Cleaning log with before/after evidence

EDA from overview to segment and anomaly

Funding recommendation and alternative explanation

Phase gate: independent delivery and transfer

Public Library Programme Equity Study

Phase Complete!