/Python & Data Analytics/Phase 4

Phase 4Days 37-48

NumPy & Pandas Foundations

Move from Python records to vectorized, auditable tables

Phase Goal

Use arrays and DataFrames with deliberate shapes, dtypes, indexes, selections, transformations, and grouped summaries.

Progress

Notes

Open the written lectures for this course before checking off the phase topics.

Projects

Coastal Water-Sampling Review

Day 37: NumPy Arrays, Shape & Dtype

Build the first-principles mental model for Array values, axes, shape, and dtype; name the question it answers and the unit of analysisPredict and trace a tiny example of Array values, axes, shape, and dtype before inspecting the resultApply Array values, axes, shape, and dtype to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Array values, axes, shape, and dtype; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Creation, inspection, and explicit types; name the question it answers and the unit of analysisPredict and trace a tiny example of Creation, inspection, and explicit types before inspecting the resultApply Creation, inspection, and explicit types to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Creation, inspection, and explicit types; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Vectorized arithmetic and units; name the question it answers and the unit of analysisPredict and trace a tiny example of Vectorized arithmetic and units before inspecting the resultApply Vectorized arithmetic and units to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Vectorized arithmetic and units; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Mismatch, overflow, truncation, and coercion; name the question it answers and the unit of analysisPredict and trace a tiny example of Mismatch, overflow, truncation, and coercion before inspecting the resultApply Mismatch, overflow, truncation, and coercion to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Mismatch, overflow, truncation, and coercion; diagnose the visible failure and perform the corrective check

Retrieve one recent mechanism and one older prerequisite before consulting notesComplete a guided example, then reconstruct its critical path from a blank artifactSolve a changed task that requires choosing among already-taught toolsRecord the decision changed, failure tested, help used, and adaptive 1/3/7/14/30-day review cue

Day 38: Indexing, Slicing & Masks

Build the first-principles mental model for One- and multi-dimensional indexing; name the question it answers and the unit of analysisPredict and trace a tiny example of One- and multi-dimensional indexing before inspecting the resultApply One- and multi-dimensional indexing to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind One- and multi-dimensional indexing; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Slices as views and copy evidence; name the question it answers and the unit of analysisPredict and trace a tiny example of Slices as views and copy evidence before inspecting the resultApply Slices as views and copy evidence to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Slices as views and copy evidence; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Boolean masks and conditions; name the question it answers and the unit of analysisPredict and trace a tiny example of Boolean masks and conditions before inspecting the resultApply Boolean masks and conditions to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Boolean masks and conditions; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Broadcasting and chained-selection mistakes; name the question it answers and the unit of analysisPredict and trace a tiny example of Broadcasting and chained-selection mistakes before inspecting the resultApply Broadcasting and chained-selection mistakes to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Broadcasting and chained-selection mistakes; diagnose the visible failure and perform the corrective check

Day 39: Broadcasting & Vectorization

Build the first-principles mental model for Elementwise operations versus loops; name the question it answers and the unit of analysisPredict and trace a tiny example of Elementwise operations versus loops before inspecting the resultApply Elementwise operations versus loops to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Elementwise operations versus loops; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Trailing-dimension compatibility; name the question it answers and the unit of analysisPredict and trace a tiny example of Trailing-dimension compatibility before inspecting the resultApply Trailing-dimension compatibility to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Trailing-dimension compatibility; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for where, clip, and conditional transformation; name the question it answers and the unit of analysisPredict and trace a tiny example of where, clip, and conditional transformation before inspecting the resultApply where, clip, and conditional transformation to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind where, clip, and conditional transformation; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Memory cost and unreadable vector tricks; name the question it answers and the unit of analysisPredict and trace a tiny example of Memory cost and unreadable vector tricks before inspecting the resultApply Memory cost and unreadable vector tricks to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Memory cost and unreadable vector tricks; diagnose the visible failure and perform the corrective check

Day 40: Aggregations & Numerical Checks

Build the first-principles mental model for Axis-aware aggregations and percentiles; name the question it answers and the unit of analysisPredict and trace a tiny example of Axis-aware aggregations and percentiles before inspecting the resultApply Axis-aware aggregations and percentiles to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Axis-aware aggregations and percentiles; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for NaN propagation and nan-aware operations; name the question it answers and the unit of analysisPredict and trace a tiny example of NaN propagation and nan-aware operations before inspecting the resultApply NaN propagation and nan-aware operations to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind NaN propagation and nan-aware operations; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for isclose and tolerances; name the question it answers and the unit of analysisPredict and trace a tiny example of isclose and tolerances before inspecting the resultApply isclose and tolerances to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind isclose and tolerances; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Empty slices and silent warnings; name the question it answers and the unit of analysisPredict and trace a tiny example of Empty slices and silent warnings before inspecting the resultApply Empty slices and silent warnings to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Empty slices and silent warnings; diagnose the visible failure and perform the corrective check

Day 41: Series, DataFrame & Index

Build the first-principles mental model for Labeled one- and two-dimensional data; name the question it answers and the unit of analysisPredict and trace a tiny example of Labeled one- and two-dimensional data before inspecting the resultApply Labeled one- and two-dimensional data to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Labeled one- and two-dimensional data; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Columns, index, shape, dtypes, and info; name the question it answers and the unit of analysisPredict and trace a tiny example of Columns, index, shape, dtypes, and info before inspecting the resultApply Columns, index, shape, dtypes, and info to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Columns, index, shape, dtypes, and info; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Frames from records and columns; name the question it answers and the unit of analysisPredict and trace a tiny example of Frames from records and columns before inspecting the resultApply Frames from records and columns to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Frames from records and columns; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Duplicate labels and mixed types; name the question it answers and the unit of analysisPredict and trace a tiny example of Duplicate labels and mixed types before inspecting the resultApply Duplicate labels and mixed types to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Duplicate labels and mixed types; diagnose the visible failure and perform the corrective check

Day 42: Reading Data & Controlling Types

Build the first-principles mental model for read_csv selection and parsing; name the question it answers and the unit of analysisPredict and trace a tiny example of read_csv selection and parsing before inspecting the resultApply read_csv selection and parsing to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind read_csv selection and parsing; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Nullable strings, integers, booleans, and categories; name the question it answers and the unit of analysisPredict and trace a tiny example of Nullable strings, integers, booleans, and categories before inspecting the resultApply Nullable strings, integers, booleans, and categories to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Nullable strings, integers, booleans, and categories; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Missing markers and NA policy; name the question it answers and the unit of analysisPredict and trace a tiny example of Missing markers and NA policy before inspecting the resultApply Missing markers and NA policy to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Missing markers and NA policy; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Inference drift and identifier corruption; name the question it answers and the unit of analysisPredict and trace a tiny example of Inference drift and identifier corruption before inspecting the resultApply Inference drift and identifier corruption to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Inference drift and identifier corruption; diagnose the visible failure and perform the corrective check

Day 43: Selecting Rows & Columns

Build the first-principles mental model for Bracket selection, loc, and iloc; name the question it answers and the unit of analysisPredict and trace a tiny example of Bracket selection, loc, and iloc before inspecting the resultApply Bracket selection, loc, and iloc to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Bracket selection, loc, and iloc; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Boolean filtering and parentheses; name the question it answers and the unit of analysisPredict and trace a tiny example of Boolean filtering and parentheses before inspecting the resultApply Boolean filtering and parentheses to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Boolean filtering and parentheses; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for query expression boundaries; name the question it answers and the unit of analysisPredict and trace a tiny example of query expression boundaries before inspecting the resultApply query expression boundaries to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind query expression boundaries; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Chained assignment diagnosis; name the question it answers and the unit of analysisPredict and trace a tiny example of Chained assignment diagnosis before inspecting the resultApply Chained assignment diagnosis to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Chained assignment diagnosis; diagnose the visible failure and perform the corrective check

Day 44: Creating & Transforming Columns

Build the first-principles mental model for Vectorized arithmetic and string accessors; name the question it answers and the unit of analysisPredict and trace a tiny example of Vectorized arithmetic and string accessors before inspecting the resultApply Vectorized arithmetic and string accessors to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Vectorized arithmetic and string accessors; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for assign, map, replace, and categories; name the question it answers and the unit of analysisPredict and trace a tiny example of assign, map, replace, and categories before inspecting the resultApply assign, map, replace, and categories to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind assign, map, replace, and categories; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for apply as fallback; name the question it answers and the unit of analysisPredict and trace a tiny example of apply as fallback before inspecting the resultApply apply as fallback to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind apply as fallback; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Index alignment and accidental NaN; name the question it answers and the unit of analysisPredict and trace a tiny example of Index alignment and accidental NaN before inspecting the resultApply Index alignment and accidental NaN to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Index alignment and accidental NaN; diagnose the visible failure and perform the corrective check

Day 45: Sorting, Ranking & Deduplication

Build the first-principles mental model for Multi-key stable sorting; name the question it answers and the unit of analysisPredict and trace a tiny example of Multi-key stable sorting before inspecting the resultApply Multi-key stable sorting to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Multi-key stable sorting; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Rank methods and ties; name the question it answers and the unit of analysisPredict and trace a tiny example of Rank methods and ties before inspecting the resultApply Rank methods and ties to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Rank methods and ties; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Duplicate detection with explicit keys; name the question it answers and the unit of analysisPredict and trace a tiny example of Duplicate detection with explicit keys before inspecting the resultApply Duplicate detection with explicit keys to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Duplicate detection with explicit keys; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Wrong survivor and legitimate repeats; name the question it answers and the unit of analysisPredict and trace a tiny example of Wrong survivor and legitimate repeats before inspecting the resultApply Wrong survivor and legitimate repeats to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Wrong survivor and legitimate repeats; diagnose the visible failure and perform the corrective check

Day 46: GroupBy from Split to Combine

Build the first-principles mental model for Grouping grain; name the question it answers and the unit of analysisPredict and trace a tiny example of Grouping grain before inspecting the resultApply Grouping grain to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Grouping grain; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Named aggregations; name the question it answers and the unit of analysisPredict and trace a tiny example of Named aggregations before inspecting the resultApply Named aggregations to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Named aggregations; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for transform for group-relative features; name the question it answers and the unit of analysisPredict and trace a tiny example of transform for group-relative features before inspecting the resultApply transform for group-relative features to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind transform for group-relative features; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Dropped NA groups and mixed grains; name the question it answers and the unit of analysisPredict and trace a tiny example of Dropped NA groups and mixed grains before inspecting the resultApply Dropped NA groups and mixed grains to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Dropped NA groups and mixed grains; diagnose the visible failure and perform the corrective check

Day 47: Combining Tables Safely

Build the first-principles mental model for concat axes; name the question it answers and the unit of analysisPredict and trace a tiny example of concat axes before inspecting the resultApply concat axes to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind concat axes; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for merge keys, join types, cardinality, and validate; name the question it answers and the unit of analysisPredict and trace a tiny example of merge keys, join types, cardinality, and validate before inspecting the resultApply merge keys, join types, cardinality, and validate to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind merge keys, join types, cardinality, and validate; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Indicators and unmatched reconciliation; name the question it answers and the unit of analysisPredict and trace a tiny example of Indicators and unmatched reconciliation before inspecting the resultApply Indicators and unmatched reconciliation to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Indicators and unmatched reconciliation; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Many-to-many explosions and collisions; name the question it answers and the unit of analysisPredict and trace a tiny example of Many-to-many explosions and collisions before inspecting the resultApply Many-to-many explosions and collisions to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Many-to-many explosions and collisions; diagnose the visible failure and perform the corrective check

Day 48: Phase Project — Water Sample Review

Build the first-principles mental model for Station, visit, measurement, and threshold grains; name the question it answers and the unit of analysisPredict and trace a tiny example of Station, visit, measurement, and threshold grains before inspecting the resultApply Station, visit, measurement, and threshold grains to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Station, visit, measurement, and threshold grains; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Typed ingestion and vectorized quality flags; name the question it answers and the unit of analysisPredict and trace a tiny example of Typed ingestion and vectorized quality flags before inspecting the resultApply Typed ingestion and vectorized quality flags to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Typed ingestion and vectorized quality flags; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Validated joins and exception summaries; name the question it answers and the unit of analysisPredict and trace a tiny example of Validated joins and exception summaries before inspecting the resultApply Validated joins and exception summaries to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Validated joins and exception summaries; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Repeat-visit recommendation and reproducibility; name the question it answers and the unit of analysisPredict and trace a tiny example of Repeat-visit recommendation and reproducibility before inspecting the resultApply Repeat-visit recommendation and reproducibility to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Repeat-visit recommendation and reproducibility; diagnose the visible failure and perform the corrective check

Deliver Coastal Water-Sampling Review for a watershed monitoring coordinator; support the decision: which sites need a repeat visit before the monthly report?Reconstruct the critical workflow without the example open and disclose help usedFind and repair a planted data, logic, or communication failure with evidenceRespond to a changed requirement without being told which technique to useState limitations, non-goals, uncertainty, and what evidence would change the recommendation

Capstone

Coastal Water-Sampling Review

Build for a watershed monitoring coordinator. The result must support this decision: which sites need a repeat visit before the monthly report? Include reproducible inputs, validation evidence, one tested failure, a changed requirement, and an explicit non-goal.

Reproducible source-to-output workflow
Documented quality checks and assumptions
Decision-ready output for a named user
Independent reconstruction and transfer evidence

Phase Complete!

After this phase, you'll be able to:

NumPy Arrays, Shape & Dtype
Indexing, Slicing & Masks
Broadcasting & Vectorization
Aggregations & Numerical Checks
Series, DataFrame & Index
Reading Data & Controlling Types
Selecting Rows & Columns
Creating & Transforming Columns

Independently deliver Coastal Water-Sampling Review, diagnose a planted failure, explain the decision logic, and transfer the workflow to a changed requirement.

Back to all phases

Phase Goal

Day 37: NumPy Arrays, Shape & Dtype

Array values, axes, shape, and dtype

Creation, inspection, and explicit types

Vectorized arithmetic and units

Mismatch, overflow, truncation, and coercion

Practice, retrieval, and analyst judgment

Day 38: Indexing, Slicing & Masks

One- and multi-dimensional indexing

Slices as views and copy evidence

Boolean masks and conditions

Broadcasting and chained-selection mistakes

Practice, retrieval, and analyst judgment

Day 39: Broadcasting & Vectorization

Elementwise operations versus loops

Trailing-dimension compatibility

where, clip, and conditional transformation

Memory cost and unreadable vector tricks

Practice, retrieval, and analyst judgment

Day 40: Aggregations & Numerical Checks

Axis-aware aggregations and percentiles

NaN propagation and nan-aware operations

isclose and tolerances

Empty slices and silent warnings

Practice, retrieval, and analyst judgment

Day 41: Series, DataFrame & Index

Labeled one- and two-dimensional data

Columns, index, shape, dtypes, and info

Frames from records and columns

Duplicate labels and mixed types

Practice, retrieval, and analyst judgment

Day 42: Reading Data & Controlling Types

read_csv selection and parsing

Nullable strings, integers, booleans, and categories

Missing markers and NA policy

Inference drift and identifier corruption

Practice, retrieval, and analyst judgment

Day 43: Selecting Rows & Columns

Bracket selection, loc, and iloc

Boolean filtering and parentheses

query expression boundaries

Chained assignment diagnosis

Practice, retrieval, and analyst judgment

Day 44: Creating & Transforming Columns

Vectorized arithmetic and string accessors

assign, map, replace, and categories

apply as fallback

Index alignment and accidental NaN

Practice, retrieval, and analyst judgment

Day 45: Sorting, Ranking & Deduplication

Multi-key stable sorting

Rank methods and ties

Duplicate detection with explicit keys

Wrong survivor and legitimate repeats

Practice, retrieval, and analyst judgment

Day 46: GroupBy from Split to Combine

Grouping grain

Named aggregations

transform for group-relative features

Dropped NA groups and mixed grains

Practice, retrieval, and analyst judgment

Day 47: Combining Tables Safely

concat axes

merge keys, join types, cardinality, and validate

Indicators and unmatched reconciliation

Many-to-many explosions and collisions

Practice, retrieval, and analyst judgment

Day 48: Phase Project — Water Sample Review

Station, visit, measurement, and threshold grains

Typed ingestion and vectorized quality flags

Validated joins and exception summaries

Repeat-visit recommendation and reproducibility

Phase gate: independent delivery and transfer

Coastal Water-Sampling Review

Phase Complete!