/Python & Data Analytics/Phase 10

Phase 10Days 109-120

Inference & Experiment Design

Estimate effects without turning thresholds into truth machines

Phase Goal

Use intervals, tests, power, and A/B design with assumptions, practical significance, and guardrails.

Progress

Notes

Open the written lectures for this course before checking off the phase topics.

Projects

Museum Wayfinding Experiment

Day 109: Estimation & Confidence Intervals

Build the first-principles mental model for Point and interval estimates; name the question it answers and the unit of analysisPredict and trace a tiny example of Point and interval estimates before inspecting the resultApply Point and interval estimates to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Point and interval estimates; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Repeated-sampling interpretation; name the question it answers and the unit of analysisPredict and trace a tiny example of Repeated-sampling interpretation before inspecting the resultApply Repeated-sampling interpretation to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Repeated-sampling interpretation; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Intervals for means and proportions; name the question it answers and the unit of analysisPredict and trace a tiny example of Intervals for means and proportions before inspecting the resultApply Intervals for means and proportions to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Intervals for means and proportions; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Confidence level and false probability claims; name the question it answers and the unit of analysisPredict and trace a tiny example of Confidence level and false probability claims before inspecting the resultApply Confidence level and false probability claims to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Confidence level and false probability claims; diagnose the visible failure and perform the corrective check

Retrieve one recent mechanism and one older prerequisite before consulting notesComplete a guided example, then reconstruct its critical path from a blank artifactSolve a changed task that requires choosing among already-taught toolsRecord the decision changed, failure tested, help used, and adaptive 1/3/7/14/30-day review cue

Day 110: Hypotheses & Test Logic

Build the first-principles mental model for Null, alternative, statistic, and reference distribution; name the question it answers and the unit of analysisPredict and trace a tiny example of Null, alternative, statistic, and reference distribution before inspecting the resultApply Null, alternative, statistic, and reference distribution to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Null, alternative, statistic, and reference distribution; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Conditional p-value meaning; name the question it answers and the unit of analysisPredict and trace a tiny example of Conditional p-value meaning before inspecting the resultApply Conditional p-value meaning to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Conditional p-value meaning; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for One- versus two-sided plans; name the question it answers and the unit of analysisPredict and trace a tiny example of One- versus two-sided plans before inspecting the resultApply One- versus two-sided plans to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind One- versus two-sided plans; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Accepting null and p-as-effect errors; name the question it answers and the unit of analysisPredict and trace a tiny example of Accepting null and p-as-effect errors before inspecting the resultApply Accepting null and p-as-effect errors to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Accepting null and p-as-effect errors; diagnose the visible failure and perform the corrective check

Day 111: Tests for Means

Build the first-principles mental model for One-sample, paired, and independent designs; name the question it answers and the unit of analysisPredict and trace a tiny example of One-sample, paired, and independent designs before inspecting the resultApply One-sample, paired, and independent designs to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind One-sample, paired, and independent designs; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for t statistic and degrees of freedom; name the question it answers and the unit of analysisPredict and trace a tiny example of t statistic and degrees of freedom before inspecting the resultApply t statistic and degrees of freedom to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind t statistic and degrees of freedom; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Welch comparison; name the question it answers and the unit of analysisPredict and trace a tiny example of Welch comparison before inspecting the resultApply Welch comparison to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Welch comparison; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Pairing and dependence failures; name the question it answers and the unit of analysisPredict and trace a tiny example of Pairing and dependence failures before inspecting the resultApply Pairing and dependence failures to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Pairing and dependence failures; diagnose the visible failure and perform the corrective check

Day 112: Tests for Proportions & Categories

Build the first-principles mental model for Proportion comparisons; name the question it answers and the unit of analysisPredict and trace a tiny example of Proportion comparisons before inspecting the resultApply Proportion comparisons to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Proportion comparisons; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Chi-square tests; name the question it answers and the unit of analysisPredict and trace a tiny example of Chi-square tests before inspecting the resultApply Chi-square tests to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Chi-square tests; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Expected counts; name the question it answers and the unit of analysisPredict and trace a tiny example of Expected counts before inspecting the resultApply Expected counts to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Expected counts; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Absolute versus relative difference; name the question it answers and the unit of analysisPredict and trace a tiny example of Absolute versus relative difference before inspecting the resultApply Absolute versus relative difference to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Absolute versus relative difference; diagnose the visible failure and perform the corrective check

Day 113: Nonparametric & Resampling

Build the first-principles mental model for Rank-based questions; name the question it answers and the unit of analysisPredict and trace a tiny example of Rank-based questions before inspecting the resultApply Rank-based questions to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Rank-based questions; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Permutation tests; name the question it answers and the unit of analysisPredict and trace a tiny example of Permutation tests before inspecting the resultApply Permutation tests to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Permutation tests; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Bootstrap intervals; name the question it answers and the unit of analysisPredict and trace a tiny example of Bootstrap intervals before inspecting the resultApply Bootstrap intervals to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Bootstrap intervals; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Wrong resampling units; name the question it answers and the unit of analysisPredict and trace a tiny example of Wrong resampling units before inspecting the resultApply Wrong resampling units to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Wrong resampling units; diagnose the visible failure and perform the corrective check

Day 114: Effect Size & Practical Significance

Build the first-principles mental model for Mean and standardized difference; name the question it answers and the unit of analysisPredict and trace a tiny example of Mean and standardized difference before inspecting the resultApply Mean and standardized difference to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Mean and standardized difference; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Risk difference and lift; name the question it answers and the unit of analysisPredict and trace a tiny example of Risk difference and lift before inspecting the resultApply Risk difference and lift to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Risk difference and lift; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Minimum worthwhile effect; name the question it answers and the unit of analysisPredict and trace a tiny example of Minimum worthwhile effect before inspecting the resultApply Minimum worthwhile effect to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Minimum worthwhile effect; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Detectable but trivial results; name the question it answers and the unit of analysisPredict and trace a tiny example of Detectable but trivial results before inspecting the resultApply Detectable but trivial results to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Detectable but trivial results; diagnose the visible failure and perform the corrective check

Day 115: Errors, Power & Sample Size

Build the first-principles mental model for Type I and II errors; name the question it answers and the unit of analysisPredict and trace a tiny example of Type I and II errors before inspecting the resultApply Type I and II errors to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Type I and II errors; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Power inputs; name the question it answers and the unit of analysisPredict and trace a tiny example of Power inputs before inspecting the resultApply Power inputs to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Power inputs; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Minimum detectable effect; name the question it answers and the unit of analysisPredict and trace a tiny example of Minimum detectable effect before inspecting the resultApply Minimum detectable effect to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Minimum detectable effect; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Observed-power misuse; name the question it answers and the unit of analysisPredict and trace a tiny example of Observed-power misuse before inspecting the resultApply Observed-power misuse to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Observed-power misuse; diagnose the visible failure and perform the corrective check

Day 116: Multiple Comparisons & Plans

Build the first-principles mental model for Family-wise error; name the question it answers and the unit of analysisPredict and trace a tiny example of Family-wise error before inspecting the resultApply Family-wise error to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Family-wise error; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Bonferroni and FDR intuition; name the question it answers and the unit of analysisPredict and trace a tiny example of Bonferroni and FDR intuition before inspecting the resultApply Bonferroni and FDR intuition to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Bonferroni and FDR intuition; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Primary and exploratory outcomes; name the question it answers and the unit of analysisPredict and trace a tiny example of Primary and exploratory outcomes before inspecting the resultApply Primary and exploratory outcomes to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Primary and exploratory outcomes; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Outcome switching and subgroup fishing; name the question it answers and the unit of analysisPredict and trace a tiny example of Outcome switching and subgroup fishing before inspecting the resultApply Outcome switching and subgroup fishing to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Outcome switching and subgroup fishing; diagnose the visible failure and perform the corrective check

Day 117: Experiment Units & Randomization

Build the first-principles mental model for Treatment, control, experimental unit, and outcome unit; name the question it answers and the unit of analysisPredict and trace a tiny example of Treatment, control, experimental unit, and outcome unit before inspecting the resultApply Treatment, control, experimental unit, and outcome unit to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Treatment, control, experimental unit, and outcome unit; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Assignment versus sampling; name the question it answers and the unit of analysisPredict and trace a tiny example of Assignment versus sampling before inspecting the resultApply Assignment versus sampling to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Assignment versus sampling; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Blocking and cluster randomization; name the question it answers and the unit of analysisPredict and trace a tiny example of Blocking and cluster randomization before inspecting the resultApply Blocking and cluster randomization to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Blocking and cluster randomization; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Interference and contamination; name the question it answers and the unit of analysisPredict and trace a tiny example of Interference and contamination before inspecting the resultApply Interference and contamination to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Interference and contamination; diagnose the visible failure and perform the corrective check

Day 118: A/B Metrics & Guardrails

Build the first-principles mental model for Primary behavior metric; name the question it answers and the unit of analysisPredict and trace a tiny example of Primary behavior metric before inspecting the resultApply Primary behavior metric to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Primary behavior metric; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Exposure and eligibility; name the question it answers and the unit of analysisPredict and trace a tiny example of Exposure and eligibility before inspecting the resultApply Exposure and eligibility to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Exposure and eligibility; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Guardrail harm metrics; name the question it answers and the unit of analysisPredict and trace a tiny example of Guardrail harm metrics before inspecting the resultApply Guardrail harm metrics to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Guardrail harm metrics; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Novelty, ratios, and instrumentation; name the question it answers and the unit of analysisPredict and trace a tiny example of Novelty, ratios, and instrumentation before inspecting the resultApply Novelty, ratios, and instrumentation to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Novelty, ratios, and instrumentation; diagnose the visible failure and perform the corrective check

Day 119: Experiment Analysis & Decision Memo

Build the first-principles mental model for Balance and quality checks; name the question it answers and the unit of analysisPredict and trace a tiny example of Balance and quality checks before inspecting the resultApply Balance and quality checks to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Balance and quality checks; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Intent-to-treat effect; name the question it answers and the unit of analysisPredict and trace a tiny example of Intent-to-treat effect before inspecting the resultApply Intent-to-treat effect to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Intent-to-treat effect; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Decision thresholds; name the question it answers and the unit of analysisPredict and trace a tiny example of Decision thresholds before inspecting the resultApply Decision thresholds to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Decision thresholds; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Peeking, exclusions, and inconclusive outcomes; name the question it answers and the unit of analysisPredict and trace a tiny example of Peeking, exclusions, and inconclusive outcomes before inspecting the resultApply Peeking, exclusions, and inconclusive outcomes to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Peeking, exclusions, and inconclusive outcomes; diagnose the visible failure and perform the corrective check

Day 120: Phase Project — Wayfinding Experiment

Build the first-principles mental model for Gallery constraints and randomization; name the question it answers and the unit of analysisPredict and trace a tiny example of Gallery constraints and randomization before inspecting the resultApply Gallery constraints and randomization to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Gallery constraints and randomization; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Success, time, and accessibility guardrails; name the question it answers and the unit of analysisPredict and trace a tiny example of Success, time, and accessibility guardrails before inspecting the resultApply Success, time, and accessibility guardrails to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Success, time, and accessibility guardrails; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Power and pre-specified analysis; name the question it answers and the unit of analysisPredict and trace a tiny example of Power and pre-specified analysis before inspecting the resultApply Power and pre-specified analysis to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Power and pre-specified analysis; diagnose the visible failure and perform the corrective check

Build the first-principles mental model for Expand, revise, or gather-more decision; name the question it answers and the unit of analysisPredict and trace a tiny example of Expand, revise, or gather-more decision before inspecting the resultApply Expand, revise, or gather-more decision to a realistic analyst decision while stating assumptions, units, and boundariesBreak one assumption behind Expand, revise, or gather-more decision; diagnose the visible failure and perform the corrective check

Deliver Museum Wayfinding Experiment for a museum visitor-experience team; support the decision: whether a new sign system should expand beyond pilot galleries?Reconstruct the critical workflow without the example open and disclose help usedFind and repair a planted data, logic, or communication failure with evidenceRespond to a changed requirement without being told which technique to useState limitations, non-goals, uncertainty, and what evidence would change the recommendation

Capstone

Museum Wayfinding Experiment

Build for a museum visitor-experience team. The result must support this decision: whether a new sign system should expand beyond pilot galleries? Include reproducible inputs, validation evidence, one tested failure, a changed requirement, and an explicit non-goal.

Reproducible source-to-output workflow
Documented quality checks and assumptions
Decision-ready output for a named user
Independent reconstruction and transfer evidence

Phase Complete!

After this phase, you'll be able to:

Estimation & Confidence Intervals
Hypotheses & Test Logic
Tests for Means
Tests for Proportions & Categories
Nonparametric & Resampling
Effect Size & Practical Significance
Errors, Power & Sample Size
Multiple Comparisons & Plans

Independently deliver Museum Wayfinding Experiment, diagnose a planted failure, explain the decision logic, and transfer the workflow to a changed requirement.

Back to all phases

Phase Goal

Day 109: Estimation & Confidence Intervals

Point and interval estimates

Repeated-sampling interpretation

Intervals for means and proportions

Confidence level and false probability claims

Practice, retrieval, and analyst judgment

Day 110: Hypotheses & Test Logic

Null, alternative, statistic, and reference distribution

Conditional p-value meaning

One- versus two-sided plans

Accepting null and p-as-effect errors

Practice, retrieval, and analyst judgment

Day 111: Tests for Means

One-sample, paired, and independent designs

t statistic and degrees of freedom

Welch comparison

Pairing and dependence failures

Practice, retrieval, and analyst judgment

Day 112: Tests for Proportions & Categories

Proportion comparisons

Chi-square tests

Expected counts

Absolute versus relative difference

Practice, retrieval, and analyst judgment

Day 113: Nonparametric & Resampling

Rank-based questions

Permutation tests

Bootstrap intervals

Wrong resampling units

Practice, retrieval, and analyst judgment

Day 114: Effect Size & Practical Significance

Mean and standardized difference

Risk difference and lift

Minimum worthwhile effect

Detectable but trivial results

Practice, retrieval, and analyst judgment

Day 115: Errors, Power & Sample Size

Type I and II errors

Power inputs

Minimum detectable effect

Observed-power misuse

Practice, retrieval, and analyst judgment

Day 116: Multiple Comparisons & Plans

Family-wise error

Bonferroni and FDR intuition

Primary and exploratory outcomes

Outcome switching and subgroup fishing

Practice, retrieval, and analyst judgment

Day 117: Experiment Units & Randomization

Treatment, control, experimental unit, and outcome unit

Assignment versus sampling

Blocking and cluster randomization

Interference and contamination

Practice, retrieval, and analyst judgment

Day 118: A/B Metrics & Guardrails

Primary behavior metric

Exposure and eligibility

Guardrail harm metrics

Novelty, ratios, and instrumentation

Practice, retrieval, and analyst judgment

Day 119: Experiment Analysis & Decision Memo

Balance and quality checks

Intent-to-treat effect

Decision thresholds

Peeking, exclusions, and inconclusive outcomes

Practice, retrieval, and analyst judgment

Day 120: Phase Project — Wayfinding Experiment

Gallery constraints and randomization

Success, time, and accessibility guardrails

Power and pre-specified analysis

Expand, revise, or gather-more decision

Phase gate: independent delivery and transfer

Museum Wayfinding Experiment

Phase Complete!