Education Analytics Curriculum Research Case Study

Do university math courses
deliver what they promise?

A structured gap analysis of 74 courses and 469 learning outcomes at one of Latin America's top universities — coded, indexed, and mapped to reveal what's taught versus what's promised.

74 Courses analyzed
469 Learning outcomes coded
6 Degree programs
0 Uses of "validate"
Tools & methods: Structured text coding Custom index design Gap analysis Data visualization

01 · The question

A promise written in the
graduate profile

PUC-Chile publicly commits — in six official degree program profiles — to producing graduates who can build, apply, and validate mathematical models to solve real-world problems. These are not vague aspirations; they are binding institutional statements.

But nobody had measured whether the actual catalog of 74 mathematics courses, with their hundreds of declared learning outcomes, delivers on that commitment. This project asked the data.

"The catalog teaches students to work inside mathematical models with growing cognitive depth — but not how to build, validate, or communicate those models in connection with reality. Graduates master the tools but not the complete process their profile promises."

02 · Data & methodology

From 469 raw syllabi texts
to four custom indices

Official syllabi were collected from PUC-Chile's public course catalog. Each course's learning outcomes were extracted, structured into rows, and coded across three independent analytical dimensions. Four composite indices were then computed per course.

📐

Cognitive Level (CCI)

Each outcome's main verb was classified into one of six cognitive levels using the revised Bloom's taxonomy — from Remember (L1) to Create (L6).

Anderson & Krathwohl 2001
📚

Knowledge Type (KDI)

The object of each outcome was coded as Factual, Conceptual, Procedural, or Metacognitive knowledge — revealing what kind of knowing is developed.

4-dimension taxonomy
🔄

Modeling Cycle (MCI)

Each course was scored for the presence of all 7 phases of the modeling cycle — from understanding a real situation (F1) all the way to validation (F6) and presentation (F7).

Blum & Leiß 2007
🗺️

5 Curricular Strata

Courses were segmented into five layers: E1 Foundational, E2 Service, E3 Bachelor's core, E4 Electives, and E5 Teacher Education.

📊

4 Custom Indices

CCI (cognitive complexity), MCI (modeling coverage), KDI (knowledge orientation), and TDI (taxonomic diversity) — each 0-to-max-scaled and computed per course.

🧩

Course Typology

Each of the 74 courses was classified into one of four action categories (A–D) based on combined indices, to prioritize intervention efforts.

03 · Key findings

Four findings that tell
a consistent story

01

The word "validate" appears zero times

Across all 469 learning outcomes in the catalog, the verb validate — the act of checking whether a mathematical model actually fits reality — does not appear once. Phase 6 of the modeling cycle (Validation) is absent in 94.6% of courses.

Evaluate-level outcomes (L5): 2.6% of all coded outcomes — the second-lowest cognitive level by frequency.

This is not a minor gap. Validation is precisely what the graduate profiles promise and what employers expect of a trained mathematician.

Cognitive level distribution — 469 outcomes
0 Occurrences of "validate" in 469 outcomes
02

Only 1 in 4 courses is in the ideal zone

Plotting every course on a two-dimensional map — cognitive complexity (CCI) on the Y-axis, modeling cycle coverage (MCI) on the X-axis — reveals that only 26.1% of courses fall in the desired top-right quadrant: high complexity and high modeling coverage.

26.1% in Q-II: courses with high cognitive depth but almost no modeling exposure — the highest-potential target for low-cost curriculum reform.

The average modeling coverage index across the full catalog is just 23.8% of the 7-phase cycle.

Curriculum map — 74 courses by complexity × modeling
Q-I: Ideal zone (26.1%)
Q-II: High potential, low modeling (26.1%)
Q-III: Needs full redesign (26.1%)
Q-IV: Low depth, some modeling (21.7%)
03

Cognitive depth grows — modeling coverage stays flat

Tracking both indices semester by semester through the Mathematics Bachelor's degree reveals a striking divergence: cognitive complexity rises steadily from level 3.5 to 4.4 as students advance — exactly as intended.

Modeling coverage never breaks 40% at any semester. There is no point in the trajectory where modeling is systematically introduced.

Students graduate analytically capable but without having completed a full modeling cycle — precisely the opposite of what the degree profile promises.

Semester trajectory — Mathematics Bachelor's degree
04

Graduate profiles promise what courses don't deliver

The two degree programs with the most explicit modeling commitments in their graduate profiles — Engineering and Mathematics — show the largest gap between what is promised and what the course catalog provides.

Engineering and Mathematics average MCI = 29% across their MAT courses, while explicitly committing to modeling competencies in their official profiles.

Only the Teacher Education program shows an isolated spike — a single course (ECM202M, MCI = 85.7%) acting as a revelation rather than a culmination.

Average modeling coverage by degree program (MCI %)

03b · Modeling cycle deep dive

Which phases exist — and which are missing

Scoring the presence of each of the 7 modeling cycle phases across all 74 courses reveals a structural pattern: the catalog is anchored in Working Mathematically (F4) and Understanding the situation (F1), while the phases that connect math to reality — Simplification (F2), Interpretation (F5), and especially Validation (F6) — are virtually absent.

Average phase coverage across all 74 courses

The missing half of the cycle

A complete modeling cycle requires moving from reality into mathematics (F1→F4) and then back into reality (F5→F7). The catalog is strong in the first direction — and nearly silent in the second.

F4 Work mathematically ~47% avg
F1 Understand situation ~38% avg
F5 Interpret results ~13% avg
F2 Simplify / assume ~11% avg
F6 Validate ≈ 0%

04 · Recommendations

Three evidence-based
actions

The data does not just identify gaps — it prioritizes them. The combined indices create a triage system: where to act first, and at what cost.

Target the 18 high-potential courses first

Q-II courses already have the cognitive depth; they just need modeling phases added to learning outcomes and assessments. These are the highest-ROI interventions.

High impact · Low cost

Make validation (F6) explicit in at least one course per stratum

The word "validate" appears zero times in 469 outcomes. A single targeted change per curricular level would close the most glaring gap in the catalog.

Critical fix · Immediate

Build a progressive modeling trajectory, not isolated spikes

ECM202M works as a benchmark — but students should not encounter a complete modeling cycle for the first time in semester 9. Distribute phases progressively from year 1.

Structural reform · Medium-term

05 · Skills demonstrated

What this project shows
about my work as an analyst

🔍

Text data extraction & structuring

Transformed 74 unstructured syllabi into 469 coded, analyzable rows — without any existing dataset.

📐

Custom metric design

Designed four domain-specific indices (CCI, MCI, KDI, TDI) grounded in theoretical frameworks, not off-the-shelf measures.

🗺️

Multi-dimensional gap analysis

Built a 2D curriculum map that cross-cuts cognitive complexity and modeling coverage to surface actionable quadrants.

📈

Longitudinal pattern detection

Tracked both indices across semesters to identify the divergence between cognitive growth and modeling stagnation.

🎯

Evidence-based prioritization

Classified all 74 courses into a four-tier action typology (A–D) to focus reform effort where it matters most.

📣

Data storytelling

Translated a rigorous 8-step academic analysis into findings that are clear, visual, and actionable for decision-makers.