Learning Analytics Dashboards Overstate Evidence, Warns

A learning analytics dashboard can make far-reaching decisions from surprisingly narrow evidence. A system might judge a learner's readiness for a role based on just eight correct answers in a ten-question quiz.

Every step from data to decision may appear reasonable. The underlying data can be perfectly accurate. Yet the final conclusion often claims more than the original evidence justifies. The report states that what a system observes is specific, like answers submitted correctly. What it may eventually represent is much broader, like a learner's understanding or readiness. The hidden interpretive step between these points is a core, yet often invisible, part of learning analytics.

From Activity Trace to Automated Action

Between a learner's activity and a system's action lies a chain that dashboards rarely make fully visible. It progresses from an activity trace to a metric, then to an inference, a profile, and finally a decision or action. Technical standards like xAPI and 1EdTech Caliper can record that an event happened. They do not establish what that event means for learning.

A metric organizes traces, such as an 80% score. An inference then attaches meaning, like declaring a learner understands a concept. A profile stabilizes that inference as a personal attribute. A decision acts on it, perhaps by changing a learning pathway. The report notes that every transition in this chain involves a design choice. The system applies a model where a pattern is treated as evidence of competence. That model may be reasonable, but it is still a model.

The Validity Gap in Flawless Data

Discussions often focus on data quality, like complete records or reliable timestamps. The report argues this is only the first layer. Data can be flawless and the conclusion still weak. An 80% score is strong evidence that eight answers were correct. It does not automatically demonstrate durable understanding or readiness for a consequential decision.

The issue is not only reliability but validity: whether the evidence actually supports the attached interpretation and use. Time-on-task metrics illustrate the problem clearly. A system knows two events occurred 15 minutes apart. It does not know what happened throughout those 15 minutes. A study cited by the report showed that different estimation methods can materially change analytics findings.

Precision of presentation is not strength of evidence. A 100% completion rate or an "advanced" label can look like facts. The metric becomes misleading when asked to answer a question it was never designed to answer.

The Persuasive Power of Profiles

Profiles are persuasive because they use stable-sounding nouns like skill, proficiency, or readiness. The verbs that produced them-observed, aggregated, inferred-are harder to see. A profile might display "Data Analysis, Advanced" without showing it was "inferred from two assessment scores, three completed activities, and a weighting rule applied six months ago."

This has practical consequences. A 2023 review of learning analytics research found that 71.1% of empirical articles did not include any measure of learning outcomes. The report says this shows why learner data, activity data, and evidence of learning should not be treated as interchangeable. Once an inference enters a profile, it can shape future evidence. A learner classified as advanced receives different opportunities, producing new data that feed the profile again. A hypothesis can become self-reinforcing.

AI Accelerates Consequential Inferences

AI does not invent the move from inference to action. Rules-based systems have long turned metrics into actions. What changes is the speed, scale, and opacity. A system can analyze a profile, identify a gap, generate an assessment, evaluate the response, update the profile, and select the next resource rapidly. The distance between "the model thinks" and "the system acts" becomes very short.

The quality of an inference cannot be separated from the authority granted to it. The report cites NIST's AI Risk Management Framework, which treats validity and transparency as context-dependent. In learning systems, context includes the consequence attached to the output. An AI recommendation leaves room for a pause where a human can accept or reject it. An automated action can remove that pause entirely. The report concludes that once a system is authorized to act on its own conclusions, asking "How good is this inference?" must always be paired with "What is this inference allowed to do?"