Learning designers rarely say they are unconcerned with impact. It is an assumed good: something to be demonstrated, reported, and, where possible, improved. Yet the way impact is typically handled reveals a quieter truth. It is often treated as something that happens after design, not something that shapes it.
Evaluation frameworks, metrics, and post‑hoc analysis imply a sequence: design first, measure later. This creates a predictable illusion. If the metrics look acceptable, the design is validated. If they do not, adjustments can be made next time. It is a familiar and widely acceted approach. It is also insufficient.
Treating impact as a design constraint rather than an evaluation step changes that logic entirely. It removes the option of deferring judgement. It requires decisions to be justified in advance, rather than retrospectively explained once outcomes are known. The question is no longer “how will we measure whether this worked?” but “what must be true of this design for impact to be possible at all?”
This shift exposes the fragility of many common design assumptions.
For example, engagement is frequently used as a proxy for effectiveness. Completion rates, interaction levels, and satisfaction scores are routinely used to justify design decisions. Yet none of these reliably indicate impact. Learners can complete, click through, and report satisfaction without understanding, applying, or retaining anything of value. Engagement may be necessary, but it is not sufficient.
Designing for engagement can, in some cases, compete with designing for application, particularly where interaction is prioritised over cognitive effort or transfer.
This is where design begins to change in more substantive ways and where the consequences become difficult to ignore.
The same problem appears in content design. Clarity, structure, and alignment are often taken as indicators of quality. They are not. Well‑organised content can still fail to produce meaningful change. When impact is treated as a constraint, the focus shifts away from what is presented and onto what learners are able to do with it. If the design does not support application under real conditions, then it cannot be justified by how clearly it was explained. This often changes design choices in subtle but consequential ways, for example reducing content coverage to allow for practice, or prioritising feedback over additional explanation.
First, impact as a constraint forces attention onto behaviour, not just knowledge. It is not enough to design for successful task completion within a controlled environment. The design must account for what happens when learners return to work, experience partial recall, face competing demands, and operate without instructional support. If the design cannot plausibly function under those conditions, then it does not support impact.
Second, it exposes that interpretation is not a neutral analytical step but a design decision. Data does not carry meaning on its own. What counts as sufficient evidence, which indicators are prioritised, and how outcomes are inferred are all determined in advance through design choices. The selection of metrics is not separate from design; it is part of it.
Third, it introduces constraint in a literal sense. If impact is treated seriously, then design decisions cannot be justified by preference, habit, or efficiency. They either support the claimed outcome, or they do not. Some approaches will fail this test, even if they appear effective in the short term. Others will require additional structures (practice, feedback, integration with context) to make impact plausible. Design becomes a matter of sufficiency, not style.
One way to make this concrete is to take a current design and ask:
- What has to be true for this design to produce the claimed outcome?
- Which of those conditions are outside the learning design?
- What would count as failure but currently wouldn’t be measured?
- Which claim is being supported only by proxy indicators (e.g. engagement, completion)?
- Where would this design fail under realistic use conditions?
This kind of scrutiny has limits when done alone but has practical implications that are frequently ignored. If these questions cannot be answered convincingly, claims about impact are no longer defensible.
Designs that prioritise delivery over application do not hold up when judged against impact. Post‑course evaluation cannot compensate for decisions that were never aligned with meaningful outcomes. Common metrics such as completion, satisfaction, and short‑term recall provide only limited assurance, yet they are still used to justify broad claims about effectiveness.
Treating impact as a constraint makes those gaps visible. It raises the threshold for what can be claimed and, in doing so, reveals where design and evidence are misaligned. This is often the point where external scrutiny becomes useful, not to validate the design, but to test whether the claims being made can realistically hold.
This is not an abstract concern. It creates forward pressure.
If impact is genuinely a design constraint, then the conditions that enable or prevent it cannot be treated as external or secondary. They become part of the design problem. Accessibility, usability, and inclusion are not additional considerations or compliance requirements; they are points at which design assumptions are tested, and often where trade‑offs become unavoidable.
At that point, it is no longer enough to ask whether a design is well structured or engaging. The more difficult question is whether it remains valid when used by the full range of learners it is intended to serve, in the contexts where it is expected to function. That is where impact stops being something we measure and becomes something design either sustains or fails to support.
This kind of scrutiny is difficult to sustain when working inside a design. External review is often most useful here, not to validate the design, but to test whether the claims being made can realistically hold under the conditions in which it will be used.
Leave a Reply