How Do You Actually Test an Augmented Reality Interface? Building a Valid HCI Experiment
How Do You Actually Test an Augmented Reality Interface? Building a Valid HCI Experiment
Building an AR prototype is the visible part of this thesis. The part that is harder to explain, and in many ways more intellectually demanding, is designing an experiment that can actually tell you whether the prototype works, under what conditions, and at what cost to the user.This post is about that design. Every decision in it has a reason, and the reasons matter.
Building an AR prototype is the visible part of this thesis. The part that is harder to explain, and in many ways more intellectually demanding, is designing an experiment that can actually tell you whether the prototype works, under what conditions, and at what cost to the user.This post is about that design. Every decision in it has a reason, and the reasons matter.
Why Controlled, and Why Within-Subjects
The study compares two conditions.
In the AR condition, participants use the Meta Quest 3 prototype: a floor path guiding them to each item, proximity-triggered product information overlays showing price, nutritional data, and allergen flags, and a quick-comparison display for the two items on the list that require choosing between alternatives.
In the baseline condition, participants receive a paper shopping list and shop the same environment using whatever they naturally use, including their own phone, with no AR assistance.
Every participant completes both conditions. This is called a within-subjects design, and the reason for choosing it over a between-subjects design is statistical: it eliminates individual performance differences as a source of noise. Shopping speed, spatial memory, and familiarity with product categories vary substantially between people. If those differences are distributed unevenly across groups, a between-subjects comparison cannot reliably attribute performance differences to the interface rather than to the people using it. With a within-subjects design, each participant serves as their own control.
Condition order is counterbalanced: half the participants complete the AR condition first, half complete the unassisted baseline first. This controls for order effects and for the possibility that completing one condition first changes performance in the second through spatial learning, fatigue, or practice effects.
Why the Baseline Is Unassisted Shopping, Not a Smartphone App
This is the most consequential methodological decision in the study, and it came directly from a flaw identified in the original proposal.
The original proposal described the comparison as AR against a smartphone app with a 2D store map. The academic review of that proposal flagged this as a confounding variable problem: comparing against a purpose-built smartphone navigation app simultaneously changes both the hardware platform and the interface design. Any performance difference found could be attributed to either change. You cannot isolate what the AR interface contributed.
The revised baseline is unassisted shopping with natural phone access: a paper list and the freedom to use a personal phone however the participant normally would. This represents what people actually do today, and it tests the question that matters practically: does AR-assisted shopping improve the experience compared to how people currently shop?
This choice also has support in the literature. Neeson and Lee (2025) found that a purpose-built smartphone AR navigation condition performed significantly worse than a control group with no navigation aid at all, because the head-down interaction pattern created a divided attention penalty. Testing AR against a smartphone app that might itself degrade performance would produce a misleading comparison. Testing against unassisted shopping is the cleaner and more honest baseline.
The Task: Why Compound Structure Matters
The shopping task is six items from a list, completed in a simulated grocery store built in a lab space with real shelves stocked with real grocery products. Participants physically walk through the space. Two of the six items require a comparison decision, for example "find the pasta that is gluten-free" or "pick the cheaper yogurt between these two brands."
That compound structure is deliberate. Pure navigation tasks have been studied extensively: find the room, reach the exit, walk from A to B. Grocery shopping is not a pure navigation task. It is navigation interleaved with visual search, product inspection, information retrieval, and comparison decisions, all at the same time.
A task design that captures only the navigation component would miss the cognitive demands that make the application context meaningful and would not test the product information and comparison features that represent the prototype's most novel design decisions.
Participants: Who and How Many
The target is 16 to 20 participants: adults aged 18 to 40 who grocery shop at least once per week. Prior AR experience is recorded as a covariate but not an exclusion criterion. The study is about how regular shoppers respond to the interface, not about how people who already know AR perform with it.
The sample size is justified by the within-subjects design and the expected effect magnitude. The XR versus tablet study (2025) found a large effect, Cohen's dz of 0.56, with 29 participants in a within-subjects design. Within-subjects designs require smaller samples than between-subjects designs to achieve equivalent statistical power because variance between individuals is removed from the error term. For n of 16 to 20 in a within-subjects design, the study is adequately powered to detect medium to large effects, which is the realistic target for a prototype comparison where the interface differences are substantial.
What Is Being Measured, and Why Each Instrument
Task performance is logged automatically by the system during the task:
Total task completion time
Number of items found correctly
Product selection accuracy on the comparison sub-tasks
Navigation path length versus the optimal path from system logs
These are objective and do not depend on self-report.
Usability is measured using the System Usability Scale (SUS), administered after the AR condition. SUS produces a validated 0 to 100 score with published grade benchmarks. Haesen et al. (2019) reported SUS scores of 66.6 for HoloLens 1 navigation and 86.6 for smartphone navigation, which gives my results a published benchmark to be positioned against rather than evaluated in isolation.
Cognitive workload is measured using the Raw NASA-TLX, administered after each condition. The instrument has six subscales: Mental Demand, Physical Demand, Temporal Demand, Effort, Performance, and Frustration. Reporting all six rather than a single total score is what allows the theoretical predictions to be tested separately. The reason that matters is explained in the next section.
Qualitative data is collected through semi-structured post-task interviews of five to eight minutes per condition, combined with observer notes taken during the task. These capture breakdowns, confusion points, ignored features, and contextual explanations for the quantitative patterns that numbers alone cannot provide.
The Theoretical Tension That Makes Subscale Analysis Necessary
The study is grounded in three theoretical frameworks that make competing predictions. Those predictions are what make the subscale breakdown analytically interesting rather than just statistically thorough.
Situated Cognition (Suchman, 1987; Lave, 1988) predicts that spatially anchored AR overlays reduce cognitive demand by offloading spatial memory and information retrieval to the environment. Instead of remembering aisle locations, the floor path shows the way. Instead of reading packaging fine print, key data appears in the visual field. This predicts lower Mental Demand in the AR condition.
The Technology Acceptance Model (Davis, 1989) predicts that even when a system objectively improves performance, user acceptance depends on perceived ease of use and social comfort. The requirements survey found that only 4 of 20 respondents felt comfortable wearing AR glasses in a store. Device unfamiliarity and social discomfort are likely to elevate Frustration and Effort subscale scores in the AR condition, even if Mental Demand decreases.
Cognitive Load Theory (Sweller, 1988) predicts that AR simultaneously reduces extraneous load by eliminating aimless searching and introduces new extraneous load through visual clutter, split attention between physical products and virtual overlays, and the overhead of managing an unfamiliar device. The net direction depends entirely on how well the interface is designed.
These three frameworks do not agree. Mental Demand may decrease while Frustration increases. Task performance may improve while overall workload increases. The subscale structure of NASA-TLX is what allows these competing effects to be separated in the data rather than averaged into a single score that would obscure them.
The Session Structure and Analysis Plan
Each session runs approximately 65 minutes:
Consent and introduction → 3-minute familiarisation with first condition → Task Set A → Post-condition questionnaires (NASA-TLX + SUS if AR condition) → 5–8 minute interview → Short break → Familiarisation with second condition → Task Set B → Questionnaires → Interview → Closing preference question
For quantitative analysis: normality is tested using Shapiro-Wilk for each dependent variable. If normally distributed, paired-samples t-tests are used. If not, Wilcoxon signed-rank tests. Effect sizes are reported as Cohen's d for parametric results and r for non-parametric, alongside p-values. Reporting effect sizes is not optional with this sample size. It is the only way to assess practical significance independently of whether a result crosses a significance threshold.
For qualitative analysis: interview transcripts are combined with observer notes and subjected to thematic analysis using a deductive coding framework covering navigation issues, product-finding difficulties, overlay problems, visual clutter episodes, comparison overlay usage, social comfort concerns, and device comfort. Inductive codes are added for patterns that emerge beyond the framework.
The Ecological Validity Trade-Off, Stated Plainly
The study environment is a physical lab space. It has real shelves with real grocery products. Participants physically walk through it. The AR overlays appear on top of actual packaging with real visual complexity.
What it does not have: the scale of a real supermarket, ambient noise, other shoppers, time pressure, or the full attentional context of a real shopping trip. These are real limitations and they are acknowledged as such in the thesis.
Ecological validity is always a trade-off against measurement control. A real supermarket would produce data that reflects real shopping more closely. It would also make it impossible to precisely measure task completion time, navigation path efficiency, or comparative workload under controlled conditions.
The lab environment preserves the physical interaction that makes the task meaningful while providing the measurement control that makes the findings interpretable. That is the trade-off, stated plainly rather than buried in a limitations section.
You can have the review of the Proposal here https://mseymur.framer.website/masters-proposal
Why Controlled, and Why Within-Subjects
The study compares two conditions.
In the AR condition, participants use the Meta Quest 3 prototype: a floor path guiding them to each item, proximity-triggered product information overlays showing price, nutritional data, and allergen flags, and a quick-comparison display for the two items on the list that require choosing between alternatives.
In the baseline condition, participants receive a paper shopping list and shop the same environment using whatever they naturally use, including their own phone, with no AR assistance.
Every participant completes both conditions. This is called a within-subjects design, and the reason for choosing it over a between-subjects design is statistical: it eliminates individual performance differences as a source of noise. Shopping speed, spatial memory, and familiarity with product categories vary substantially between people. If those differences are distributed unevenly across groups, a between-subjects comparison cannot reliably attribute performance differences to the interface rather than to the people using it. With a within-subjects design, each participant serves as their own control.
Condition order is counterbalanced: half the participants complete the AR condition first, half complete the unassisted baseline first. This controls for order effects and for the possibility that completing one condition first changes performance in the second through spatial learning, fatigue, or practice effects.
Why the Baseline Is Unassisted Shopping, Not a Smartphone App
This is the most consequential methodological decision in the study, and it came directly from a flaw identified in the original proposal.
The original proposal described the comparison as AR against a smartphone app with a 2D store map. The academic review of that proposal flagged this as a confounding variable problem: comparing against a purpose-built smartphone navigation app simultaneously changes both the hardware platform and the interface design. Any performance difference found could be attributed to either change. You cannot isolate what the AR interface contributed.
The revised baseline is unassisted shopping with natural phone access: a paper list and the freedom to use a personal phone however the participant normally would. This represents what people actually do today, and it tests the question that matters practically: does AR-assisted shopping improve the experience compared to how people currently shop?
This choice also has support in the literature. Neeson and Lee (2025) found that a purpose-built smartphone AR navigation condition performed significantly worse than a control group with no navigation aid at all, because the head-down interaction pattern created a divided attention penalty. Testing AR against a smartphone app that might itself degrade performance would produce a misleading comparison. Testing against unassisted shopping is the cleaner and more honest baseline.
The Task: Why Compound Structure Matters
The shopping task is six items from a list, completed in a simulated grocery store built in a lab space with real shelves stocked with real grocery products. Participants physically walk through the space. Two of the six items require a comparison decision, for example "find the pasta that is gluten-free" or "pick the cheaper yogurt between these two brands."
That compound structure is deliberate. Pure navigation tasks have been studied extensively: find the room, reach the exit, walk from A to B. Grocery shopping is not a pure navigation task. It is navigation interleaved with visual search, product inspection, information retrieval, and comparison decisions, all at the same time.
A task design that captures only the navigation component would miss the cognitive demands that make the application context meaningful and would not test the product information and comparison features that represent the prototype's most novel design decisions.
Participants: Who and How Many
The target is 16 to 20 participants: adults aged 18 to 40 who grocery shop at least once per week. Prior AR experience is recorded as a covariate but not an exclusion criterion. The study is about how regular shoppers respond to the interface, not about how people who already know AR perform with it.
The sample size is justified by the within-subjects design and the expected effect magnitude. The XR versus tablet study (2025) found a large effect, Cohen's dz of 0.56, with 29 participants in a within-subjects design. Within-subjects designs require smaller samples than between-subjects designs to achieve equivalent statistical power because variance between individuals is removed from the error term. For n of 16 to 20 in a within-subjects design, the study is adequately powered to detect medium to large effects, which is the realistic target for a prototype comparison where the interface differences are substantial.
What Is Being Measured, and Why Each Instrument
Task performance is logged automatically by the system during the task:
Total task completion time
Number of items found correctly
Product selection accuracy on the comparison sub-tasks
Navigation path length versus the optimal path from system logs
These are objective and do not depend on self-report.
Usability is measured using the System Usability Scale (SUS), administered after the AR condition. SUS produces a validated 0 to 100 score with published grade benchmarks. Haesen et al. (2019) reported SUS scores of 66.6 for HoloLens 1 navigation and 86.6 for smartphone navigation, which gives my results a published benchmark to be positioned against rather than evaluated in isolation.
Cognitive workload is measured using the Raw NASA-TLX, administered after each condition. The instrument has six subscales: Mental Demand, Physical Demand, Temporal Demand, Effort, Performance, and Frustration. Reporting all six rather than a single total score is what allows the theoretical predictions to be tested separately. The reason that matters is explained in the next section.
Qualitative data is collected through semi-structured post-task interviews of five to eight minutes per condition, combined with observer notes taken during the task. These capture breakdowns, confusion points, ignored features, and contextual explanations for the quantitative patterns that numbers alone cannot provide.
The Theoretical Tension That Makes Subscale Analysis Necessary
The study is grounded in three theoretical frameworks that make competing predictions. Those predictions are what make the subscale breakdown analytically interesting rather than just statistically thorough.
Situated Cognition (Suchman, 1987; Lave, 1988) predicts that spatially anchored AR overlays reduce cognitive demand by offloading spatial memory and information retrieval to the environment. Instead of remembering aisle locations, the floor path shows the way. Instead of reading packaging fine print, key data appears in the visual field. This predicts lower Mental Demand in the AR condition.
The Technology Acceptance Model (Davis, 1989) predicts that even when a system objectively improves performance, user acceptance depends on perceived ease of use and social comfort. The requirements survey found that only 4 of 20 respondents felt comfortable wearing AR glasses in a store. Device unfamiliarity and social discomfort are likely to elevate Frustration and Effort subscale scores in the AR condition, even if Mental Demand decreases.
Cognitive Load Theory (Sweller, 1988) predicts that AR simultaneously reduces extraneous load by eliminating aimless searching and introduces new extraneous load through visual clutter, split attention between physical products and virtual overlays, and the overhead of managing an unfamiliar device. The net direction depends entirely on how well the interface is designed.
These three frameworks do not agree. Mental Demand may decrease while Frustration increases. Task performance may improve while overall workload increases. The subscale structure of NASA-TLX is what allows these competing effects to be separated in the data rather than averaged into a single score that would obscure them.
The Session Structure and Analysis Plan
Each session runs approximately 65 minutes:
Consent and introduction → 3-minute familiarisation with first condition → Task Set A → Post-condition questionnaires (NASA-TLX + SUS if AR condition) → 5–8 minute interview → Short break → Familiarisation with second condition → Task Set B → Questionnaires → Interview → Closing preference question
For quantitative analysis: normality is tested using Shapiro-Wilk for each dependent variable. If normally distributed, paired-samples t-tests are used. If not, Wilcoxon signed-rank tests. Effect sizes are reported as Cohen's d for parametric results and r for non-parametric, alongside p-values. Reporting effect sizes is not optional with this sample size. It is the only way to assess practical significance independently of whether a result crosses a significance threshold.
For qualitative analysis: interview transcripts are combined with observer notes and subjected to thematic analysis using a deductive coding framework covering navigation issues, product-finding difficulties, overlay problems, visual clutter episodes, comparison overlay usage, social comfort concerns, and device comfort. Inductive codes are added for patterns that emerge beyond the framework.
The Ecological Validity Trade-Off, Stated Plainly
The study environment is a physical lab space. It has real shelves with real grocery products. Participants physically walk through it. The AR overlays appear on top of actual packaging with real visual complexity.
What it does not have: the scale of a real supermarket, ambient noise, other shoppers, time pressure, or the full attentional context of a real shopping trip. These are real limitations and they are acknowledged as such in the thesis.
Ecological validity is always a trade-off against measurement control. A real supermarket would produce data that reflects real shopping more closely. It would also make it impossible to precisely measure task completion time, navigation path efficiency, or comparative workload under controlled conditions.
The lab environment preserves the physical interaction that makes the task meaningful while providing the measurement control that makes the findings interpretable. That is the trade-off, stated plainly rather than buried in a limitations section.
You can have the review of the Proposal here https://mseymur.framer.website/masters-proposal
