← Project page

One replay interview, as the moderator saw it

Study 2 combined semi-structured interviews with the replay of each participant's own prototyping session. This page presents one session of a designer, following the structure of the interview guide, with the prompts and prototype states that the moderator used.

Before the interview

  1. 1TasksThree ten-minute tasks with Mark, in a familiar domain, on an existing prototype, and in an unfamiliar domain.
  2. 2SelectionDuring a short pause, a script selected anchor events from the research log.

The interview

  1. 3ReplayEach anchor event was recalled, watched, and discussed.Four events
  2. 4Shared vignetteA fabricated generation event, identical for all participants.One event
  3. 5DebriefThe session as a whole and the participant's practice.Six questions

Replay

The four anchor events selected from this session, in chronological order. For each event the participant first recalled the moment, then watched it in the session recording, and then answered the questions. In the second screenshot, red outlines mark the elements that the generation changed; a dark tag within a prompt denotes a pointed element or a sketch.

Familiar-domain task, a planner for language study04:42 in the session recording

A typical generation

The generation closest to the median number of changed nodes among those not followed by a repair.

Before the request
The prototype before the request
After the generation
The prototype after the generation, with the changed elements outlined in red
Prompt

make the number more self explanatory

Next prompt

put also among the past study session examples of skipped sessions

Questions for all participants

  1. Did this do what you asked?
  2. What did you do between this landing and your next prompt?
  3. Did seeing this change what you wanted to build?

Additional questions for designers

  1. What would you check before trusting this screen?

Anchor type A4 in the paper's appendix

Modification task, the events page of a library17:09 in the session recording

The first repaired generation

The first generation followed by a repair. The next prompt restated the request and extended it.

Before the request
The prototype before the request
After the generation
The prototype after the generation, with the changed elements outlined in red
Prompt

make a sign up completed page for when sign up is clicked. the corresponding selected event should be labeld as booked

Next prompt

when sign up is clicked the corresponding event should be labeled as booked and should have a button to cancel the reservation

Questions for all participants

  1. What was wrong here, in your words?
  2. Why do you think it came out like this?
  3. What options did you feel you had at that point?

Additional questions for designers

  1. If the result looked plausible: how would you have flagged this in a design review?
  2. If it was broken: what did this failure tell you about the tool?

Anchor type A2 in the paper's appendix

Unfamiliar-domain task, consignments for an auction house34:49 in the session recording

The first repaired generation of the last task

The next prompt restated the request and added a sketch indicating where the form should appear.

Before the request
The prototype before the request
After the generation
The prototype after the generation, with the changed elements outlined in red
Prompt

make a small form to the left to record new consignments

Next prompt

sketch make a small form to the left of the consignments to add new upcoming ones

Questions for all participants

  1. What was wrong here, in your words?
  2. Why do you think it came out like this?
  3. What options did you feel you had at that point?

Additional questions for designers

  1. If the result looked plausible: how would you have flagged this in a design review?
  2. If it was broken: what did this failure tell you about the tool?

Anchor type A2 in the paper's appendix

Unfamiliar-domain task, consignments for an auction house44:28 in the session recording

The final prototype

The state of the prototype when the last task ended.

Before the request
The prototype before the request
After the generation
The prototype after the generation, with the changed elements outlined in red
Prompt

button put this more to the right and make it actually round

Next prompt

None; the task ended here.

Questions for all participants

  1. What told you this was done?
  2. If the timer hadn't been there, what would you have done next?
  3. How much of this feels like yours?

Additional questions for designers

  1. Would you show this to a client as a starting point? What would you fix first?

Anchor type A3 in the paper's appendix

Shared vignette

A fabricated generation event, presented as coming from another participant's session and identical for all participants. The page of a neighborhood gym received the prompt "make the sign up button stand out and show how many spots are left for each class". The moderator showed the state before the request and then the state after the generation.

Before the request
The gym page before the request: weekday classes with plain sign-up buttons, a weekend program, and an about section
After the generation
The gym page after the generation: prominent sign-up buttons with spots left, a Become a member button, and an altered weekend program

Differences between the two states (not disclosed to participants)

Requested change
More prominent sign-up buttons and the number of remaining places for each class.
Unrequested addition
A Become a member button in the header.
Altered content
Saturday's Open gym removed and Sunday's Family circuit duplicated, while the page renders normally.

Questions for all participants

  1. Talk me through what you see. What changed between the two screens?
  2. Anything else that changed?
  3. What would you do next? Say it exactly as you would type it.
  4. What did the tool do here, in your judgment?

Debrief

The debrief addressed the session as a whole: whether participants had considered starting over, whether they would use such a tool outside the study, and what would allow them to trust its results without verifying every change. Designers were additionally asked how such a tool would relate to their professional practice. The final question invited topics that the interview had not covered.

Questions for all participants

  1. At any point, did you consider starting over from scratch? What made you stay, or restart?
  2. Outside this study, would you use a tool like this? For what?
  3. What would it take for you to trust what it gives you, without checking everything yourself?
  4. Is there anything you expected me to ask and I did not?

Additional questions for designers

  1. If a tool like this entered your design work, what would it change in how you work?
  2. Did anything in the tool conflict with how you normally work?

Before the request
Prompt

After the generation