Skip to content
EN
EN RU
Previous Case

Case study / Enterprise security awareness

Reviewing simulation results and training

A security awareness administrator reviews simulated email responses and course completion. I redesigned reporting so they can distinguish events, find relevant training records and justify the next action.

Role
Product Designer
Team
4 designers / 4 devs / 1 PM / 1 QA
Project
Awareness10
Platform
Web App
Awareness10 dashboard with an organisation risk score, phishing trend and behaviour map

01

Problem

Companies send simulated phishing emails and assign security training. An administrator checks who clicked a link, who reported the message and who has not finished a course.

The previous reports kept those records apart. A recipient count did not explain people’s actions, and a campaign status did not show whether a particular employee had completed training.

I led the reporting redesign and designed the main views. I focused on one task: helping the administrator inspect an employee’s response and training before choosing the next action.

02

Research

Product goal

Identify whose training needs review

Give the administrator evidence about an employee’s response, assigned training and deadlines. A specialist still decides what further training is appropriate.

I reviewed post-simulation work with administrators: finding participants, checking training and preparing a report. The tasks asked them to explain events, choose report filters and identify a campaign needing attention.

The first round examined their reasoning; the second revisited the clarified structure. I assessed whether they could justify a next step, rather than count the pages they opened.

Administrators
6
The same people took part in both rounds; a repeat session was not counted as a new sample.
Tasks
3
Interpret a response, choose training filters and review campaign progress.
  • Interviews
  • Report review
  • Journey mapping
  • Two review rounds

I checked NIST research when interpreting the results. It helped separate course completion from behaviour and avoid explaining every change with a single number.

The email affects the result

The NIST Phish Scale considers how difficult an email is for its audience to recognise. I therefore described the lower simulated click rate as a programme outcome, without equating it with fewer real attacks.

Learning and behaviour have separate measures

NIST SP 800-50r1 treats completion and behavioural measures separately. I kept course completion and simulation responses in different reports, connected by the administrator’s task.

03

Insights

Repeated clicks are not additional people

One employee can click several times. Follow-up starts by identifying that person, then inspecting their individual actions.

BasisThe campaign has a Clicked measure; the expanded row separately shows Number of Clicks.

Reporting does not undo a click

An employee may have both a click and a report recorded against them. A single pass-or-fail outcome would hide one of those responses.

BasisClicked and Reported can both be marked against the same participant.

A completed campaign does not mean everyone finished training

Campaign state and employee progress answer different questions. Identifying whom to review requires individual training records.

BasisThe campaign list separates Status from Users progress; the training report shows a person, course and status.

04

Jobs and hypotheses

I defined two administrator jobs and tested whether the reports helped them prepare a next step.

Review a simulation result

When a simulation has ended, I want to understand individual employees’ responses and their training history, so I can decide whom to follow up with and why.

Review unfinished training

When a campaign deadline is approaching, I want to find people with unfinished training and inspect their records, so I can discuss timely follow-up with the people responsible.

Business hypothesis

B1

From a result to a justified review

If an administrator can compare a person’s simulation response with their training, they will be able to justify whose records to review next rather than give everyone the same next action.

Basis
A campaign summary shows scale, but does not replace reviewing a person’s response and training records.
How I tested it
I asked whose records they would review and why. A correct answer required interpreting the event and choosing the relevant training report. A choice based only on the total number of clicks did not support the hypothesis.

Supported in the task

In the repeat review, 5 of 6 administrators explained whom they would review next and which evidence they would use. Automatic training assignment was outside this task.

Design decision

Separated the overview, participant events and training records. The administrator retains the follow-up decision.

Interface hypotheses

I1

Separate people from actions

If participant responses appear in the row and repeated clicks in its detail, administrators will distinguish people from actions and account for both responses when Clicked and Reported are marked.

Basis
One participant may click several times and also be counted among those who reported the email.
How I tested it
I asked for the number of people who clicked, the selected participant’s click count and their other responses. Only an answer distinguishing these quantities passed; confusion required revisiting the labels.

Supported in the review

All 6 participants distinguished Clicked, repeated clicks and the Reported marker.

Design decision

Kept participant responses in the summary and row, with technical detail and click counts in the expansion.

I2

Review the selected employee’s training

If the training report has person and status filters beside course and campaign records, the administrator will choose the conditions needed to review that employee’s unfinished training.

Basis
A simulated click does not establish whether an employee was assigned a course or completed it.
How I tested it
I asked which fields would select the person and unfinished training. Opening the report alone did not count as success. Missing either condition required revisiting the filter panel.

Supported for filter selection

5 of 6 participants chose Users and Status without a hint. One chose status alone, which would include other people’s records.

Design decision

Grouped Campaign name, Groups, Users and Status, with explicit apply and reset controls.

I3

Review progress before opening a campaign

If a campaign row separates its state, employee progress and end date, the administrator will choose what to review first without mistaking campaign status for learning completion.

Basis
Campaign status describes its lifecycle. A campaign ending does not establish that every employee finished their courses.
How I tested it
I asked which campaign to review and how status, progress and timing informed that choice. An answer conflating campaign state and learning did not pass. Repeated confusion required clarifying the term.

Partially supported

5 of 6 participants distinguished campaign state from employee progress. One needed an explanation of Completed.

Design decision

Kept Status, Users progress and End Date separate. Individual training is reviewed in the training report, rather than inferred from campaign status.

05

User journey

The map separates the administrator’s previous work from my decisions for simulation review and training review.

Role
Security awareness administrator
Situation and goal
Inspect simulated email responses and decide whose training to review next.
Previous workflowMy decisions

  1. Review the overall picture

    Goal
    Choose which results need attention.
    Actions and touchpoints
    Collects figures from separate training and simulation reports.Separate summaries
    Difficulty
    Separate counts do not explain individual actions.
    Need
    An overview leading to a focused review.
    My decision
    Separated programme summaries from individual records.
    Screens

  2. Find participant responses

    Goal
    Understand who responded to the email.
    Actions and touchpoints
    Finds a person in the recipient list and compares report records.Recipient list / reports
    Difficulty
    A recipient may be mistaken for someone who clicked.
    Need
    Distinct events beside the person.
    My decision
    Showed Delivered, Opened, Clicked and Reported separately.
    Screens

  3. Inspect the event

    Goal
    Separate a person from repeated actions.
    Actions and touchpoints
    Clarifies the selected participant’s events by comparing separate records.Event records / exports
    Difficulty
    Several clicks can look like several people.
    Need
    Detail for the selected record.
    My decision
    Placed Number of Clicks and technical fields in the expansion.
    Screens

  4. Check the training record

    Goal
    Identify which personal records need review.
    Actions and touchpoints
    Matches the participant to records in a separate training report.Participants / training report
    Difficulty
    An email response does not describe course completion.
    Need
    Selection by employee and training status.
    My decision
    Grouped Users, Status, course and campaign in one report.
    Screens

  5. Review campaign and timing

    Goal
    Prioritise further review.
    Actions and touchpoints
    Opens campaigns in turn and checks progress and deadlines.Campaigns / training records
    Difficulty
    A campaign ending can be confused with everyone completing training.
    Need
    Separate measures and a deadline.
    My decision
    Added Status, Users progress and End Date to the row.
    Screens

  6. Prepare the review evidence

    Goal
    Share the basis for follow-up.
    Actions and touchpoints
    Combines relevant records and removes unrelated ones before a discussion.Selected records / working table
    Difficulty
    A broad report includes unrelated records.
    Need
    Clear filter conditions beside export controls.
    My decision
    Kept selection and export in the same view.
    Screens

In the proposed workflow the administrator moves between sections and chooses filters. The map ends with preparing a review; automatic course assignment is not shown.

06

Interface decisions

The four views answer different administrator questions. They compare the person and training records themselves, moving between sections.

Programme overview

Kept trends, training summaries and the department behaviour map on the dashboard. An overall measure helps choose what to inspect; it does not replace individual event records.

The selected person’s events

Separated email responses in the row. Expansion reveals device, browser, location and click count while keeping the main list compact. Clicked and Reported are not collapsed into a single outcome.

Selecting training records

Placed Campaign name, Groups, Users and Status beside the table. Each row shows the person, course, enrolment date, campaign and training status. The design shows selection controls, not an automatically assembled group.

Monitoring training campaigns

Brought campaign state, employee progress and end date into the list. The row menu offers Preview and Edit; creating a campaign remains a separate action. The assignment and launch wizard is outside these views.

Screens by stage

Choose a screen or scroll the row. Open any design to inspect the full screen.

Trends and training summaries help an administrator choose what to inspect.
Programme overview. Trends and training summaries help an administrator choose what to inspect.

Trends and training summaries help an administrator choose what to inspect.

On the screenSelect date / Employees Training / Behavior Map

The row shows a participant’s responses; repeated clicks are counted in the expanded detail.
Participant activity. The row shows a participant’s responses; repeated clicks are counted in the expanded detail.

The row shows a participant’s responses; repeated clicks are counted in the expanded detail.

On the screenUsers / Clicked / Reported / Number of Clicks

Employee and status filters sit beside training records and export controls.
Reviewing training. Employee and status filters sit beside training records and export controls.

Employee and status filters sit beside training records and export controls.

On the screenUsers / Status / Filter / Clear all filters / XLS / PDF

Campaign status, employee progress and the end date appear separately.
Campaign progress. Campaign status, employee progress and the end date appear separately.

Campaign status, employee progress and the end date appear separately.

On the screenStatus / Users progress / End Date / Preview

07

Design validation

I tested three product distinctions: people versus repeated events, an email response versus course completion, and campaign state versus employee progress. I asked for the reasoning behind a choice, rather than just locating a button.

People and repeated actions

Difficulty
A single total cannot explain how many people clicked or how many times a particular person did so.
What I changed
I separated responses in the summary and row, with Number of Clicks in the expansion. In the repeat review all 6 participants distinguished these quantities and accounted for Reported separately.

A response and a training record

Difficulty
Clicking a link does not show whether a course was assigned or completed. An administrator could choose too broad a report.
What I changed
I grouped Users and Status beside the employee, course and campaign. 5 of 6 participants chose person and training status as separate conditions; one retained too broad a selection.

Campaign state and employee progress

Difficulty
Completed can be read as confirmation that everyone finished learning. Campaign state and individual progress need separate interpretation.
What I changed
I separated campaign state, employee progress and timing. 5 of 6 participants explained the distinction; one needed clarification. I kept course-completion checks in the separate report containing individual training records.

08

Results

I connected simulation review and training review through clear levels: programme, campaign and person. The administrator has evidence for a next step and retains the decision about further training.

The first design review examined event meanings and report boundaries. The structure, fields and shared patterns were agreed at the second presentation; I then supported implementation and reviewed states with the team.

Over three quarters after rollout, simulated click rate fell from 27% to 6%, on-time training completion rose from 58% to 91%, and campaign configuration and launch took 16 minutes rather than 45.

People clicking a simulated link
Before the changes27%
Three quarters after rollout6%
21 percentage points lower
Participants in one simulation who clicked a test link at least once, divided by all participants in that simulation. The unit is a person within a particular simulation; repeated clicks do not add more people.
Training completed on time
Before the changes58%
Three quarters after rollout91%
33 percentage points higher
Assigned employees who completed that training by its deadline, divided by all employees assigned it whose deadline fell within the reporting window. Overdue unfinished records remain in the denominator.
Configure and launch a campaign
Before the changes45
Three quarters after rollout16
min 29 fewer minutes
From starting configuration to launching one campaign. This covers the whole preparation task, not employees’ learning time.

The measures describe the programme over three quarters after rollout. Email difficulty, training content and participant mix also affect responses, so the comparison does not isolate the interface’s contribution. A lower simulated click rate does not establish the same reduction in real attacks. Launch time covers the full campaign preparation workflow.

Durus production board showing eight stages of the galvanising line

Next case

Connecting orders to production batches / Durus

Continue

Let’s Connect

Download CV