Experiments
Company
Skai (Formerly: Kenshoo)
Year
2020
Role
Design guidance
Case / 06
Experiments
Making complex A/B tests easier to create, monitor, and act on
Project
Company
Skai (Formerly: Kenshoo)
Year
2020
Environment
Web SaaS
Type
Zero-to-one
Engagement
Role
Design guidance
Hands-on
End-to-end for the Monitoring experience
Team
2 designer · 1 PM · 2 Dev. teams
Context
Turning experimentation into a decision-making capability.
Experiments are A/B tests that help advertisers evaluate potential campaign changes before applying them at scale. A user might test a new audience, geography, targeting strategy, or configuration on a limited group, observe its effect, and then decide whether the change should become part of the campaign.
Skai's ambition extended beyond reproducing the experimentation capabilities available in Google Ads and Facebook Ads Manager. The module was part of a broader strategy to add a decision-making layer above Skai's campaign activation platform.
This created an opportunity to build an experimentation experience specifically for expert advertisers—users who manage complex campaigns, work with large amounts of performance data, and need more than a simplified indication of which variation performed better.
The experience needed to support the complete decision journey: creating a valid experiment, following its progress, understanding the differences between test groups, and deciding what to do with the results.
Challenge
Expert users had the data, but not the clarity needed to make a decision.
Competitor analysis and client interviews exposed two connected problems: existing monitoring experiences were too shallow, while experiment setup was too complex.
The monitoring screens in Google Ads and Facebook Ads Manager provided a general overview of an experiment, but they did not give expert users enough information to confidently evaluate its performance.
Users wanted to understand how both test groups behaved over time—not only see their final results. They needed to compare groups across multiple metrics, recognize meaningful changes, and investigate the information behind the headline outcome. The existing tools lacked both the necessary depth and the visual representations required to make complex performance data understandable.
At the same time, creating an experiment required users to make several interconnected decisions. Some were tactical, such as defining the experiment's purpose. Others were technical, such as configuring the campaign entities and parameters involved.
Insight
An experiment is not a configuration followed by a result—it is a decision process unfolding over time.
The research revealed that the module's central purpose was not test creation or data presentation. Its purpose was to help users make a confident campaign decision.
That decision develops gradually. Users begin with a hypothesis, define how it should be tested, monitor the experiment while it is active, compare the behavior of its groups, and finally decide whether the tested change should be adopted.
For expert users, simplicity did not mean removing data. It meant organizing complexity so that the right information appeared at the right point in the decision process.
The interface therefore needed to remain visually quiet while making dense information easy to scan, compare, and understand. Data would be the primary content; the surrounding interface would act as a low-impact facilitator.
Design Solution
Home screen: A scannable experiment overview
The module's home screen needed to help users understand the state of multiple experiments quickly.
Each experiment exposed three essential dimensions:
- General information about the test.
- Its current stage.
- Its overall result.
The initial direction used a card-based layout. Direct comparison between experiments was not the primary use case, making a traditional table less suitable. Cards also offered an opportunity to introduce a new layout rather than repeat the data grids used extensively across Skai.
Development constraints made maintaining a new, complex component difficult. We also recognized that filtering and sorting were essential for managing a large number of experiments. These considerations led us back to the existing data-grid component—but with a new visual treatment. Generous spacing, reduced visual weight, and clearer separation softened the grid, making it feel more like a structured list while preserving its functional strengths.
Monitoring designed for evaluation
The monitoring screen became the module's main point of differentiation.
Instead of presenting only an experiment summary, it exposed the metrics expert users needed to evaluate both test groups. Users could compare group performance, examine changes over time, and move from a general result into more detailed evidence.
Charts and visual indicators translated dense performance data into recognizable patterns. This allowed users to understand direction, difference, and progression without relying solely on numerical values.
The result cards combined total values with their corresponding charts. Rather than positioning totals above the visualization—as conventionally done—I placed them directly within the chart area. This made it easier to compare multiple cards in one view and reduced the distance between a metric and its visual behavior.
Less frequently used metrics were moved into an Advanced tab. This preserved access to the depth expert users expected while keeping the default monitoring experience focused and readable.
The approach did not reduce the amount of information available. It introduced hierarchy, separating the information required for an initial evaluation from the information needed for deeper investigation.
Separating tactical and technical decisions
Mapping the complete creation flow revealed that two early decisions determined which configuration path the user would need later.
These decisions were tactical: they established the type and purpose of the experiment. The remaining decisions were more technical and depended on those initial selections.
We therefore separated the creation process into two stages.
The first stage collected the defining choices and allowed the system to generate the relevant configuration flow. The second stage presented only the technical inputs required for that experiment type.
This reduced unnecessary complexity and prevented users from having to interpret options that were irrelevant to their chosen path.
Selections from the previous stage remained visible at the top of the form. This supported recognition rather than recall, helping users understand the current configuration without remembering information from an earlier step.
Making progress visible over time
An experiment is a continuous process, not a one-time action. After setup, it moves through several stages before producing a result.
We introduced a timeline that explained this progression and made the experiment's current position visible. By showing what had happened, what was happening, and what would happen next, the timeline reduced uncertainty and strengthened users' sense of control.
Where numerical or technical information was difficult to interpret, we translated it into natural language. This made the meaning of the configuration more accessible without removing the underlying data expert users depended on.
Across the module, generous whitespace, restrained interface elements, clear typography, and selective color allowed the data to remain prominent. The interface became quieter as the information became richer.
Impact
A differentiated experimentation experience built around informed action.
The new module extended experimentation beyond the basic capabilities available in publisher platforms.
For monitoring, it gave expert users the detailed metrics, group comparisons, and visualizations they had identified as missing from existing tools. Instead of relying on a general result, users could examine how the experiment developed and evaluate the evidence behind its outcome.
For experiment creation, separating tactical choices from technical configuration made the process more understandable while preserving the flexibility required by different use cases. Persistent selections, a visible timeline, and natural-language explanations helped users maintain context and understand the process they were initiating.
The project also reinforced a broader direction for Skai's product design: expert tools do not become simpler by hiding their complexity indiscriminately. They become simpler when information is structured according to the decisions users need to make.
The available source material does not include quantified post-launch results. The clearest measures of success for the experience are therefore:
- Whether users can create experiments with less uncertainty.
- Whether they can understand the relationship between their initial choices and the generated configuration.
- Whether they can compare test groups without assembling information manually.
- Whether the monitoring experience helps them reach a decision faster.
- Whether users feel more confident applying—or rejecting—the tested campaign change.