Significance and Metrics

Overview

Your new flavour scores 72% and the current one scores 65%. Is that a win, or is it the sample? Insights tests the difference and marks the higher value, so you don't have to guess.

There are two comparisons. Letters compare one column against another. Numbers compare one answer choice against another inside the same column — the comparison a preference or ranking question is actually made of.

Where You'll See It

Letters appear in Crosstabs, Scorecard, and the files you generate from the Reports tab. Those are the only places they appear, and the only places the Confidence level control is offered.

Answer-choice numbers appear on screen in Crosstabs only. They aren't on the Scorecard, which reads scale questions rather than the preference and ranking questions this test needs, and they don't come through in the CSV or a generated deck.

Neither runs on Explore, Summary, Penalty Analysis, Alienation Analysis or Downloads. If a gap in an Explore chart looks meaningful, open the same question in Crosstabs to find out whether it is.

Reading the Letters

Every column carries a letter after its label — Product X (A). A value significantly higher than another column is marked with that column's letter, so 85% B beat column B.

Three things catch people out:

Only the higher value is marked. The winner carries the letter and the loser carries nothing, so a cell with no letters may just be the lower side of a pair.
Case means threshold. Uppercase clears your Confidence level; lowercase clears only the looser Secondary confidence level. Ab beat column A at your main threshold and column B at the secondary one.
The green sits in different places. Crosstabs turns the letters green; Scorecard turns the value green and bold. Neither fills the cell with colour, and neither hides anything behind a hover.

Reading the Answer-Choice Numbers

On a preference or ranking question, every answer choice is compared with every other choice in the same column. The choices are numbered — 1. Vanilla, 2. Chocolate — and a choice significantly preferred over another carries that choice's number. A cell reading 38% #3,4 was preferred over choices 3 and 4.

A cell can carry both, letters first: Ab #2,3 beat columns A and B, and choices 2 and 3. Two differences from the letters:

Only your main Confidence level applies. There's no lowercase tier here, so a secondary level changes nothing about the numbers.
A note under the table tells you the test ran — Answer choices are significance tested at the 95% confidence level, naming whichever level you've set. That's how you tell a table where nothing reached significance from one never tested at all.

Which questions are treated this way is set up by your Highlight team. If one you'd expect to see tested isn't marked, ask your account manager.

Setting Your Confidence Level

The Confidence level control is in the Settings sidebar. Your primary threshold defaults to 95% and can't be cleared — every table is always tested at something. You can raise it to 99% or relax it to 90%, 85% or 80%.

Secondary confidence level is optional, starts at None, and only offers levels below your primary.

Tip: Set a secondary level when a difference matters commercially but isn't clearing your main threshold. You keep the rigour of the primary test and still see the weaker signal, told apart by case.

Why a Value Has No Marks

Fewer than 15 responses. Insights needs at least 15 before it will test — in both columns for a letter, and in the column itself before any of its answer choices are compared. What counts is how many people answered that question, not the column's overall base. A Scorecard column can read N=248 and still go untested on a question only 12 people reached. Nothing on screen flags this.

Nobody picked one of the choices. An answer choice with no responses isn't compared — there's no preference to measure against an option no one chose.

The metric is Median. Median is never tested, anywhere in Insights.

The row isn't testable. Total Participants and Sigma are descriptive and never carry marks, and a numeric entry question — where participants typed a number rather than picked a scale point — has neither its Mean nor its Median tested.

Both columns are at 0%, or both at 100%. There's no variance to test.

What Each Metric Measures

Box metrics group adjacent scale points and give you the percentage who answered within that group — top boxes for the positive end, bottom boxes for the negative end, middle boxes for the centre, which is what you want on a Just-About-Right question.

Mean is the average of every response. Median is the middle response once they're sorted, so with an even number it can land on a half point.

Which metrics you're offered depends on your scale length, and Crosstabs and Scorecard don't offer the same set. Two differences surprise people: Crosstabs never shows the single-point boxes — Top Box, Middle Box and Bottom Box are Scorecard only — and an 11-point scale offers nothing but Mean and Median, in Crosstabs alone.

Tip: Read Mean and Median together. Mean is the one that gets tested; Median shrugs off outliers. When the two diverge sharply, something is skewing your distribution.

For the curious: columns are compared with a two-proportion Z-test and Means with a two-sample t-test. Answer choices within a column use a Z-test adjusted for the fact that the same people are choosing between those options. Every pair is tested two-tailed, with no correction for multiple comparisons.

Crosstabs — every metric a question supports, one table per question
Scorecard — one chosen metric per question, compared across products
Instant Reports — carries your confidence level into the generated deck

← Back to Insights Overview