Significance and Metrics

Overview

Two things sit behind every number in Crosstabs and Scorecard: the metric that produced the value, and the letters that tell you whether a gap between two columns is real rather than noise.

Your new flavour scores 72% and the current one scores 65%. Is that a win, or is it the sample? Insights tests every pair of columns and marks the higher value with the letter of the column it beat, so you don't have to guess.

Where You'll See It

Significance testing runs in three places: Crosstabs, Scorecard, and the files you generate from the Reports tab. Those are the only places letters appear, and the only places the Confidence level control is offered.

It doesn't run on Explore, Summary, Penalty Analysis, Alienation Analysis or Downloads. Explore never receives significance data at all — so if a gap in an Explore chart looks meaningful, open the same question in Crosstabs to find out whether it is.

Reading the Letters

Every column carries a letter after its label — Product X (A). A value that's significantly higher than another column is marked with that column's letter, so a cell reading 85% B beat column B.

Three things catch people out:

• Only the higher value is marked. In a significant pair the winner carries the letter and the loser carries nothing. A cell with no letters hasn't necessarily failed a test — it may just be the lower side of one.
• Case means threshold. An uppercase letter clears your Confidence level. A lowercase letter clears only the looser Secondary confidence level. So Ab beat column A at your main threshold and column B only at the secondary one.
• The green sits in different places. Crosstabs turns the letters green and leaves the value plain; Scorecard turns the value green and bold and leaves the letters plain. Neither fills the cell with colour, and neither hides anything behind a hover — everything is visible as printed.

Setting Your Confidence Level

The Confidence level control is in the Settings sidebar. Your primary threshold defaults to 95% and can't be cleared — every table is always tested at something. You can raise it to 99% or relax it to 90%, 85% or 80%.

Secondary confidence level is optional and starts at None. It only offers levels below your primary, because its whole job is to be more permissive. Set your primary to 80% and the secondary has nothing left to offer.

Tip: Set a secondary level when a difference matters commercially but isn't clearing your main threshold. You keep the rigour of the primary test and still see the weaker signal, told apart by case.

Why a Value Has No Letters

An unmarked value is the most common source of confusion, and there are several different reasons for it.

One of the columns has fewer than 15 responses. Insights needs at least 15 in both columns before it will test a pair, and the count that matters is how many people answered that question — not the column's overall base. A Scorecard column can read N=248 and still go untested on a question only 12 people reached. Nothing on screen flags this; the value simply has no letters.

The metric is Median. Median is never tested, anywhere in Insights.

The question is a numeric entry question, where participants typed a number rather than picked a scale point. Neither its Mean nor its Median is tested.

The row isn't a testable one. Total Participants and Sigma are descriptive and never carry letters.

Both columns are at 0%, or both at 100%. There's no variance to test.

What Each Metric Measures

Box metrics group adjacent scale points and give you the percentage of people who answered within that group — top boxes for the positive end, bottom boxes for the negative end, middle boxes for the centre, which is what you want on a Just-About-Right question where the middle is the good answer.

Mean is the average of every response. Median is the middle response once they're sorted, so with an even number of responses it can land on a half point.

Which metrics you're offered depends on your scale length, and Crosstabs and Scorecard don't offer the same set. Two differences surprise people: Crosstabs never shows the single-point boxes — Top Box, Middle Box and Bottom Box are Scorecard only — and an 11-point scale offers nothing but Mean and Median, in Crosstabs alone.

Tip: Read Mean and Median together. Mean counts every response in proportion to its value and is the one that gets tested; Median tells you where the typical person sat and shrugs off outliers. When the two diverge sharply, something is skewing your distribution and is worth a look before you report the average.

For the curious: percentages are compared with a two-proportion Z-test and Means with a two-sample t-test, every pair tested against every other, two-tailed, with no correction for multiple comparisons.

Related

• Crosstabs — every metric a question supports, one table per question
• Scorecard — one chosen metric per question, compared across products
• Instant Reports — carries your confidence level into the generated deck

← Back to Insights Overview