Licensing measurement mistakes, from calling activity a result to merging case states

Licensing data can look precise while mixing unlike events, places and units. The following mistakes lead to confident but unusable reporting. Each correction begins with a defined decision and a record that can be checked.

What to take away

1. Calling activity a result

Page views, searches and form starts show activity. They do not prove application, grant or lawful operation. The Department for Business and Trade's Find a licence service lists licences and permits by activity and place, but it is a discovery tool and its catalogue may be incomplete. Measure the later confirmation separately.

2. Merging case states

Case states defined

Submitted, acknowledged, decided and conditions implemented require different evidence. A dashboard should not label all four complete. Audit a sample against underlying documents.

3. Hiding unknowns from the denominator

Removing unresolved cases can improve a rate without improving the service. Define missing, excluded and unknown records and report their counts. Explain any change to the population.

For example, a council records 200 premises licence applications. It decides 160 within the target period and leaves 40 unresolved. Reporting 160 of 200 gives 80 per cent. Reporting 160 of 160 decided gives 100 per cent and hides the 40 open cases. Publish the 40 unknown cases beside the rate. Illustrative figures.

4. Mixing businesses, sites and applications

The Office for National Statistics' business activity, size and location tables use stated business definitions and regional breakdowns. They count businesses and local units, not licences. One enterprise may have several premises or permissions.

5. Treating form completion as acceptance

The Food Standards Agency's registration service collects operator, establishment and activity information for routing. The service records a notification for food hygiene routing; it does not grant a licence or inspect premises. Progress through those fields is a service event, not evidence of the authority's later conclusion or of unrelated permissions.

6. Ignoring geography and case mix

An England service can cover district councils, London boroughs and licence types such as premises licence and food hygiene registration. Those differences change case mix. Compare like with like and log definition changes.

7. Claiming causation from timing

A metric may improve after new content or software while other factors also change. Use case review, cohorts or process tracing before making a contribution claim. Government Digital Service guidance on using data to improve a service covers baselines, measure selection and review; it does not provide licence-specific definitions.

8. Optimising the target instead of the journey

Pressure to raise completion can weaken checks or exclude difficult users. Pair speed and throughput with quality, accessibility and recovery measures. Investigate negative cases.

9. Treating missing data as zero

An absent decision, authority or condition owner is not a negative event and not a success. Create an unknown state and investigate why the record is incomplete. Use that unknown flag in case review so the gap does not become a zero or a pass.

10. Changing definitions without a break

A new form, tracking method or case rule can move a chart even when user experience has not changed. Version the metric definition and mark the first period under the new method. Where possible, calculate an overlap period both ways to show the effect.

11. Building a public claim from an internal signal

An internal dashboard can use provisional indicators to prompt investigation. Advertising or comparative publishing needs firmer evidence, stable definitions and disclosed limitations. Review the exact wording against the data file before release and keep the claim to what the file supports.

Keep a correction route for users and case owners who spot a misclassified record. Review the evidence without exposing unnecessary personal information, update affected calculations and note the date of any public amendment. Repeated disputes about one metric may show that the definition itself is unusable.

Before publication, ask another analyst to reproduce the denominator, event and chart. Withdraw any statement whose data lineage or interpretation cannot survive that review.

Before you act

Common questions

Why is form completion not evidence of a licence being granted?

Food hygiene registration is a notification to a local authority, not a permission. A completed form proves routing data arrived, not that a decision followed. For a premises licence, grant and conditions implemented are later states.

What should happen when a metric definition changes?

If the new method starts in April, recalculate April cases under both old and new definitions. Suppose the old rule gives 70 per cent and the new rule gives 62 per cent. The 8 point gap is a method effect, not a service fall.

When can an internal dashboard signal be used in a public claim?

Set a higher evidence bar for publication. Two analysts should reproduce the figure, and the file should show unknown cases. If either condition is missing, publish the method note instead of the claim.