Validate Data Definitions Before Analysis

Haley Vicini

1. Abstract

Before analyzing or modeling any dataset, confirm that its definition, calculation method, and source have remained consistent over time. Definition changes can introduce artificial structural breaks that distort correlations, weaken forecasts, and reduce analytical credibility.

2. Context

Apply this best practice when onboarding new internal data, integrating third-party indicators, or revisiting existing Workbenches and models. It is especially important for long historical series or datasets maintained across multiple reporting systems.

3. Content

3.1 Why It Matters

Time-series analysis assumes that a metric represents the same underlying concept across time. When definitions shift, even subtly, the data may appear continuous while the meaning changes.

Common sources of definition drift include:

• Changes in revenue recognition policies
• Reclassification of product lines or customer segments
• Modifications to survey methodology
• Unit scaling or currency conversion adjustments
• System migrations (e.g., ERP transitions)

If these changes are not identified and addressed, they can:

• Create artificial jumps or drops in the series
• Inflate or suppress correlation strength
• Introduce unstable coefficients in Predict
• Mislead stakeholders about trend direction

A model can only be as reliable as the consistency of its inputs.

3.2 How to Apply

Before adding a dataset to a Workbench, Predict, or a model:

1. Review documentation and metadata.
   Confirm how the metric is defined and whether any revisions are recorded.
2. Look for structural breaks in charts.
   Visual inspection often reveals abrupt level shifts that may reflect definition changes rather than real behavior.
3. Confirm ownership and source.
   Speak with data owners or review reporting notes to verify whether calculation methods changed.
4. Segment or annotate where needed.
   If definitions changed materially, consider:
   • Segmenting analysis into pre- and post-change periods
   • Adding a structural break indicator
   • Excluding earlier data if it no longer reflects the current operating model
5. Document findings.
   Record any definition changes and how they were handled for transparency and governance.

3.3 Example

A company tracks “Net Revenue” across eight years. In year five, accounting rules change to include shipping revenue. The time series shows a step increase that appears to signal growth. After reviewing documentation, the analyst identifies the definition shift and segments the analysis accordingly to avoid misleading correlations.

3.4 Common Pitfalls

• Assuming internal metrics are definitionally stable
• Ignoring small label changes in reporting dashboards
• Treating step changes as economic events without verification
• Failing to document adjustments for future analysts

3.5 Expected Results

• Cleaner, more interpretable time-series patterns
• Reduced risk of artificial structural breaks
• Stronger correlation validity
• More defensible forecasts
• Improved trust from stakeholders reviewing model outputs