Analysis Considerations and Procedures

Overview

SEMIP compares a variety of models with each other and with observations on a number of model output levels each with its own list of variables.   Because these output levels have significantly different characteristics, including data normalcy, availability, quantity, and more, the analysis protocols will vary depending on the output level (Basic Fire Information, Fuel Loading, etc...)

Specific concerns and notes about each output level are listed below.  The variables defined for each are listed in the Data tab.

SEMIP also conforms to the Model evaluation principles detailed here.

Model Performance Metrics

Model performance metrics are used to describe total model performance or model performance across a subset of data. 

Use of multiple statistics

Several statistics and metrics should be used to describe total model performance.  Using only one metric can give an unbalanced account. Additionally, model performance metrics should be flexible, and based on use cases (see Model Evaluation Principles), and using more than one statistic can facilitate this.

Observational error

At applicable output levels, model results will be analyzed with respect to observation data.  Appropriate model performance metrics and statistics will vary and be used to evaluate model performance depending on both the observation data in use and the modeling output level.

Understanding of observation measurement physical limitations and errors will assist with intelligent model evaluation.  Metrics that consider observation data the absolute truth should be used only when the known observational error is small. Often the modeling results do not completely physically (in space and time) represent the observation data (e.g. the model data represent a 2-d horizontal plane while the observational data are point locations) and this should be considered during the analyses.

Knowledge of observational error, magnitude, and variability may help determine if statistical metric results are substantial.  Consideration should be given to the area the point observation represents; the radius of representation may vary with distance and height (Boylan and Russell, 2006).  If the measurement’s radius of representation is similar in size to the grid cell used in the simulation than the point observation may represent the volumetric simulated value.  If the radius or representation is smaller then the grid cell then the measurement may not be the absolute truth and other metrics should be used to evaluate the model.

Non-normality and robust statistics

Much of the data is expected to be non-normal in distribution.  For example, ground concentrations of smoke are often zero, but never negative.  Fuel loadings vary considerably but overall do not conform to a normal curve.

Analyzing non-normal data requires careful use of statistics to avoid over emphasizing portions of the distribution.  In addition to simpiler statistics such as the mean or standard deviation, we also utilize so-called "robust" statistics such as the median and third - first quartile spread to avoid being over sensitive to outliers.

Metric and statistic definitions

Several performance metrics and statistics exist to determine whether the simulated dataset is within an acceptable range of the observed dataset.  A model performance metric is a measurement of how the model performs relative to observation values.  A statistic is a description of a dataset population or sample.  Evaluation tools include quantitative metrics and statistics paired in time and space and paired in space, graphical plots displaying paired and unpaired data, and categorical forecast analysis.

Equations for the metrics listed can be found in Metric and statistic definitions.

Model-to-model vs. model-to-observation

While the focus of this study is on the intercomparison of models (model-to-model comparisons), the origins of metrics and statics used in evaluating models focus on model-to-measurement comparisons.  While some of these methods only make sense when comparing models to measurements, most may be also used in model-to-model comparisons by substituting a second model’s estimates for the observations.  Alternatively, the median or mean of the model ensemble is sometimes used in place of observations as "truth".

Temporal and spatial scales

General considerations

The observational, model, and use case temporal and spatial scales vary considerably both across the various output levels and even within an output level.  See the Scales of the test cases page.

Observations vs. models

The temporal and spatial scales at each modeling output level may vary based on the model.  The observational data used in the comparison analysis will be aggregated to match the temporal/spatial scale of the modeling results.

Comparing model grid cells to point observations presents particular challenges.  A point measurement may not accurately represent the simulated value, if the simulated value is based on a 3-dimensional volume and in some cases the observational value may only represent a small area surrounding he measurement location or the observation value may not represent the absolute in-situ value (Boylan and Russell, 2006).  Using performance metrics that consider the observations as absolute truth should be done only when known measurement error is small and when the point measurements represents an area similar in size to the grid cell used in the simulation (US EPA, 2007).  Incorporating knowledge of measurement error into the analysis will assist with intelligent model evaluation.

Analysis considerations by output level

The goal of each step's analysis is to quantify the model-to-model variations and the model-to-observation differences at that output level both for scientific and user guidance, and to best describe the variability as it relates to modeling steps downstream in the modeling chain.  

Basic Fire Information

defined variables | potential observational datasets |  identified observational datasets

Specific considerations:

Fire occurrence data is of significantly varying quality.  Previous studies have found that ground reports can be significantly inaccurate in location and size.  Satellites have size and cloud issues.  Size from all sources is further complicated by not all lands burning in a perimeter.  The number of dataset source types (satellite, ground, helicopter, etc...) presents a qa/qc and bookkeeping challenge.

Identified analysis outputs:

• Basic statistics (mean/median, quartiles, peak)
• by Size categorization:
• Largest fires only
• Smaller fires only
• by Type categorization:
• Unplanned ignition
• Planned ignition
  
• by Regional categorization:
• all (CONUS)
• Geographic region (Northwest, Southeast, etc...)
  
• Forest Service region
  
• Regional Planning Organization (e.g. WRAP)
  
• State

Fuel Loading

Specific considerations:

Enormous amount of plot data available.  Fuel maps more limited.  Fuel loading variables are not standardized.  Some "fuel loading" maps are not originally intended for consumption, despite being used that way now.  Fuel loading maps are often at 1-km grids incapable of fine-scale variablity representation.  Newer fuels maps go down to 30-m but it is unclear how much fine-scale variability is captured.   Most consumption models can only take 1 fuel loading at a time.  Cross-dependencies between Fuel Loading and Total Consumption models make evaluation of Total Consumption dependent on the specific fuel loading variables available.

Identified analysis outputs:

• Basic statistics and non-normal statistics 
• Spatial statistics
• Categorical statistics to compare fuel types
  
• by Vegetation type categorization
  
• by Size categorization
• by Regional categorization

Total Consumption

Specific considerations:

Lots of observational pre- and post- burn plot data done by many different groups.  Different methodologies pose qa/qc issue in aggregating observations. across vegetation type.  Cross-dependencies between Fuel Loading and Total Consumption models make evaluation of Total Consumption dependent on the specific fuel loading variables available.  Not all fuel loading maps intended for consumption despite being used that way now.  Not all models can do different fire phases.  Flamming phase is most studied.   Several models programmatically linked (not fully independent). 

Identified analysis outputs:

• Basic statistics and non-normal statistics 
• Spatial statistics
• Normalized by total fuels
  
• by Vegetation type categorization
  
• by Size categorization
• by Regional categorization
• by Total fuels
  

Time Profile of Consumption

Specific considerations:

Very limited data.  Wildfire and prescribed fire profiles very different.  Every fire somewhat different.  No "right" answer except fire by fire;  otherwise "right" best defined by how influences next modeling steps (e.g. dispersion).  Time rate of consumption is sometimes dynamic (model created) sometimes static.  Easiest to analyze as % / hour profile (normalized to total consumption or total phase consumption). 

Identified analysis outputs:

• Basic statistics and non-normal statistics 
• Spatial statistics
• Categorical statistics to compare fuel types
  
• Vegetation type categorization
  
• Size categorization
• Regional categorization

Speciated Emissions

Specific considerations:

Nearly all speciated emissions created by simple emissions factors (EFs).  Sometimes split out by fire phase.  Easiest and most useful to directly compare EFs.  Heat release may require different statistical measures than other "species".

Identified analysis outputs:

• Basic statistics and non-normal statistics
  
• by Fire phase
• by Species, Species type
  

Vertical Plume Profiles

Specific considerations:

Virtually no observational data.  Some coincident satellite data.  Models critically depend on atmospheric profiles used, and in some cases on number of levels in atmospheric profile.  Models range from simple (e.g. plume top) to full plume profiles.  Questions on how to handle different fire phase components (e.g. 1 plume per phase).  Variations shown to critically affect resulting downwind ground concentrations.   Some models (e.g. current version of DAYSMOKE) require parameters such as number of cores to compute profile.  These parameters are often set in hindcast mode (by fitting downwind ground concentrations), making them only diagnostic, not prognostic.

Identified analysis outputs:

• Basic statistics, normalized by total smoke (% / m by height)
  
• by Fire size categorization
  
• by Fire type categorization (e.g. Wildfire, prescribed)
• by Regional categorization

Total Column Smoke

Specific considerations:

Lots of aerosol optical depth (AOD) satellite data.  AOD sensors/definitions vary depending on satellite.  Lots of visible smoke satellite data. AOD and visible smoke may be influenced by background color (e.g. veg type, region, clouds).  Smoke only models do not include other species affecting AOD/visible smoke.  Dispersion models often need modification to produce total column numbers.  Total column smoke not equal to AOD/visible due to height in atmosphere affecting optical depth, which requires de-convolution.  Total column smoke does not translate to ground smoke (e.g. smoke can exist only aloft).  

Identified analysis outputs:

• Basic statistics and non-normal statistics 
• Spatial statistics
• by Regional categorization

Ground Concentrations

Specific considerations:

Large in-situ networks exist, but data are generally widely separated in space.  Only some more intensive portable or temporary monitoring campaigns exist.  Data often reflects few fires.  Data also contains non-fire components.  Point to model issue with comparisons.  Additional issue with exactly what types of model data are useful (e.g., false alarm rates, non-detection rates tolerance by users).  Timing a known issue, so may need timed to within X hours type model evaluations.   Many air quality model evaluation statistics not appropriate for land management/air regulator use cases for fire.

Identified analysis outputs:

• Non-normal statistics 
• Spatial statistics
• Categorical statistics
• Threshold -> threshold model statistics
  
• by Region categorization
• by Season categorization
• by Fire type categorization
• by Fire size categorization 

Error propagation considerations

Because SEMIP chains models together, error propagation must be explicitly considered.  SEMIP does this by running all possible combinations of models, and by providing standard input and intermediate data sets for each test case.  For a full discussion of error propagation considerations, see:

• Error propagation