About Test Cases

What is a test case?

SEMIP is set up to perform standard analyses on a number of test cases.  These test cases represent real-world applications of the various models contained in SEMIP.  In aggregate these test cases ideally cover the majority of real-world use cases for the types of models being analyzed by SEMIP.  In practice, the test cases are necessarily more limited, due to restrictions imposed primarily because of data availability, particularly for cases being tested against observations.

Each test case consists of:

1. a set of fires
2. an analysis procedure
3. collected data sets that can be used to either:

• initialize and run the models, thereby allowing all of the models involved to be directly compared based on equal inputs; and/or
• evaluate the models, thereby distinguishing not only model differences but also model accuracy.

An initial set of test cases has been identified for SEMIP Phase 1.  It is expected that additional test cases will be added to complement these initial cases, and that revisions to the test case list will occur periodically under guidance from the JFSP Board.

For information on obtaining the test case input information in order to run your own model and submit the results see How to run your own model for a test case below.

Identified initial test cases

The initial test cases were designed to match, as much as possible, a variety of use cases, temporal and spatial scales, and diversity of vegetation, region, fire type, and other considerations.  

• Identification of initial test cases
• Scales of the test cases
  
• Detailed description of test cases

How a test case works

Utilizing any collected test case input data, models or combinations of models are run to produce output at one of the output levels for the test case.  The model can be run locally, by the SEMIP project team, or run remotely, by any interested group, based on the test case guidelines (see below). 

Once the model output is created, it is translated into a format and units readable by the SEMIP analysis scripts for that case.  These scripts create statistical output data that is then incorporated into tables and graphics.  Generally only the aggregated statistics are shared, but in some cases, to illustrate a specific situation or problem, selected individual model run data may be shown in the analysis. 

The tables, graphics, and analysis text are then placed on the SEMIP Viewer website for broader community access.  Submitted model output data is not shared unless agreed to by the data creator.

Input data for identified test cases

Some of the test cases do not require specific input data to run, but others do.  For test cases that do, these input data are archived and made available through the SEMIP Viewer.  Those wishing to run their model for the test case and submit output data should download the input and follow the test case's guidelines to be included the test case analysis.

How to run your model for a test case

Each test case includes guidelines that model runs must follow to included in the test case comparison.  These guidelines separate the observational data used to create or train the models from that used to validate the models, and provide an equal footing for all model outputs.  See the Test case descriptions for more info.