From 17 April to 8 June 2001, a real-time forecast and research experiment was conducted in the Science Support Area of the NSSL/SPC facility. This program was a collaborative effort, organized by Mike Baldwin and Jack Kain, CIMMS colleagues within NSSL's Mesoscale Applications Group, and Paul Janish and Steve Weiss of the SPC. A diverse group of forecasters and scientists participated from NSSL and SPC, the Norman NWS/WFO, NCEP's Environmental Modeling Center (EMC), NOAA's Forecast Systems Laboratory, the University of Oklahoma, and Iowa State University.

The primary goal of the program was to investigate whether operational and experimental numerical weather prediction (NWP) models could be utilized more effectively to predict the initiation and development of severe convection. This overriding objective was mutually beneficial to NSSL and SPC because it incorporated important mission priorities from both sides. Specifically, it promoted NSSL's efforts to evaluate and improve NWP models while it supported the SPC's exploration of ways to reliably increase the projection time for severe thunderstorm and tornado watches.

Daily operations in the program involved three scientists and/or forecasters: 1) an SPC forecaster, 2) an NSSL/ CIMMS scientist, 3) an "external"scientist or forecaster (Figure 1). In addition, a number of visiting scientists participated at different times. These teams were responsible for issuing two forecast products and providing an extensive evaluation of numerical guidance products. The next day, the same group was tasked with providing a detailed subjective verification of both the forecast products and the model guidance.

Model evaluation became a core component of this program because NSSL and SPC collaborators have developed the perception that popular objective verification metrics often yield numbers that are inconsistent with the subjective impressions of SPC forecasters. Specifically, these metrics often fail to reflect the value of model forecasts to human forecasters. This deficiency has far reaching implications because model development strategies are guided largely by these verification standards. This implies that the needs of the SPC (and mesoscale forecasters in general) may not be well represented in development strategies unless different verification measures are applied.

As part of a larger goal to develop new verification metrics, one focus of this year's program was to document the disparity between prominent objective verification measures and the subjective impressions of severe weather forecasters and scientists at NSSL/CIMMS/SPC. Preliminary results from the program show success in this regard. For example, statistics show that an experimental version of EMC's Eta model, configured at NSSL in collaboration with SPC forecasters, typically provides better guidance for convective initiation and evolution than the operational Eta model (Figure 2). Meanwhile, the equitable threat score, a bellwether verification metric at EMC, favors the operational model (Figure 3).

Details about the 2001 NSSL/SPC Spring Program can be found at http:// www.spc.noaa.gov/exper/Spring_2001/.
By Jack Kain

Figure 2. Selected results from subjective verification of quantitative precipitation forecasts, including 00 and 12 UTC runs of the operational Eta model and NSSLŐs experimental version of the same model (Etakf). Models were ranked from 1 (best) to 4 (worst) every day during the Spring Program based on agreement between model forecasts and observed precipitation. Note that the Etakf runs ranked higher on average. Statistical significance can be inferred from results of paired t-tests. Larger view.

Fig. 3. Equitable threat score as a function of precipitation threshold for the same model runs and spatial and temporal domains as the subjective verification statistics. Note that the scores are very close but appear to favor the operational Eta model (higher scores are favored). Larger view