HomeMy WebLinkAboutAQ_GEN_PLNG_20220404_SIP_RH-SIP_AppE5 MPE
Appendix E-5
Model Performance Evaluation for Ozone of the
CAMx 6.40 Modeling System and the VISTAS II 2011
Updated Modeling Platform
August 17, 2020
This page intentionally left blank.
Model Performance Evaluation for Ozone of
the CAMx 6.40 Modeling System and the
VISTAS II 2011 Updated Modeling Platform
(Task 8.0)
Prepared for:
Southeastern States Air Resource Managers, Inc.
(SESARM)
205 Corporate Center Dr., Suite D
Stockbridge, GA 30281-7383
Under Contract No. V-2018-03-01
Prepared by:
Alpine Geophysics, LLC
387 Pollard Mine Road
Burnsville, NC 28714
and
Eastern Research Group, Inc.
1600 Perimeter Park Dr., Suite 200
Morrisville, NC 27560
Final – August 17, 2020
Alpine Project Number: TS-527
ERG Project Number: 4133.00.006
Model Performance Evaluation – Ozone
August 17, 2020 i
This page is intentionally blank.
Model Performance Evaluation – Ozone
August 17, 2020 ii
Contents
Page
1.0 Introduction .......................................................................................................................1
2.0 Results ...............................................................................................................................7
2.1 Performance Statistics by State and Month ..........................................................8
2.2 Spatial Performance Evaluation ..........................................................................10
2.3 Time Series Plots by Monitor .............................................................................14
2.4 Concentration Correlation Plots ..........................................................................20
3.0 Summary .........................................................................................................................27
Appendix A Model Performance Statistics for MDA8 Ozone at Individual Monitoring Sites
Based on Days with Observed Values ≥ 60 ppb
Model Performance Evaluation – Ozone
August 17, 2020 iii
This page is intentionally blank.
Model Performance Evaluation – Ozone
August 17, 2020 iv
TABLES
Table 2-1. Performance Statistics for MDA8 Ozone ≥ 60 ppb by Month for VISTAS States
Based on Data at AQS Network Sites..................................................................................8
Table 2-2. Performance Statistics for MDA8 Ozone ≥ 60 ppb by Month and VISTAS State
Within VISTAS12 Domain Based on Data at AQS Network Sites. ....................................9
Table 2-3. Monitoring Sites Included in the Ozone Time Series Analysis. ..................................15
FIGURES
Figure 1-1. VISTAS12 Modeling Domain. Areas in green denote Class I Areas. ..........................4
Figure 2-1. Mean Bias (ppb) of MDA8 Ozone ≥ 60 ppb Over the Period May-September
2011 at AQS Monitoring Sites in VISTAS12 Domain. .....................................................11
Figure 2-2. Normalized Mean Bias (%) of MDA8 Ozone ≥ 60 ppb Over the Period May-
September 2011 at AQS Monitoring Sites in VISTAS12 Domain....................................12
Figure 2-3. Mean Error (ppb) of MDA8 Ozone ≥ 60 ppb Over the Period May-September
2011 at AQS Monitoring Sites in VISTAS12 Domain. .....................................................13
Figure 2-4. Normalized Mean Error (%) of MDA8 Ozone ≥ 60 ppb Over the Period May-
September 2011 at AQS Monitoring Sites in VISTAS12 Domain....................................14
Figure 2-5. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 010731005 in Alabama. .................................................16
Figure 2-6. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 121130015 in Florida. ....................................................16
Figure 2-7. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 131210055 in Georgia. ...................................................17
Figure 2-8. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 211110051 in Kentucky. ................................................17
Figure 2-9. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 280470008 in Mississippi...............................................18
Figure 2-10. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 371190041 in North Carolina. ........................................18
Figure 2-11. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 450830009 in South Carolina. ........................................19
Figure 2-12. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 470090101 in Tennessee. ...............................................19
Model Performance Evaluation – Ozone
August 17, 2020 v
Figure 2-13. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 510590030 in Virginia. ..................................................20
Figure 2-14. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 540690010 in West Virginia. .........................................20
Figure 2-15. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 010731005 in Alabama. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. ................21
Figure 2-16. Correlation of Observed and Predicted MDA8 Ozone for May Through
September 2011 at Site 121130015 in Florida. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. ................22
Figure 2-17. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 131210055 in Georgia. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. ................22
Figure 2-18. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 211110051 in Kentucky. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. ................23
Figure 2-19. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 280470008 in Mississippi. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. ................23
Figure 2-20. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 371190041 in North Carolina. The Red Square Indicates 4th
High Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. .......24
Figure 2-21. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 450830009 in South Carolina. The Red Square Indicates 4th
High Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. .......24
Figure 2-22. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 470090101 in Tennessee. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. ................25
Figure 2-23. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 510590030 in Virginia. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. ................25
Figure 2-24. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 540690010 in West Virginia. The Red Square Indicates 4th
High Observed Value and the Yellow Diamond Indicates 4th High Modeled Value. .......26
Model Performance Evaluation – Ozone
August 17, 2020 vi
Abbreviations/Acronym List
Alpine Alpine Geophysics, LLC
AQS Air Quality Subsystem
CAMx Comprehensive Air quality Model with eXtensions
ERG Eastern Research Group, Inc.
EGU Electric Generating Unit
EPA United States Environmental Protection Agency
FL Florida
GA Georgia
km Kilometer
MAPS Model Performance Evaluation, Analysis, and Plotting Software
MB Mean Bias
MDA8 Maximum Daily 8-Hour
ME Mean Error
MPE Model Performance Evaluation
MS Mississippi
n Number of observations
NC North Carolina
NMB Normalized Mean Bias
NME Normalized Mean Error
OSAT Ozone Source Apportionment Technology
ppb parts per billion
PSAT Particulate Source Apportionment Technology
R2 Pearson correlation coefficient, squared
RADM-AQ Regional Acid Deposition Model – aqueous chemistry
SC South Carolina
SESARM Southeastern States Air Resource Managers, Inc.
SIPS State Implementation Plans
SOA Secondary organic aerosol
SOAP Secondary organic aerosol partitioning
TN Tennessee
VA Virginia
VISTAS Visibility Improvement – State and Tribal Association of the Southeast
WI Wisconsin
WV West Virginia
Model Performance Evaluation – Ozone
August 17, 2020 vii
This page is intentionally blank.
Model Performance Evaluation – Ozone
August 17, 2020 1
1.0 INTRODUCTION
Southeastern States Air Resource Managers, Inc. (SESARM) has been designated by the
United States Environmental Protection Agency (EPA) as the entity responsible for coordinating
regional haze evaluations for the ten Southeastern states of Alabama, Florida, Georgia,
Kentucky, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia.
The Eastern Band of Cherokee Indians and the Knox County, Tennessee local air pollution
control agency are also participating agencies. These parties are collaborating through the
Regional Planning Organization known as Visibility Improvement - State and Tribal Association
of the Southeast (VISTAS) in the technical analyses and planning activities associated with
visibility and related regional air quality issues. VISTAS analyses will support the VISTAS
states in their responsibility to develop, adopt, and implement their State Implementation Plans
(SIPs) for regional haze.
The state and local air pollution control agencies in the Southeast are mandated to protect
human health and the environment from the impacts of air pollutants. They are responsible for
air quality planning and management efforts including the evaluation, development, adoption,
and implementation of strategies controlling and managing all criteria air pollutants including
fine particles and ozone as well as regional haze. This project will focus on regional haze and
regional haze precursor emissions. Control of regional haze precursor emissions will have the
additional benefit of reducing criteria pollutants as well.
The 1999 Regional Haze Rule (RHR) identified 18 Class I Federal areas (national parks
greater than 6,000 acres and wilderness areas greater than 5,000 acres) in the VISTAS region.
The 1999 RHR required states to define long-term strategies to improve visibility in Federal
Class I national parks and wilderness areas. States were required to establish baseline visibility
conditions for the period 2000-2004, natural visibility conditions in the absence of anthropogenic
influences, and an expected rate of progress to reduce emissions and incrementally improve
visibility to natural conditions by 2064. The original RHR required states to improve visibility on
the 20% most impaired days and protect visibility on the 20% least impaired days.1 The RHR
1 RHR summary data is available at: http://vista.cira.colostate.edu/Improve/rhr-summary-data/
Model Performance Evaluation – Ozone
August 17, 2020 2
requires states to evaluate progress toward visibility improvement goals every five years and
submit revised SIPs every ten years.
To demonstrate progress toward the improvement goals, the SESARM partners modeled
visibility and air quality conditions for a base year of 2011 and future year of 2028. The
SESARM VISTAS II Regional Haze modeling analysis was performed by the contractor team
Eastern Research Group, Inc. (ERG) and Alpine Geophysics, LLC (Alpine). The preparation and
modeling were conducted over several contract tasks, including emission inventory development,
ambient data collection, Comprehensive Air quality Model with extensions (CAMx) modeling,
and model performance evaluation (MPE) of the base year. The VISTAS II modeling included
particulate matter simulations and source apportionment studies using the 12-kilometer (km) grid
based on EPA’s 2011/2028el modeling platform and preliminary source contribution
assessment,2 updated to include a 12-km subdomain over the VISTAS region and augmented
with revisions to electric generating unit (EGU) and non-EGU point source projections. The air
quality modeling was conducted using CAMx. A detailed description of the modeling platform
can be found in the Task 6 modeling report.
Under Task 8 of the Regional Haze Modeling for Southeastern VISTAS II Regional Haze
Analysis Project, a thorough MPE was conducted for Maximum Daily 8-Hour (MDA8) ozone
concentrations to examine the ability of the CAMx v6.40 modeling system to simulate 2011
measured concentrations. This report documents the MPE for the base year CAMx modeling.
The VISTAS II modeling for 2011 is based on the EPA modeling conducted for Regional
Haze Analysis, sometimes referred to as the “2011el” modeling. Updates to the EPA platform in
the VISTAS II modeling include updating the version of CAMx from version 6.32 to 6.40. Many
updates to the CAMx model were implemented between the 6.32 and 6.40 release. According to
the CAMx 6.40 release notes, the significant changes included:
1. Updates to the chemistry to include a condensed halogen mechanism for ocean-borne
inorganic reactive iodine, hydrolysis of isoprene-derived organic nitrate and SO2
2 EPA. 2017. Documentation for the EPA’s Preliminary 2028 Regional Haze Modeling. U.S. Environmental Protection
Agency, Office of Air Quality Planning and Standards. October. Available at:
https://www3.epa.gov/ttn/scram/reports/2028_Regional_Haze_Modeling-TSD.pdf.
Model Performance Evaluation – Ozone
August 17, 2020 3
oxidation on primary crustal fine PM. This update includes the changes to the Ozone and
Particulate Source Apportionment Technology (OSAT/PSAT) algorithms;
2. Inclusion of in-line inorganic iodine emissions to support halogen chemical mechanisms;
3. A major revision to the Secondary organic aerosol partitioning (SOAP) and secondary
organic aerosol (SOA) chemistry algorithm;
4. Updates to the Regional Acid Deposition Model aqueous chemistry (RADM-AQ)
algorithm; and
5. A major revision to the wet deposition algorithm to identify assumptions or processes
that were unintentionally or otherwise unreasonably limiting gas and PM update into
precipitation. The wet deposition algorithm was simplified and improved in several ways,
resulting in the increased scavenging of gases and PM.
In addition to the model version, the CAMx 6.32 and 6.40 simulations contained
differences in the EPA modeling platform that had been made subsequent to the 2011el/2028el
model release. In the most current 2023en simulation, EPA developed new photolysis rates and
ozone column data. These updates were included in the updated modeling platform and resulting
CAMx 6.40 simulation and were used in the VISTAS II 2011el (hereafter “VISTAS12”)
simulations.
Figure 1-1 presents the VISTAS12 modeling domain.
Model Performance Evaluation – Ozone
August 17, 2020 4
Figure 1-1. VISTAS12 Modeling Domain. Areas in green denote Class I Areas.
An operational model evaluation was conducted for the 2011 base year CAMx v6.40
simulation performed for the VISTAS12 modeling domain defined by SESARM and shown in
Figure 1-1. The purpose of this evaluation is to examine the ability of this 2011 air quality
modeling platform to represent the magnitude and spatial and temporal variability of measured
(i.e., observed) ozone concentrations within the modeling domain. The evaluation presented here
Model Performance Evaluation – Ozone
August 17, 2020 5
is based on model simulations using the 2011 emissions platform (i.e., scenario name
2011el.ag.v6_40.vistas12). This model evaluation for ozone focuses on comparisons of model
predicted 8-hour daily maximum concentrations to the corresponding observed data at
monitoring sites in the EPA Air Quality System (AQS).
Included in the evaluation are statistical measures of model performance based upon
model-predicted versus observed concentrations that were paired in space and time. Model
performance statistics were calculated for several spatial scales and temporal periods. Statistics
were calculated for individual monitoring sites, and in aggregate for monitoring sites within
states of the 12-km modeling domain.
For MDA8 ozone, model performance statistics were created for the periods May through
September. The aggregated statistics by state and VISTAS region as a whole are presented in this
document. Model performance statistics for MDA8 ozone at individual monitoring sites based on
days with observed values ≥ 60 ppb can be found as Appendix A to this document.
In addition to the above performance statistics, we prepared several graphical
presentations of model performance for MDA8 ozone. These graphical presentations include:
1. spatial maps that show the mean bias and error as well as normalized mean bias and error
calculated for MDA8 ≥ 60 ppb for May through September at individual AQS monitoring
sites within the VISTAS12 modeling domain;
2. time series plots (May through September) of observed and predicted MDA8 ozone
concentrations for select sites from each VISTAS state located within the VISTAS12
modeling domain; and
3. scatter plots (May through September) that show the correlation of the predicted and
observed MDA8 ozone concentrations for select sites from each VISTAS state located
within the VISTAS12 modeling domain.
The Model Performance Evaluation, Analysis, and Plotting Software (MAPS) tool was
used to calculate the model performance statistics used in this document.3 For this evaluation we
have selected the mean bias, mean error, normalized mean bias, and normalized mean error to
characterize model performance, statistics which are consistent with the recommendations in
3 McNally, D. and T. W. Tesche. 1993. Model Performance Evaluation, Analysis, and Plotting Software (MAPS). Alpine
Geophysics, LLC. Arvada, CO.
Model Performance Evaluation – Ozone
August 17, 2020 6
Simon et al. (2012),4 the photochemical modeling guidance (U.S. EPA, 2018),5 and EPA’s recent
performance evaluation of the 2011en platform (EPA, 2018).
Mean bias (MB) is the average difference between predicted (P) and observed (O)
concentrations for a given number of samples (n):
𝑀𝐵(𝑝𝑝𝑏)= 1
𝑛∑(𝑀𝑖−𝑀𝑖)
𝑛
𝑖=1
Mean error (ME) is the average absolute value of the difference between predicted and
observed concentrations for a given number of samples:
𝑀𝐸(𝑝𝑝𝑏)= 1
𝑛∑|𝑀𝑖−𝑀𝑖|
𝑛
𝑖=1
Normalized mean bias (NMB) is the sum of the difference between predicted and
observed values divided by the sum of the observed values:
𝑀𝑀𝐵(%)= ∑(𝑀−𝑀)𝑛
1
∑(𝑀)𝑛
1
∗100
Normalized mean error (NME) is the sum of the absolute value of the difference between
predicted and observed values divided by the sum of the observed values:
𝑀𝑀𝐸(%)= ∑|𝑀−𝑀|𝑛
1
∑(𝑀)𝑛
1
∗100
As described in more detail below, the model performance statistics indicate that the 8-
hour daily maximum ozone concentrations predicted by the VISTAS12 modeling platform
closely reflect the corresponding 8-hour observed ozone concentrations in each region in the
modeling domain. The acceptability of model performance was judged by considering the 2011
CAMx performance results in light of the range of performance found in recent regional ozone
4 Simon, H., K. Baker and S. Phillips. 2012. Compilations and Interpretation of Photochemical Model Performance Statistics
Published between 2006 and 2012. Atmos. Env. 61 (2012) 124-139. December.
5 EPA, 2018. Modeling Guidance for Demonstrating Air Quality Goals for Ozone, PM2.5, and Regional Haze. Internet address:
https://www3.epa.gov/ttn/scram/guidance/guide/O3-PM-RH-Modeling_Guidance-2018.pdf
Model Performance Evaluation – Ozone
August 17, 2020 7
model applications.5,6,7,8,9,10,11,12 These other modeling studies represent a wide range of
modeling analyses that cover various models, model configurations, domains, years and/or
episodes, chemical mechanisms, and aerosol modules.
Overall, the ozone model performance results for the VISTAS12 modeling are within the
range found in other recent peer-reviewed and regulatory applications. The model performance
results, as described in this document, demonstrate that the predictions from the VISTAS12
modeling domain using the EPA’s 2011el modeling platform corresponds closely to observed
concentrations in terms of the magnitude, temporal fluctuations, and geographic differences for
8-hour daily maximum ozone.
2.0 RESULTS
The 8-hour ozone model performance bias and error statistics for the months of May
through September for the region and VISTAS states in the VISTAS12 modeling domain are
provided in Tables 2-1 and 2-2, respectively. The statistics shown in Tables 2-1 and 2-2 were
calculated using data pairs on days with observed 8-hour ozone of ≥ 60 ppb. Spatial plots of the
mean bias and error as well as the normalized mean bias and error for individual monitors are
shown in Figures 2-3 through 2-6. Time series plots of observed and predicted MDA 8-hour
ozone during the period May through September at select sites listed in Table 5 are provided in
Figures 2-1 through 2-16. The correlations of observed and predicted 8-hour ozone by month in
the period of May through September for each region are shown in Figures 2-17 through 2-27.
6 NRC, 2002. National Research Council (NRC), 2002. Estimating the Public Health Benefits of Proposed Air Pollution
Regulations, Washington, DC: National Academies Press.
7 Phillips et al., 2007. Phillips, S., K. Wang, C. Jang, N. Possiel, M. Strum, T. Fox, 2007. Evaluation of 2002 Multi-pollutant
Platform: Air Toxics, Ozone, and Particulate Matter, 7th Annual CMAS Conference, Chapel Hill, NC, October 6 -8,
2008.
8 EPA. 2005. Guidance on the Use of Models and Other Analyses in Attainment Demonstrations for the 8-hr Ozone NAAQS --
Final. U.S. Environmental Protection Agency, Atmospheric Sciences Modeling Division, Research Triangle Park, N.C.
October.
9 EPA. 2009. U.S. Environmental Protection Agency, Proposal to Designate an Emission s Control Area for Nitrogen Oxides,
Sulfur Oxides, and Particulate Matter: Technical Support Document. EPA-420-R-007.
10 EPA. 2010. U.S. Environmental Protection Agency, 2010, Renewable Fuel Standard Program (RFS2) Regulatory Impact
Analysis. EPA-420-R-10-006. February 2010. Sections 3.4.2.1.2 and 3.4.3.3. Docket EPA -HQ-OAR-2009-0472-
11332.
11 EPA. 2016. Air Quality Modeling Technical Support Document for the 2015 Ozone NAAQS Preliminary Interstate Transport
Assessment. U.S. Environmental Protection Agency, Office of Air Quality Planning and Standards. December 2016 .
12 EPA. 2018. Air Quality Modeling Technical Support Document for the Updated 2023 Projected Ozone Design Values. Office
of Air Quality Planning and Standards, United States Environmental Protection Agency. June 2018.
Model Performance Evaluation – Ozone
August 17, 2020 8
Overall, model performance for MDA8 ozone concentrations for the VISTAS12
modeling is similar to what was found in EPA’s model performance evaluation conducted for the
EPA’s 2011en CAMx v6.40 simulation performed in support of the 2008 and 2015 ozone
NAAQS reviews.11
2.1 Performance Statistics by State and Month
As indicated by the statistics in Table 2-1, bias and error for 8-hour daily maximum
ozone are relatively low in the region. Generally, mean bias (MB) for 8-hour ozone ≥ 60 ppb
during each month of the May through September period, demonstrating within ±5 ppb at AQS
sites in VISTAS states, ranging from -0.13 ppb (September) to 3.79 ppb (July). The mean error
(ME) is less than 10 ppb in all months. Normalized mean bias (NMB) is within ±5 percent for
AQS sites in all months except July (5.63%). The mean bias and normalized mean bias statistics
indicate a tendency for the model to overpredict MDA8 ozone concentrations in month of May
through August and slightly underpredict MDA8 ozone concentrations in September for AQS
sites. The normalized mean error (NME) is less than 15 percent in the region across all months.
Table 2-1. Performance Statistics for MDA8 Ozone ≥ 60 ppb by
Month for VISTAS States Based on Data at AQS Network Sites.
Region Month
# of
Obs
MB
(ppb)
ME
(ppb)
NMB
(%)
NME
(%)
VISTAS May 838 2.48 6.11 3.79 9.34
VISTAS Jun 2028 1.73 7.11 2.57 10.55
VISTAS Jul 1233 3.79 8.88 5.63 13.21
VISTAS Aug 1531 2.38 6.94 3.59 10.48
VISTAS Sep 681 -0.13 6.09 -0.19 9.08
Looking at 12-km model performance for individual states located within the VISTAS12
domain (Table 2-2) indicates that mean bias is within ± 5 ppb for the majority of the months and
states for all but July in Alabama (6.18ppb), July in Florida (-5.32 ppb), August in Georgia (5.67
ppb), July in Kentucky (5.04 ppb), May in Virginia (5.57 ppb), and July in West Virginia (5.27
ppb). The mean error is less than 10 ppb for nearly all months and states, with exceptions
occurring in July (Alabama, Florida, and Georgia) and August (Florida). The normalized mean
bias is within ±10 percent in all months and states. The normalized mean error is within 15
Model Performance Evaluation – Ozone
August 17, 2020 9
percent for all months and states with again exceptions occurring in July (Alabama, Florida, and
Georgia) and August (Florida).
Table 2-2. Performance Statistics for MDA8 Ozone ≥ 60 ppb by
Month and VISTAS State Within VISTAS12 Domain Based on Data
at AQS Network Sites.
Month # of Obs
MB
(ppb)
ME
(ppb)
NMB
(%)
NME
(%)
Alabama
May 75 2.55 4.89 3.89 7.47
June 235 3.30 7.53 4.95 11.29
July 83 6.18 10.64 9.12 15.71
August 241 3.56 6.77 5.30 10.09
September 80 1.67 5.83 2.61 9.11
Florida
May 241 2.47 6.44 3.72 9.72
June 137 1.23 7.59 1.83 11.30
July 20 -5.32 14.73 -8.21 22.74
August 62 3.17 10.49 4.74 15.67
September 78 0.98 7.52 1.48 11.40
Georgia
May 130 3.91 5.87 5.85 8.78
June 251 2.07 8.43 3.05 12.41
July 111 2.89 11.09 4.19 16.06
August 218 5.67 7.95 8.44 11.84
September 97 1.22 5.03 1.81 7.48
Kentucky
May 25 3.93 6.03 6.30 9.66
June 227 0.68 6.86 1.03 10.37
July 170 5.04 9.83 7.57 14.76
August 167 -0.32 7.30 -0.49 11.03
September 78 -0.82 6.60 -1.20 9.62
Mississippi
May 33 -2.97 5.50 -4.49 8.30
June 64 1.38 8.80 2.07 13.23
July 24 2.42 8.18 3.74 12.64
August 74 2.25 9.02 3.39 13.60
September 37 2.19 8.12 3.35 12.39
North Carolina
May 117 4.44 6.52 6.99 10.27
June 473 2.36 6.22 3.46 9.12
July 299 4.29 7.41 6.39 11.03
August 257 2.68 5.65 4.10 8.66
Model Performance Evaluation – Ozone
August 17, 2020 10
Table 2-2. Performance Statistics for MDA8 Ozone ≥ 60 ppb by
Month and VISTAS State Within VISTAS12 Domain Based on Data
at AQS Network Sites.
Month # of Obs
MB
(ppb)
ME
(ppb)
NMB
(%)
NME
(%)
September 129 -1.35 5.36 -2.00 7.96
South Carolina
May 46 3.34 4.56 5.30 7.23
June 148 0.34 5.25 0.50 7.82
July 74 0.94 7.52 1.42 11.38
August 86 2.15 6.81 3.32 10.53
September 49 -0.44 4.34 -0.66 6.56
Tennessee
May 108 -1.18 5.38 -1.82 8.32
June 237 1.98 7.96 2.93 11.77
July 158 4.28 9.39 6.41 14.08
August 295 -0.03 6.04 -0.04 9.09
September 99 -2.67 6.83 -3.87 9.91
Virginia
May 41 5.57 9.47 8.01 13.62
June 200 0.55 7.40 0.82 10.99
July 225 2.82 8.63 4.12 12.59
August 90 2.93 7.27 4.50 11.18
September 17 1.32 6.53 2.07 10.25
West Virginia
May 22 0.40 7.54 0.63 11.90
June 56 0.95 5.00 1.44 7.56
July 69 5.27 6.96 8.03 10.60
August 41 2.61 5.91 4.01 9.08
September 17 0.21 5.78 0.28 7.82
Monitor specific performance metrics for monitors in the VISTAS12 modeling domain
are provided as Appendix A to this document.
2.2 Spatial Performance Evaluation
Figures 2-1 through 2-4 show the spatial variability in bias and error at monitor locations.
Mean bias, as seen from Figure 2-1, is within ±5 ppb at most sites across the VISTAS12 domain
with a maximum under-prediction of 23.44 ppb at one site (AQS monitor 550030010) in
Ashland County, WI and a maximum over-prediction of 17.95 ppb in York County, SC (AQS
monitor 450910006); both with small sample sizes (n=1 and n=7, respectively). A positive mean
Model Performance Evaluation – Ozone
August 17, 2020 11
bias is generally seen in the range of 5 to 10 ppb with regions of 10 to 15 ppb over-prediction
seen scattered throughout the domain. The model has a tendency to underestimate in the western
portion of the domain and overestimate in the eastern portion of the domain.
Figure 2-1. Mean Bias (ppb) of MDA8 Ozone ≥ 60 ppb Over the Period May-September
2011 at AQS Monitoring Sites in VISTAS12 Domain.13
Figure 2-2 indicates that the normalized mean bias for days with observed 8-hour daily
maximum ozone ≥ 60 ppb is within ± 10 percent at the vast majority of monitoring sites across
the VISTAS12 modeling domain. Monitors in Ashland County, WI and York County, SC again
bookend the NMB range with 38.03% and 27.44%, respectively. There are regional differences
in model performance, as the model tends to overpredict at most sites in eastern region of the
13 Appendix A-2 presents the Mean Bias data in a Google Earth .kmz file to allow the user to zoom to specific geographic
regions.
Model Performance Evaluation – Ozone
August 17, 2020 12
VISTAS12 domain and generally underpredict at sites in and around the western and north
western borders of the domain.
Figure 2-2. Normalized Mean Bias (%) of MDA8 Ozone ≥ 60 ppb Over the Period May-
September 2011 at AQS Monitoring Sites in VISTAS12 Domain.14
Mean error (ME), as seen from Figure 2-3, is generally 10 ppb or less at most of the sites
across the VISTAS12 modeling domain although the Ashland, WI and York, SC monitors show
much higher ME of 23.44 and 17.95 ppb, respectively. VISTAS states show less than ten percent
of their monitors above 10 ppb model error, with the majority of those within this value. Figure
2-4 indicates that the normalized mean error (NME) for days with observed 8-hour daily
14 Appendix A-3 presents the Normalized Mean Bias data in a Google Earth .kmz file to allow the user to zoom to specific
geographic regions.
Model Performance Evaluation – Ozone
August 17, 2020 13
maximum ozone ≥ 60 ppb is less than 15 percent at the vast majority of monitoring sites across
the VISTAS12 modeling domain. Noted exceptions seen are monitors 450910006 (York County,
SC), 470370011 (Davidson County, TN), and 120713002 (Lee County, FL) with NMEs of
27.44%, 25.4%, and 23.07%, respectively. Somewhat elevated NMEs (> 15%) are seen in and
around many of the VISTAS state metro areas.
Figure 2-3. Mean Error (ppb) of MDA8 Ozone ≥ 60 ppb Over the Period May-September
2011 at AQS Monitoring Sites in VISTAS12 Domain. 15
15 Appendix A-4 presents the Mean Error data in a Google Earth .kmz file to allow the user to zoom to specific geographic
regions.
Model Performance Evaluation – Ozone
August 17, 2020 14
Figure 2-4. Normalized Mean Error (%) of MDA8 Ozone ≥ 60 ppb Over the Period May-
September 2011 at AQS Monitoring Sites in VISTAS12 Domain.16
2.3 Time Series Plots by Monitor
In addition to the above analysis of overall model performance, we also examined how
well the modeling platform replicates day to day fluctuations in observed 8-hour daily maximum
concentrations. Table 2-3 presents data for the highest 2011 3-year design value site in each
VISTAS state.
16 Appendix A-5 presents the Normalized Mean Error data in a Google Earth .kmz file to allow the user to zoom to specific
geographic regions.
Model Performance Evaluation – Ozone
August 17, 2020 15
Table 2-3. Monitoring Sites Included in the Ozone Time Series Analysis.
AQS Monitor
ID State County
2009-2011 Ozone
Design Value
(ppb)
010731005 Alabama Jefferson 75
121130015 Florida Santa Rosa 74
131210055 Georgia Fulton 80
211110051 Kentucky Jefferson 78
280470008 Mississippi Harrison 75
371190041 North Carolina Mecklenburg 79
450830009 South Carolina Spartanburg 74
470090101 Tennessee Blount 76
510590030 Virginia Fairfax 82
540690010 West Virginia Ohio 73
For this site-specific analysis we present the time series of observed and predicted 8-hour
daily maximum concentrations by site in the 12-km simulation over the period of May through
September. The results, as shown in Figures 2-5 through 2-14, indicate that the modeling
platform generally replicates the day-to-day variability in ozone during this time period at these
sites. That is, days with high modeled concentrations are generally also days with high measured
concentrations and, conversely, days with low modeled concentrations are also days with low
measured concentrations in most cases.
For example, model predictions at several sites not only accurately capture the day-to-day
variability in the observations, but also appear to have relatively low bias on individual days.
Santa Rosa County, FL and Harrison County, MS each track closely with the observations, but
there is a tendency to overpredict on several of the observed high ozone days at these coastal
state locations. Of particular note are the overpredictions at the Mecklenburg County, NC
monitor early in the ozone season, at the Fairfax County, VA monitor during a late season
episode, and at the Ohio County, WV monitor mid-season and the underprediction of MDA8 at
the Fulton County, GA monitor during an early ozone season episode and multiple days at the
coastal monitors in Florida and Mississippi.
Model Performance Evaluation – Ozone
August 17, 2020 16
Figure 2-5. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 010731005 in Alabama.
Figure 2-6. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 121130015 in Florida.
Model Performance Evaluation – Ozone
August 17, 2020 17
Figure 2-7. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 131210055 in Georgia.
Figure 2-8. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 211110051 in Kentucky.
Model Performance Evaluation – Ozone
August 17, 2020 18
Figure 2-9. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 280470008 in Mississippi.
Figure 2-10. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 371190041 in North Carolina.
Model Performance Evaluation – Ozone
August 17, 2020 19
Figure 2-11. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 450830009 in South Carolina.
Figure 2-12. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 470090101 in Tennessee.
Model Performance Evaluation – Ozone
August 17, 2020 20
Figure 2-13. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 510590030 in Virginia.
Figure 2-14. Time Series of Observed (Green) and Predicted (Red) MDA8 Ozone for May
through September 2011 at Site 540690010 in West Virginia.
2.4 Concentration Correlation Plots
Under and overpredictions can also be reviewed through examination of correlation plots
of observed vs. modeled MDA8 concentrations by location during the May through September
Model Performance Evaluation – Ozone
August 17, 2020 21
episode (Figures 2-15 through 2-24). On these graphics each daily MDA8 concentration at a
monitor is plotted as a single ordered pair with the observed ozone on the horizontal axis and the
corresponding model estimate on the vertical axis. A perfect model would show all points in a
single line with a unit slope and a y-axis intercept of zero. In the figures the fourth highest
observation is plotted with a red square and the fourth highest model estimate has a yellow
diamond.
While many of the sites generally track well and capture day-to-day variability, the
presented sites demonstrate a modeled over estimation of ozone on most days with medium
range ozone concentrations (40-60 ppb). At all monitors presented here, except in Blount
County, TN, the model has overpredicted the 4th high observed values (yellow diamond).
Figure 2-15. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 010731005 in Alabama. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value.
Model Performance Evaluation – Ozone
August 17, 2020 22
Figure 2-16. Correlation of Observed and Predicted MDA8 Ozone for May Through
September 2011 at Site 121130015 in Florida. The Red Square Indicates 4th High Observed
Value and the Yellow Diamond Indicates 4th High Modeled Value.
Figure 2-17. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 131210055 in Georgia. The Red Square Indicates 4th High Observed
Value and the Yellow Diamond Indicates 4th High Modeled Value.
Model Performance Evaluation – Ozone
August 17, 2020 23
Figure 2-18. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 211110051 in Kentucky. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value.
Figure 2-19. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 280470008 in Mississippi. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value.
Model Performance Evaluation – Ozone
August 17, 2020 24
Figure 2-20. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 371190041 in North Carolina. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value.
Figure 2-21. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 450830009 in South Carolina. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value.
Model Performance Evaluation – Ozone
August 17, 2020 25
Figure 2-22. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 470090101 in Tennessee. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value.
Figure 2-23. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 510590030 in Virginia. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value.
Model Performance Evaluation – Ozone
August 17, 2020 26
Figure 2-24. Correlation of Observed and Predicted MDA8 Ozone for May through
September 2011 at Site 540690010 in West Virginia. The Red Square Indicates 4th High
Observed Value and the Yellow Diamond Indicates 4th High Modeled Value.
Model Performance Evaluation – Ozone
August 17, 2020 27
3.0 SUMMARY
As was seen with the 12-km evaluation conducted by EPA on the 2011en platform,11 the
VISTAS12 modeling has better skill at predicting ozone concentrations in the mid-range of 40 to
60 ppb than it does at the tail ends of the concentration curves. Additionally, as noted above and
demonstrated with the statistics and figures of this analysis, the model tends to overestimate
ozone concentrations across all ranges and at all presented monitors. It is also noted that
compared to observed concentrations, the model overestimates less at high-end concentrations
(greater than 60 ppb) than at low-end observed concentrations (less than 40 ppb).
Over the entire concentration range, the model tends to overpredict MDA8 ozone in the
VISTAS12 domain. However, looking across all represented monitors in the domain, we note
that the model is able to capture site-to-site differences in the short-term (i.e., day-to-day)
variability and the general magnitude of the observed ozone concentrations for the May through
September 2011 episode.
As a result, and compared to similar results from comparable studies, we find that the
predictions from the 12-km domain using this configuration of the 2011el modeling platform
correspond closely to observed concentrations in terms of the magnitude, temporal fluctuations,
and geographic differences for 8-hour daily maximum ozone.
Thus, the model performance results demonstrate the scientific credibility of the
VISTAS12 modeling. These results provide confidence in the ability of the modeling platform to
be used for future year ozone concentration projections and contribution analyses.
Model Performance Evaluation – Ozone
August 17, 2020 28
This page is intentionally left blank.
Model Performance Evaluation – Ozone
August 17, 2020
Appendix A
Model Performance Statistics for MDA8 Ozone at Individual Monitoring Sites Based on Days
with Observed Values ≥ 60 ppb
(see: AppendixA1-OzoneMPEbyStation.xlsx
AppendixA2-MeanBias.kmz
AppendixA3-NormalizedMeanBias.kmz
AppendixA4-MeanError.kmz
AppendixA5-NormalizedMeanError.kmz)
Model Performance Evaluation – Ozone
August 17, 2020
This page is intentionally left blank.