QAPP Neuse River Watershed Model with Signatures

respec.com

NEUSE RIVER WATERSHED MODEL QUALITY ASSURANCE PROJECT PLAN
QAPP-21

PREPARED BY
Paul B. Duda Cindie M. Kirby
RESPEC 3824 Jet Drive Rapid City, South Dakota 57703
PREPARED FOR
North Carolina Department of Environmental Quality Division of Water Resources 217 West Jones Street Raleigh, North Carolina 27603
MARCH 2023
Project Number W0392.22001.001

QAPP-21
i

QUALITY ASSURANCE PROJECT PLAN GOVERNANCE
This quality assurance project plan (QAPP) has been prepared according to U.S. Environmental
Protection Agency (EPA) guidance provided in the Template for Developing a Generic or Project-
Specific Quality Assurance Project Plan for Model Applications [EPA, 2009a]1 as well as
EPA Requirements for Quality Assurance Project Plans [EPA, 2002a]2 and Guidance for Quality
Assurance Project Plans for Modeling [EPA, 2002b]3 to ensure that environmental and related data
collected, compiled, and/or generated for this project are complete; accurate; and of the type, quantity,
and quality required for their intended use.
The DWR Modeling & Assessment Branch – Process Guidance for Third Party Model QAPP
Development by the North Carolina Department of Environmental Quality [2020]4 has been used to
ensure that the contents of this QAPP meet the specific content expectations of the North Carolina
Department of Environmental Quality Division of Water Resources. While the sequence of topics listed
in the NC DEQ document are different from those listed in the EPA guidance for QAPPs, all the
information is included. The following list provides a cross reference for finding information required as
per the North Carolina Department of Environmental Quality document:
/ I. Project Goals: Section 1.5 Problem Definition and Background, Section 1.6 Project/Task
Description and Schedule
/ II. Project Organization: Section 1.4 Project Organization
/ III. Modeling Plan: Section 2.0 Modeling Approach and Chapter 5.0 Model Application discusses
model parameterization, calibration/validation, performance targets, and postprocessing tools
/ IV. Data Management: Section 3.2 Data Management
/ V. Model Review: Section 4.3 Model Review
/ VI. Quality Assurance: Section 1.7 Quality Objectives and Criteria for Measurement Data and
Models, Chapters 3.0 Data Acquisition/Management, 4.0 Assessment/Oversight, and 5.0 Model
Application address quality assurance, although it is addressed throughout this QAPP
/ VII. Schedule: Section 1.6 Project/Task Description and Schedule
1 U.S. Environmental Protection Agency, 2009a. Template for Developing a Generic or Project-Specific Quality
Assurance Project Plan for Model Applications, prepared by the U.S. Environmental Protection Agency, Office of Environmental Information, Washington, DC. Available online at https://www.epa.gov/sites/production/files/2015-07/modelqapptemplate2009.doc
2 U.S. Environmental Protection Agency, 2002a. EPA Requirements for Quality Assurance Project Plans, EPA QA/R-5, EPA/240/B-01/003, prepared by the U.S. Environmental Protection Agency, Office of Environmental Information, Washington, DC. Available online at https://www.epa.gov/quality/guidance-quality-assurance-project-plans-epa-qag-5
3 U.S. Environmental Protection Agency, 2002b. Guidance for Quality Assurance Project Plans for Modeling,
EPA 240-R-02-007, prepared by the U.S. Environmental Protection Agency, Office of Environmental Information, Washington, DC. Available online at https://www.epa.gov/sites/production/files/2015-06/documents/g5m-final.pdf
4 Behm, P., 2020. DWR Modeling & Assessment Branch – Process Guidance for Third Party Model QAPP Development, email from P. Behm, North Carolina Department of Environmental Quality, Raleigh, NC, to S. Kenner, RESPEC, Rapid City, SD, November 8.

QAPP-21
ii

Neuse River Watershed Model
Quality Assurance Project Plan
Title and Approval Page
Neuse River Watershed Model
(Project Name)

RESPEC Company, LLC
(Responsible Agency)

March 21, 2023
(Date)
RESPEC Principal-in-Charge; Signature
Name/Date Russell Persyn
RESPEC Project Manager Signature
Name/Date Seth Kenner
RESPEC Project QA/QC Officer Signature
Name/Date Paul Duda
NC DEQ Project Team Leader Signature
Name/Date Pam Behm

QAPP-21
iii

TABLE OF CONTENTS
1.0 PROJECT MANAGEMENT ................................................................................................................................................ 1
1.1 TITLE AND APPROVAL PAGE............................................................................................................................................................... 1
1.2 TABLE OF CONTENTS ........................................................................................................................................................................... 1
1.3 DISTRIBUTION LIST ............................................................................................................................................................................... 1
1.4 PROJECT ORGANIZATION ................................................................................................................................................................... 1
1.5 PROBLEM DEFINITION AND BACKGROUND................................................................................................................................... 3
1.6 PROJECT/TASK DESCRIPTION AND SCHEDULE ........................................................................................................................... 4
1.6.1 Compile and Preprocess Data and Information to Support Model Development ................................................. 6
1.6.2 Develop a Watershed Model of the Neuse River Watershed ....................................................................................... 7
1.6.3 Apply Model to Establish Load Estimates ........................................................................................................................ 7
1.6.4 Deliver Model and Documentation .................................................................................................................................... 8
1.7 QUALITY OBJECTIVES AND CRITERIA FOR MEASUREMENT DATA AND MODELS .............................................................. 8
1.7.1 Data Acceptance Criteria ..................................................................................................................................................... 8
1.7.2 Model Performance and Acceptance Criteria ................................................................................................................. 10
1.8 SPECIAL TRAINING REQUIREMENTS/CERTIFICATION ................................................................................................................ 12
1.9 DOCUMENTS AND RECORDS ............................................................................................................................................................. 12
2.0 MODELING APPROACH ................................................................................................................................................... 14
2.1 MODEL GEOGRAPHIC SCOPE ............................................................................................................................................................ 14
2.2 TEMPORAL SCOPE ................................................................................................................................................................................ 14
2.3 MODEL ENDPOINTS .............................................................................................................................................................................. 14
2.4 MODEL SELECTION AND JUSTIFICATION ...................................................................................................................................... 14
2.5 MODEL-DATA NEEDS ........................................................................................................................................................................... 15
2.5.1 Meteorological Sources ........................................................................................................................................................ 15
2.5.2 Point Sources ........................................................................................................................................................................... 15
2.5.3 Other External Time-Series Data ........................................................................................................................................ 16
2.5.4 Watershed Characteristics ................................................................................................................................................... 16
2.5.5 Impervious Land Classification ........................................................................................................................................... 16
2.5.6 Municipal Separate Storm Sewer Systems ...................................................................................................................... 16
2.5.7 Septic Systems ........................................................................................................................................................................ 17
2.5.8 Agricultural Data ..................................................................................................................................................................... 17
2.5.9 Elevation and Slope................................................................................................................................................................ 18
2.5.10 Reach Properties .................................................................................................................................................................... 18
2.5.11 Drainage.................................................................................................................................................................................... 18
2.5.12 Observed/Calibration Data................................................................................................................................................... 18
2.6 DATA GAPS .............................................................................................................................................................................................. 18

QAPP-21
iv

TABLE OF CONTENTS (CONTINUED)
3.0 DATA ACQUISITION/MANAGEMENT .............................................................................................................................. 20
3.1 DATA ACQUISITION ............................................................................................................................................................................... 20
3.2 DATA MANAGEMENT ............................................................................................................................................................................ 22
3.3 DATA REVIEW, VERIFICATION, AND VALIDATION ......................................................................................................................... 24
4.0 ASSESSMENT/OVERSIGHT ............................................................................................................................................. 25
4.1 ASSESSMENT AND RESPONSE ACTIONS ....................................................................................................................................... 25
4.2 REPORTS TO MANAGEMENT .............................................................................................................................................................. 26
4.3 MODEL REVIEW ...................................................................................................................................................................................... 26
5.0 MODEL APPLICATION ..................................................................................................................................................... 27
5.1 MODEL PARAMETERIZATION (CALIBRATION) ............................................................................................................................... 28
5.2 MODEL CORROBORATION (VALIDATION AND SIMULATION) ................................................................................................... 29
5.2.1 Model Performance Targets ................................................................................................................................................ 30
5.2.2 Postprocessing Tools ............................................................................................................................................................ 31
5.3 RECONCILIATION WITH USER REQUIREMENTS ........................................................................................................................... 32
6.0 PROJECT REPORTS ......................................................................................................................................................... 34
7.0 REFERENCES .................................................................................................................................................................... 35
APPENDIX A. FORMULAS USED FOR STATISTICAL COMPARISON ....................................................................................... A-1

QAPP-21
v

LIST OF TABLES
Table Page
1-1 Neuse River Watershed Model Quality Assurance Project Plan Distribution List ........................................................................... 2
1-2 Quality Assurance Project Plan Project Participants ............................................................................................................................. 3
1-3 Project Milestones and Completion Dates ............................................................................................................................................... 6
1-4 Data Acceptance Criteria for Secondary Data ......................................................................................................................................... 9
1-5 General Percent Error Calibration/Validation Targets for Watershed Models ................................................................................ 11
1-6 Example Numeric Targets for Streamflow Calibration and Validation .............................................................................................. 11
1-7 Example Numeric Targets for Water Quality Calibration and Validation .......................................................................................... 12
3-1 Secondary Environmental Data to Be Assembled for Watershed and Water Quality Modeling in the Neuse River
Watershed ......................................................................................................................................................................................................... 21
5-1 General Calibration Targets or Tolerances for the Percent Difference Between Simulated and Recorded Water
Quality Constituents for HSPF Applications ............................................................................................................................................. 31

QAPP-21
vi

LIST OF FIGURES
FIGURE Page
1-1 Organizational Chart Showing Relationships and Lines of Communication Among Project Participants .............................. 3
1-2 Neuse Watershed Model Study Area .......................................................................................................................................................... 5
5-1 R and R2 Value Ranges for Model Performance ...................................................................................................................................... 30

QAPP-21
1

1.0 PROJECT MANAGEMENT
The purpose of this document is to present the quality assurance project plan (QAPP) for conducting
modeling to support the development of the Neuse River Watershed Model. This project is funded
by the North Carolina Department of Environmental Quality (NC DEQ) through a contract with RESPEC
Company, LLC (RESPEC). The modeling effort will be performed primarily by the Rapid City, South
Dakota, office of RESPEC, with assistance from their Decatur, Georgia, office.

This QAPP provides general descriptions of the work to be performed to support the Neuse River
Watershed Model development, the standards to be met, and the procedures used to ensure that the
modeling results are scientifically valid and defensible and that uncertainty is reduced to a known and
practical minimum. This QAPP will also address using secondary and third-party data collected by
NC DEQ and federal agencies. This task order does not require collecting primary data.

This section addresses the project's administrative functions, concerns, goals, and approaches to be
followed.
1.1 TITLE AND APPROVAL PAGE
Please see page ii.
1.2 TABLE OF CONTENTS
Please see pages iii–v.
1.3 DISTRIBUTION LIST
The persons listed in Table 1-1 will receive copies of the approved QAPP and any subsequent revisions
of the QAPP. A complete copy of the original version and all revisions of the QAPP, including the official,
approved QAPP, will be maintained on file by the project quality assurance/quality control (QA/QC)
officer at RESPEC Company, LLC (RESPEC) and will be available upon request.
1.4 PROJECT ORGANIZATION
This QAPP supports the development of the Neuse River Watershed Model. The North Carolina Division
of Water Resources (NC DWR) provides the funding for this project, and successful completion requires
NC DWR involvement.

The organizational roles are as follows:
/ NC DWR: The NC DWR provides the point of contact for this project and facilitates
collaboration with RESPEC.
/ RESPEC: RESPEC is the contractor responsible for developing the Neuse River Watershed
Model.

QAPP-21
2

Table 1-1. Neuse River Watershed Model Quality Assurance Project Plan Distribution List
Name and
Title
Organization and
Contact Information
Pam Behm
Project Team Leader
North Carolina Department of Environmental Quality
Division of Water Resources
512 N. Salisbury St
1611 Mail Service Center
Raleigh, NC 27699
[e-mail]
919.707.3687
Paul Duda
RESPEC QA/QC Officer
RESPEC Company, LLC
3824 Jet Drive
Rapid City, SD 57703
[e-mail]
605.394.6400
Russell Persyn
Principal-in-Charge
RESPEC Company, LLC
407 Paris St., Suite 1
Castroville, TX 78009
[e-mail]
210.570.5962
Seth Kenner
Project Manager
RESPEC Company, LLC
3824 Jet Drive
Rapid City, SD 57703
[e-mail]
605.394.6400
RESPEC will conduct work for this project in conformance with the procedures detailed in this QAPP.
Table 1-2 identifies the project participants from the two organizations and summarizes each
participant's title, organization, and responsibility with respect to this QAPP. The project organization
and lines of communication are shown in Figure 1-1. The technical and QA/QC aspects of the project
are presented to the client and contractors.

Pam Behm, NC DWR, will provide overall project/program oversight for this study as the NC DEQ project
team leader. Ms. Behm will work with the RESPEC Project Manager, Seth Kenner, to ensure the project
objectives are attained.

The RESPEC Principal-in-Charge for this contract is Russell Persyn. He will provide senior-level
management oversight of the assigned project manager. The project manager, Seth Kenner, will
supervise the overall project, including study design and model applications.

RESPEC modeling staff will be responsible for developing the model input datasets, calibrating and
validating the model, applying the model results, and writing the final report. RESPEC will implement the
QA/QC program, complete assigned work on schedule, strictly adhere to established procedures, and
complete required documentation.

QAPP-21
3

Table 1-2. Quality Assurance Project Plan Project Participants
Name Title Organization Primary
Responsibility
Pam Behm Project Team Leader NC DWR Decision-making, coordinating with RESPEC, and overall
project management
John Huisman Stakeholder Coordinator NC DWR Coordinates stakeholder involvement
Andy Painter Data Coordinator NC DWR Coordinates data compilation from various sources
Russell Persyn Principal-in-Charge RESPEC
Directing and coordinating technical work, interacting with
the Neuse River Watershed Model project team leader,
tracking the budget, and performing administrative functions
Seth Kenner Project Manager RESPEC Managing the day-to-day project technical activities
Paul Duda QA/QC Officer RESPEC
Managing the QA/QC activities, including a targeted
independent review of the project's technical and
administrative practices and products
Chris Lupo Watershed Modeler RESPEC Developing watershed model applications
Cindie Kirby Watershed Modeler RESPEC Developing watershed model applications

Figure 1-1. Organizational Chart Showing Relationships and Lines of Communication Among Project Participants.
1.5 PROBLEM DEFINITION AND BACKGROUND
This effort is to meet the requirements of SL 2020-18 Section 15(c) and develop nutrient transport
factors for the Neuse River Watershed. This watershed model will have a nitrogen-based focus but will
also include phosphorus.

The Neuse Nutrient Strategy, implemented in 1997, addresses this problem by regulating major nutrient
pollution sources throughout the Neuse River Watershed, including wastewater, urban stormwater, and
agriculture. While the strategy has succeeded partly by stemming additional nutrient loading during
rapid population growth, the Neuse River Estuary remains impaired.

QAPP-21
4

The Neuse Nutrient Strategy is not supported by a calibrated watershed model, a standard tool for the
development and evaluation of more modern nutrient strategies. A watershed model informs regulatory
development and management decisions in many important ways, including a refined understanding of
the influence of geography or various regulatory sectors on estuarine algal blooms. This project uses a
contractor with oversight and support by the NC DWR to develop a watershed model that meets
agency standards for regulatory use and support in the Neuse River Watershed. The model will be a
core product the NC DWR staff, stakeholders, and the Environmental Management Commission will rely
upon in their continual refinement of the Neuse Nutrient Strategy rules.

This project will support long-overdue regulatory innovation that can drive systemic water quality
improvements in the Neuse River Estuary. Recreation, property enhancement, recreational and
commercial fishing, and greenways are some benefits of nutrient management. Excessive nutrient
inputs can negatively influence the estuarine ecosystem and the communities that benefit from them.
Conversely, these ecosystems and communities realize benefits from managing nutrient inputs.

A watershed model is a critical scientific tool in structuring regulatory programs to achieve these
broad-based environmental benefits, and several regulatory challenges or initiatives will be well-
informed by developing a watershed model. The Phase II TMDL, published in 2001, identified a specific
need for watershed modeling. A dynamic watershed modeling approach is the most efficient means of
obtaining detailed information on nonpoint-source and stormwater nutrient loads across the
watershed. The dynamic watershed modeling approach will also help to better understand the impact of
point sources. Directly measuring nutrient loads at the spatial and temporal scales needed would be
impossible. While simplified watershed yield models provide annual nutrient loads, these models lack
the temporal variability of loads, which is important for understanding episodic events or predicting
loads under different climatic conditions.

The project team recommends the EPA’s Hydrological Simulation Program – FORTRAN (HSPF) [Bicknell
et al., 2001] as the dynamic watershed model of choice for the Neuse River Watershed Model.
Section 2.4 presents the model selection details. HSPF has been widely used throughout the United
States to analyze water hydrology and quality in support of developing implementation plans based on
attaining environmental goals. This complex and dynamic model can address soil, groundwater, and
surface-water processes, storm events, and impacts from point- and nonpoint-sources of pollution.
The EPA and U.S. Geological Survey (USGS) continue to support the model. Figure 1-2 illustrates the
Neuse River Watershed Model study area.

Because the input data for the Neuse River Watershed Model will be obtained from other sources, the
data quality procedures for secondary data will be followed. This QAPP addresses the use of secondary
data for the Neuse River Watershed Model. The QAPP will define the project's QA/QC objectives and
the protocols that will be used by project personnel to achieve these objectives.
1.6 PROJECT/TASK DESCRIPTION AND SCHEDULE
RESPEC proposes a project approach that will achieve the NC DWR's project goals and objectives. The
primary goal of this effort is to meet the requirements of SL 2020-18 Section 15(c) by developing
nitrogen delivery factors for the Neuse River Watershed. The project will produce a calibrated

QAPP-21
5

watershed model for the development and evaluation of modern nutrient strategies. As directed by SL
2020-18, the priority of this effort is to determine delivery factors for point-source discharges and
nutrient offset credits. We will use the calibrated and validated watershed model to rigorously estimate
the proportion of end-of-pipe nutrient loading from wastewater sources that reach the Neuse River
Estuary. Since the delivery factors also play a key role in the availability and cost of nutrient trades
between wastewater facilities and from wastewater facilities to watershed treatment best management
practices (BMPs), the watershed model we develop will provide a rigorous and unbiased approach to
estimating relative nutrient contributions from the vast array of nonpoint nutrient sources throughout
the watershed. The current Neuse Nutrient Strategy seeks to reduce nonpoint-source nutrients from
cropland agriculture and new development while providing important protection through the
preservation of riparian buffers. The watershed model can be used to confirm existing nitrogen trading
schemes and to identify other nonpoint nutrient sources and their relative impacts on nutrient loading
to the Neuse River Estuary. Understanding these sources offers opportunities to develop new trading
strategies.

Figure 1-2. Neuse Watershed Model Study Area.
The scope of work entails four major tasks:
/ Compile and Preprocess Data and Information to Support Model Development
/ Develop a Watershed Model of the Neuse River Watershed
/ Apply Model to Establish Load Estimates

QAPP-21
6

/ Develop Scenario Application Manager (SAM)
/ Deliver Model, SAM, and Documentation.
Table 1-3 presents the major project milestones and completion dates. The following sections detail
each task.
Table 1-3. Project Milestones and Completion Dates
Project
Milestone
Estimated
Completion Date
Kickoff Meeting November 2022
QAPP Developed and Approved January 2023
Stakeholder Meeting January 2023
Division Meeting March 2023
Model-Data Technical Memoranda March 2023
Division Meeting May 2023
Stakeholder Meeting July 2023
Calibrated and Validated Model Application August 2023
Division Meeting September 2023
HSPF Training September 2023
Division Meeting November 2023
Scenario Application Manager November 2023
Division Meeting January 2024
Final Deliverables February 2024
1.6.1 COMPILE AND PREPROCESS DATA AND INFORMATION TO SUPPORT MODEL DEVELOPMENT
Model selection and this QAPP are part of the first task. Details of the model selection have been
provided as a separate memorandum. In addition to a QAPP, RESPEC has also developed a simulation
plan for nearly every model we have developed over the past 40 years. A simulation plan will also be
developed for this Neuse River Watershed Model project. The simulation plan will provide a detailed
roadmap for developing the model.

A primary step in the modeling process is gathering data, including temporal and spatial data, for the
project area. Data inputs will include physical characteristics of the watershed like land use, land cover,
topography, soils, hydraulic characteristic of streams/rivers, and hydrology. Also necessary are data
sets associated with flow, water quality, atmospheric deposition, and weather. Nutrient source inputs
will need to be characterized using best available information including point-source discharges,
stormwater, septic tank distributions, and agricultural operations. Data will be compiled in a consistent
format. As necessary, readily available information will be used to develop assumptions about nutrient
sources and features.

RESPEC will work with stakeholders from the Neuse River Watershed, including the Neuse River Basin
Associations, USGS, North Carolina Department of Administration (NC DOA), North Carolina
Department of Transportation (NCDOT), and other organizations to make sure we have gathered all the

QAPP-21
7

most relevant Geographic Information System (GIS), meteorological, point-source, atmospheric, flow,
and water quality data. Additionally, we will review supplemental reports and information to develop,
calibrate, and validate our HSPF application.
1.6.2 DEVELOP A WATERSHED MODEL OF THE NEUSE RIVER WATERSHED
The second task of the project is developing the watershed model, calibrating the model, evaluating the
model performance, and interpreting the modeling results.

The model construction includes developing subwatershed, reach, and land-cover segments for the
Neuse River Watershed HSPF model. The segmentation and characterization define how water travels
from the various land uses within each subwatershed to reach segments and the watershed outlet. The
developed watershed models will focus on representing nutrient export from various land surfaces and
nutrient cycling, dissolved oxygen dynamics, phytoplankton (benthic and floating), and sediment
dynamics in the receiving waters. Representing all nitrogen and phosphorous species is necessary to
properly represent the nutrient cycling processes. The model runs at an hourly timestep. Parameters
that are variable throughout the day, such as temperature and dissolved oxygen, are calibrated at an
hourly timestep if that level of data is available. Other parameters are calibrated daily.

Calibrating a watershed model is an iterative process that involves making parameter changes, running
the model, producing comparisons of simulated and observed values, and interpreting the results. This
process occurs first for the hydrology portions of the model, followed by the sediment and water
quality portions. The simulation results are changed by iteratively adjusting specific calibration
parameter values within accepted ranges until an acceptable comparison of simulated and recorded
data are achieved. Model performance and calibration/validation are evaluated through multiple
qualitative and quantitative measures that involve graphical comparisons and statistical tests.

Modeling results will be interpreted to support key management questions identified in the QAPP.
Interpretation of results will be subject to model and data limitations and the assumptions underlying
the model. Key topics for interpretation include delivery and transport of nutrients to the Neuse River
Estuary, loading from existing developed jurisdictions in the Neuse River Watershed, and
recommendations for future monitoring adjustments.
1.6.3 APPLY MODEL TO ESTABLISH LOAD ESTIMATES
The next task in this project is applying the model to establish load estimates. The Neuse River
Watershed Model will conduct loading scenario analyses for the modeled time period. Loading
estimates will be partitioned by transport zones, subwatershed, regulatory sector, and political
jurisdictions, to the extent practicable. Delivery factors will also be provided for the smallest feasible
hydrologic units, at modeled subwatershed scale, for use in permitting and trading. Delivery zones may
be consolidated in response to agency or stakeholder input to facilitate implementation.

The complexity of HSPF can be difficult for non-modeling end users in conservation, assessment,
implementation, and permitting to quickly get the necessary information from these models to apply
the model effectively for decision support. RESPEC's HSPF Scenario Application Manager (SAM) is a
user-friendly, comprehensive decision-support tool for planning and implementing targeted actions to

QAPP-21
8

restore or protect water quality. The framework of SAM includes a GIS for BMP site selection; a BMP
database with pollutant-removal efficiencies and associated costs; and scenario analysis, optimization,
and reporting capabilities. SAM is an easy-to-use desktop tool that harnesses the power of HSPF,
allowing resource professionals to efficiently plan and implement targeted actions to restore or protect
water quality in specific geographic areas.

A SAM will be developed after the HSPF model is constructed to understand loading throughout the
watershed. A SAM will be developed for the client and can be used to develop delivery factors from any
modeled source (including point sources) to any downstream subwatershed at any desired timestep.
1.6.4 DELIVER MODEL AND DOCUMENTATION
The final task is to deliver the model and documentation. Technical memoranda and results from
previous tasks will be combined into a preliminary draft report for the NC DWR to review. A subsequent
draft report incorporating the NC DWR comments will be required following the NC DWR review. The
report will include the model description, inputs and outputs; model assessment and interpretation,
model assumption and uncertainty; discussion of future model uses, including appropriate uses, users,
and adaptation; and the model's relationship to load reduction accounting methods.

The standard RESPEC HSPF model application deliverables package will be delivered, which includes
four folders: (1) a fully executable model application, (2) model results, (3) technical memoranda, and
(4) a project geodatabase. These files are provided as zipped folders through a ShareFile or similar
interface. In addition to the HSPF model package described above, a SAM project will be delivered. The
SAM package is in one folder with everything a SAM project needs to be opened and run.
1.7 QUALITY OBJECTIVES AND CRITERIA FOR MEASUREMENT DATA AND MODELS
This section describes the quality objectives for the project, including the performance and acceptance
criteria to achieve the objectives. This QAPP has been developed to ensure that (1) modeling input data
are valid and defensible, (2) model setup and parameterization (calibration) protocols are followed and
documented, (3) the model application and output data are reviewed and evaluated in a consistent
manner, and (4) the model application can predict hydrologic or water quality conditions over time.
Project quality objectives and criteria for data will be addressed by (1) evaluating the quality of the data
used and (2) assessing the model application results.
1.7.1 DATA ACCEPTANCE CRITERIA
Primary data are defined as data collected through personal experience or evidence, and secondary
data are data already collected and recorded by other researchers. Therefore, all of the data used in
this project are considered secondary data. The organizations generating the secondary data that may
be used in this project typically apply their review and verification procedures to evaluate a dataset's
integrity and conformance to QA/QC requirements. The data quality will be judged using information
from source documents, websites of origin, or directly from the authors. The data will be used if the
data quality can be adequately determined. If no quality requirements are determined to exist or can be
established for a dataset to be used for this task, a case-by-case determination will be made regarding
the data use. Table 1-4 summarizes the acceptance criteria for using secondary data in the setup,
calibration, and validation of the model.

QAPP-21
9

Table 1-4. Data Acceptance Criteria for Secondary Data
Quality
Criterion Description
Reasonableness Datasets will be reviewed to identify anomalous values representing data entry or analytical errors.
Such values will not be used without clarification from the agency providing the data.
Completeness
Datasets will be reviewed to determine the extent of gaps in space and time. Some data gaps will likely
be evident; these gaps and the methods used to fill the gaps will be discussed in the project
deliverables.
Comparability Datasets from different sources will be compared by checking the methods used to collect the data
and that the reporting units are standardized.
Representativeness
Datasets will be evaluated to ensure that the reported variable and its spatial and temporal resolution
are appropriate for the project; for example, datasets must be reasonably aggregated (or
disaggregated) to represent conditions in the model and must represent conditions during the
simulation periods. The goal is for data and information to reflect present-day conditions, and where
possible, data from the past 10 years will be used.
Relevance Data specific to the study site will be used. Regional data and information that most closely represent
the study site will only be used if needed.
Reliability
Data and information sources will be considered reliable if they meet at least one of the following
acceptance criteria:
/ The information or data are from a peer-reviewed, government, or industry-specific source.
/ The source is published.
/ The author is engaged in a relevant field that competent knowledge is expected.
/ The information or data were presented in a technical conference where the information was
subject to review by professional experts.
Data sources that use unknown collection methods and data review procedures are considered less
reliable and will be used only if necessary to fill data gaps and after discussion with and approval by the
NC DWR.
RESPEC currently anticipates that most of the data used in modeling will have been collected or
developed by sources commonly used for watershed model development. Whenever possible, the
modelers will electronically download secondary data directly from multiple sources to reduce the
possibility of introducing errors during data entry. In cases where multiple data sources are available,
the modelers will assess and use the best available data (i.e., that of the highest quality). Data of
unknown quality will be incorporated into the model only if approved by the NC DWR project team
leader, and the inclusion status of the data will be documented. If information is unavailable regarding
the data, either the data will not be used or will be qualified with the statement, "The quality of this
specific secondary dataset used in developing the watershed model could not be determined.” Most of
the data used for the model will be from trusted federal and state sources and collected under
appropriate QA/QC measures. RESPEC will note when data sources used in the model were not
collected under QA/QC procedures defined in an approved QAPP. Data not from trusted federal and
state sources or approved QAPP will be evaluated during the modeling process for anomalies or errors
and will only be used if the benefit of having those data is greater than a possible assumption to replace
them.

QAPP-21
10

Secondary data will be formatted and reviewed using scripting (MATLAB or Python). Data outside of the
typical ranges for a given parameter will be flagged for exclusion during the model setup, calibration,
and validation. Flagged data will only be excluded if the data are determined to be erroneous (e.g., faulty
calibration of sonde data may occur when dissolved oxygen drops quickly to near zero and stays there
for long periods of time). In such cases, characterizing the sonde data as "erroneous" is appropriate.
1.7.2 MODEL PERFORMANCE AND ACCEPTANCE CRITERIA
The EPA's Guidance for Quality and Assurance Project Plans for Modeling (EQP AQ/G-5M) [EPA, 2002]
discusses the importance of using performance criteria as the basis by which judgments are made on
whether or not the model results adequately support the decisions required to address the study
objectives. The focus of this section is to specify model performance criteria for the HSPF model to be
developed for the Neuse River Watershed. A weight-of-evidence approach that embodies the following
principles will be adopted for model calibration in this project [Donigian and Imhoff, 2009]:
/ Given that models are approximations of natural systems, the exact duplication of observed
data is not a performance criterion. The model calibration process will measure the model's
ability to simulate observed data through comparability goals.
/ No single procedure or statistic is widely accepted as measuring (nor capable of establishing)
acceptable model performance; thus, quantitative (error statistics) and qualitative (graphical)
comparisons of observed data and model results will be used to provide sufficient evidence to
weight the decision on model acceptance or rejection.
/ All model and observed data comparisons must recognize, either qualitatively or quantitatively,
the inherent errors and uncertainty in the model and the measurements of the observed
datasets. Where possible, these errors and uncertainties will be documented in the final report.
A weight-of-evidence approach ensures that graphical and statistical model comparisons are
completed to assess model performance. Graphical comparisons include time-series plots (i.e., daily,
monthly, annual), scatter plots, and frequency distributions of observed and simulated values.
Statistical comparisons include error statistics (e.g., mean error, absolute mean error, relative error, and
relative bias), correlation tests (e.g., linear correlation coefficient, coefficient of model-fit efficiency, and
Nash-Sutcliffe efficiency), and cumulative distribution tests [Donigian, 2002].

A model is considered calibrated when data reproduce within an acceptable level of accuracy, as
described in Table 1-5. The values in Table 1-5 provide general guidance regarding the percent mean
errors or differences between simulated and observed values so that users can gauge what level of
agreement or accuracy (i.e., very good, good, or fair) may be expected from the model application. The
target accuracy level for this project will be that which corresponds to “Good” or “Very Good” results at
more downstream mainstem calibration sites and “Fair” or “Good” at more upstream tributary sites.
Percent error will be calculated for total volume, storm volume, average storm peak, 5th, 10th, and
25th percentile high and low flows, median flows, seasonal volumes, and seasonal storms. The highest
priorities for comparison to meet the final accuracy levels are for the total volumes (i.e., monthly and
daily), storm volumes, and average storm peaks. Other statistical performance measures calculated
based on published calibration guidance include Correlation Coefficient, Coefficient of Determination,
Coefficient of Model Fit Efficiency (Nash-Sutcliffe Efficiency), Predictability Score, and multiple error
statistics. Formulas for different performance measures to be calculated are included in Appendix A of

QAPP-21
11

this document. Because the final accuracy targets are highly dependent on the amount and quality of
available data, the targets will be finalized after the data gaps are analyzed.
Table 1-5. General Percent Error Calibration/Validation Targets for Watershed
Models (Applicable to Monthly, Annual, and Cumulative Values)
[Donigian, 2000]
Parameter
% Difference Between Simulated
and Recorded Values
Very Good Good Fair
Hydrology/Flow < 10 10–15 15–25
Sediment < 20 20–30 30–45
Water Temperature < 7 8–12 13–18
Water Quality/Nutrients < 15 15–25 25–35
A set of parameters used in a calibrated model might not accurately represent field values, and the
calibrated parameters might not represent the system under a different set of boundary conditions or
hydrologic stresses. Therefore, a model validation period helps to establish greater confidence in the
calibration and predictive capabilities of the model. A site-specific model is considered validated if its
accuracy and predictive capability are proven to be within acceptable error limits independently of the
calibration data.

Numeric targets for this project are summarized in Table 1-6 for streamflow and Table 1-7 for water
quality. The targets are based on model performance guidelines in Donigian [2000]. During the model
calibration and validation process, RESPEC will hold monthly conference calls with state partners to
report interim calibration measures and summarize the model performance. During these discussions,
the project team will use the performance measures to determine if the model calibration was
successfully completed or if further adjustments are needed collectively.
Table 1-6. Example Numeric Targets for Streamflow Calibration and Validation
Category R R2 Percent
Difference
Nash-
Sutcliffe
Daily Flows > 0.8 > 0.65 < 25 > 0.65
Monthly Flows at Short-Term Sites > 0.8 > 0.65 < 25 > 0.65
Monthly Flows at Long-Term Sites > 0.9 > 0.8 < 15 > 0.8
Note: long-term sites have more than 1 year of streamflow records; short-term sites have 1 year or less.
The modeling team will strive to achieve the model performance targets listed in Tables 1-5 and 1-6 for
all of the monitoring sites. The modeling team expects to achieve these ratings at the majority of sites;
however, achieving satisfactory values may not be possible for some locations because of inherent
errors that can occur in observed and input datasets. In these cases, the model may still be deemed
calibrated and valid as long as such excursions are limited, and reasoning will be provided in the model
report to justify the validity of the model. More information about the model performance and
acceptance criteria is provided in Section 5.0.

QAPP-21
12

Table 1-7. Example Numeric Targets for Water Quality Calibration and
Validation
Water Quality
Constituent
Percent Difference at
Secondary Sites
Percent Difference
at Primary Sites
Sediment < 45 < 30
Water Temperature < 18 < 12
Nutrients < 35 < 25
Note: Primary sites have at least 1,000 samples of the constituent; secondary
sites have 500 to 1,000 samples of the constituent.
1.8 SPECIAL TRAINING REQUIREMENTS/CERTIFICATION
The RESPEC personnel working on this project hold advanced degrees from well-known universities for
excellence in surface-water modeling. Each RESPEC modeler has more than 5 years of experience
calibrating, validating, and applying HSPF water quality models. No special training or certifications are
required for personnel working on this project beyond the high degree of academic training and
professional experience already obtained. At the discretion of the NC DWR's project team leader, a
spreadsheet will be prepared and maintained in the project management files that provides
documentation of staff qualifications with respect to project responsibilities.
1.9 DOCUMENTS AND RECORDS
All of the data and information collected and generated during this project will be stored in a project
folder on the RESPEC network. At the project’s completion, RESPEC will transmit a copy of all the
project files to the NC DWR through ShareFile and maintain a copy of the files on our network for
5 years. The following deliverables will be completed under this project:
/ QAPP (draft and final)
/ Database containing the collective dataset and initial data analysis for model parameterization
to include GIS datasets, flow and water quality input datasets, and nutrient source input
datasets (point- and nonpoint-sources)
/ Technical memoranda documenting the development of input datasets, including the
justification for the final composition of the land-cover dataset, and summarizing the QA/QC
approaches
/ Up to two public meetings with the NC DWR and stakeholders during Task 1 (virtual or in-
person)
/ Model configuration memorandum
/ Model calibration report
/ An electronic version of the draft model, model inputs and outputs, and model preprocessors
and postprocessors
/ Model training sessions for the NC DWR staff and model support after the contract is complete
/ A minimum of two meetings with the NC DWR and stakeholders during Task 2 (virtual or in-
person)

QAPP-21
13

/ Summary table of source and delivered annual mass loads for different land covers broken out
by subwatershed, jurisdiction, and sources (point- and nonpoint-sources in the watershed)
/ Up to two meetings with the NC DWR and stakeholders during Task 3 (virtual or in-person)
/ Preliminary draft model report for the NC DWR review
/ Final draft model report that incorporates the NC DWR review comments
/ All modeling files, including all electronic model input files necessary to run the model, confirm
outputs, and any postprocessing tools.
The final report will provide a complete and clear summary of the modeling methodology and all the
data and assumptions used in the model such that the analysis can be easily validated/duplicated by
the NC DWR staff.

The final model application will include all other project documents, records, and electronic files
produced as deliverables. This documentation will include but not be limited to the following:
/ Results of technical reviews, model tests, and data quality assessments, as applicable
/ Model input and databases used
/ Response actions taken
/ Spreadsheet data files
/ Modeling reports.
If revisions to this QAPP are required during the project performance, the RESPEC QA/QC officer will
ensure that the individuals identified in the distribution list receive the most current copy of the
approved QAPP.

QAPP-21
14

2.0 MODELING APPROACH
This section outlines how RESPEC will approach building the Neuse River Watershed Model. The more
detailed simulation plan will provide a clear roadmap for developing the model in a separate document.
2.1 MODEL GEOGRAPHIC SCOPE
The Neuse River Watershed Model will be developed to characterize nutrient inputs from the Neuse
River Watershed below Falls Lake. Accurately representing those inputs from the watershed is critical
to assessing nutrient loads to the Neuse River Estuary. Four HUC8 subwatersheds are located within
the Neuse River Watershed (including the area draining to Falls Lake). A boundary condition will be
developed at the outlet of Falls Lake (USGS 02087183, NEUSE RIV AT SR 2000 NR FALLS). The Neuse
River Watershed Model study area is shown in Figure 1-2. The entire watershed below Falls Lake will be
modeled; however, calibration will only be completed in locations that are not tidally influenced.
2.2 TEMPORAL SCOPE
The Request for Proposal (RFP) mentions that the preferred modeling timeframe is 2002 through 2019.
RESPEC recommends starting the modeling timeframe in 2007 and extending it through 2022. Point-
source data collected before 2007 are often less reliable than newer data. An evaluation of flow and
precipitation throughout the final modeling timeframe will be done to ensure both dry and wet periods
exist during the time frame. Model validation will be completed by running the model during different
sets of years representing dry and wet periods.
2.3 MODEL ENDPOINTS
HSPF is currently one of the most comprehensive and flexible watershed hydrology and water quality
modeling systems representing point and nonpoint pollutant sources in a continuous hourly simulation
framework and makes HSPF well-suited to develop pollutant transport and delivery factors. An HSPF
simulation results in a time history of the runoff flow rate, sediment load, and nutrient concentrations,
along with a time history of water quantity and quality at any point in a watershed.

The Neuse River Watershed Model will focus on representing nutrient export from various land surfaces
and nutrient cycling, dissolved oxygen dynamics, phytoplankton (benthic and floating), and sediment
dynamics in the receiving waters. Modeling all nitrogen and phosphorous species is necessary to
ensure that nutrient cycling processes are properly represented.
2.4 MODEL SELECTION AND JUSTIFICATION
A model must be used that can represent each component, including the land area, stream channel,
waterbody, point source, and any diversion, to develop a scientifically sound modeling system to
represent the Neuse River Watershed. The model must also represent nutrient transport and enable the
calculation of delivery factors. RESPEC selected HSPF [Bicknell et al, 2001] as the modeling software to
develop the Neuse River Watershed Model because of its ability to simulate land surface and in-stream

QAPP-21
15

processes, including detailed in-stream nutrient simulation. RESPEC will use HSPF in conjunction with
SAM for this project. SAM will be developed and applied after the HSPF model development is
completed.

HSPF is a continuous watershed simulation model that runs on an hourly timestep and produces a time
series of water quantity and quality at any point in a watershed. The model is an extension and
reformulation of several previously developed models: the Stanford Watershed Model [Crawford and
Linsley, 1966]; the HydroComp Simulation Program [HydroComp, Inc., 1977]; Agricultural Runoff
Management model [Donigian and Davis, 1978]; and Nonpoint-Source Runoff model [Donigian and
Crawford, 1977]. HSPF Release 12.5 will be used for the Neuse River Watershed Model and is available
from the EPA BASINS website (https://www.epa.gov/ceam/basins-download-and-installation).

RESPEC's HSPF SAM is a user-friendly, comprehensive decision-support tool for planning and
implementing targeted actions to restore or protect water quality. The framework of the HSPF SAM
includes a GIS interface for BMP site selection; BMP database with pollutant-removal efficiencies and
associated costs; and scenario analysis, optimization, and reporting capabilities. SAM facilitates
planning to achieve the water quality goals established through watershed protection and restoration
programs. SAM also assists in understanding watershed conditions and identifying priority areas and
BMPs that will provide the greatest water quality benefits for each dollar invested. A separate
memorandum has provided the full rationale for selecting HSPF and SAM [Kenner, 2022].
2.5 MODEL-DATA NEEDS
While much of the spatial and temporal data required for HSPF model construction is available through
online platforms, RESPEC supplements these datasets with detailed, state-specific, and/or temporally
extended datasets and reports for developing HSPF models. A shared online cloud portal will be made
available for data sharing to streamline the process. RESPEC has used this method and found it useful
for data collection and organization. Ideally, metadata that meets a specific standard (i.e., source and
date) would be attached to every dataset uploaded to the shared site to ensure the highest level of
QA/QC possible.
2.5.1 METEOROLOGICAL SOURCES
A model application’s simulation period must align with the desired assessment periods. RESPEC has
found that using gridded datasets (PRISM and NLDAS) is the most efficient and accurate method to
meet the spatial-resolution needs and simulation periods needed for the watershed model application
development. We have developed a MATLAB tool that pulls these data for model meteorological zones
(based on a shapefile) and writes them to the proper files to be used by HSPF. This tool also makes the
extension of the meteorological data for the model possible at any time during the modeling process.
2.5.2 POINT SOURCES
Many point sources will need to be represented in the Neuse River Watershed Model application. The
state will provide major and minor point-source data. For the Neuse River Watershed Model application,
the NC DWR has requested that the EPA ECHO data not be used, DWR will provide the data directly. The
model requires some parameters that are not available through point-source monitoring. Unavailable

QAPP-21
16

parameters can be estimated, or assumptions can be used to fill gaps. Depending on the missing
constituent, the estimate could be derived using a surrogate facility with nutrient-speciation factors or
by setting a constant concentration. RESPEC will work with individual wastewater treatment plants
(WWTPs) and the NC DWR to get the needed information. RESPEC will also work with stakeholders to
ensure the most accurate methods to fill in time gaps and the most appropriate surrogates are used to
make the appropriate assumptions.
2.5.3 OTHER EXTERNAL TIME-Series DATA
Atmospheric nitrogen deposition data are obtained from the National Atmospheric Deposition Program
and Clean Air Status and trends Network (NADP) and CASTNET. RESPEC has developed streamlined
methods/scripts to fill the NADP and CASTNET nitrogen data to represent a daily time series.
Atmospheric phosphorus deposition data are generally attained from local studies. For models where
irrigation or flood-control impoundment is represented, water use or storage and release data are
obtained from state and local sources. Similarly, withdrawal and water use data are generally supplied
by state or local sources.
2.5.4 WATERSHED CHARACTERISTICS
Each region or river watershed has distinctive land-cover characteristics or land-use practices that
must be represented in the model applications, which can require specific spatial datasets. While land-
cover and elevation data are available through the National Land Cover Database (NLCD) (Figure 2-1),
RESPEC will work with stakeholders to determine if a more locally developed land-cover dataset would
be more appropriate for the modeling effort. RESPEC will also collect highly detailed light detection and
ranging- (LiDAR-) based digital elevation models (DEMs) as needed and available. RESPEC also has
methods to determine the distribution of septic systems depending on whether such data are available
spatially. Spatial datasets of feedlots and National Pollutant Discharge Elimination System (NPDES)
wastewater discharges are readily available on NC OneMap and can be used for model development.
2.5.5 IMPERVIOUS LAND CLASSIFICATION
The Effective Impervious Area (EIA) is essential to accurately represent in watershed models because
of its impact on the hydrologic processes that occur in developed environments. The term "effective"
implies that the impervious region is directly connected to a water conveyance system (e.g., gutter,
curb drain, storm sewer, open channel, or river) and that overland flow will not run onto pervious areas
to infiltrate before reaching a stream or waterbody. The Total Impervious Area (TIA) for an impervious
land segment can be used to determine the percent EIA using the Sutherland equation [Sutherland,
2000]. TIA can be determined from datasets such as the NLCD Percent Developed Imperviousness
data layer.
2.5.6 MUNICIPAL SEPARATE STORM SEWER SYSTEMS
Polluted stormwater is commonly transported through Municipal Separate Storm Sewer Systems
(MS4s) before being discharged into local waterbodies. Certain MS4s are required to obtain NPDES
permits and develop stormwater management programs [NPDES, 2020]. The model will represent
regulated MS4 areas as distinct land segments (available on NC OneMap) using a separate mass link to
differentiate flow and pollutants from non-MS4 areas.

QAPP-21
17

Figure 2-1. Land-Use Map of the Neuse River Watershed
2.5.7 SEPTIC SYSTEMS
A septic system falls under onsite wastewater treatment systems (OWTS). Many households use OWTS
in the Neuse River Watershed. OWTS generally contribute pollutant loads to either the groundwater or
tributaries. North Carolina has polygons that show sewer service areas (available on NC OneMap). Areas
outside sewer service areas will be combined with the U.S. Census Bureau Block Centroid Population
dataset to estimate the population in each model subwatershed that uses OWTS. Census blocks are
the smallest geographic entities within a county that the U.S. Census Bureau tabulates population.
OWTS may be represented in the model application as a constant load and assumed to discharge at
46.4 gallons per person per day [EPA, 1993]. If available, this information may be supplemented with
more recent, local, or more detailed data.
2.5.8 Agricultural DATA
Available agricultural data (e.g., crop types, planting and harvest dates, and nutrient applications) and
livestock manure characteristics (e.g., manure generation, nutrient content, nutrient volatilization, and
animal time in pasture) will be used in the model application if significant and available. Other data that
are helpful for modeling include tillage and tile drainage estimates. Tile drainage can be estimated by
analyzing GIS map layers such as land cover, soil attributes, and slope to identify farmed areas with
poor drainage if it is significant in the project area.

QAPP-21
18

2.5.9 ELEVATION AND SLOPE
Average elevation, average slope, and other topographic parameters for each model subwatershed will
be estimated from either the USGS 3DEP DEM or the NC One Map 20-foot grid cell DEM.
2.5.10 REACH PROPERTIES
Reach length and slope are required to determine physically based parameters in the model and to
develop hydraulic function tables (known as FTABLES in HSPF). The reach length and slope values will
be calculated using ArcGIS Pro for all nonlake reaches. The reach slope will be derived from a high-
resolution DEM grid. Data requirements for stream FTABLE development include channel cross
sections, rating curves, and discharge measurements. Sources of cross-section measurements may
include but are not limited to NCDOT bridge data; data contained in HEC-RAS models developed by
other organizations; USGS measurements; and LiDAR data, where available. The data needed to
develop FTABLES for small lakes and ponds include the waterbody surface area and volume at various
water elevations (depths) and overflow information. If available, surface area, volume, depth, spillway
length, height above sill, and runout elevation data will be used for lake FTABLE development. Lake
bathymetry and/or level data supplied by the state and U.S. Army Corps of Engineers (USACE) National
Inventory of Dams can be used for FTABLE development.
2.5.11 DRAINAGE
Watersheds and waterbodies from the National Hydrography Dataset (NHD) and layers (e.g., the DWR
Surface Water Classifications) are options for developing the model subwatersheds. Watersheds
delineated at locations with significant flow or water quality data available and at impairment locations
near significant point sources are important. As needed, the watershed can be further delineated below
the HUC12 level by using the NHD hydrologically conditioned DEM and points at desired locations for
calibration and implementation planning.
2.5.12 OBSERVED/CALIBRATION DATA
USGS discharge and EPA water quality datasets for model calibration can be obtained through the
National Water Information System (NWIS) (USGS Water Data for the Nation) and the National Water
Quality Monitoring Council's Water Quality Portal (Water Quality Portal). For this project, it was
suggested that locally provided data might be more accurate than those found on the Water Quality
Portal and, as such, will be prioritized over data from the Water Quality Portal. Additionally, stakeholders
may have other data (e.g., flow, water quality, lake level, and studies) that are not included in NWIS or the
Water Quality Portal, which, if provided, will be used for the calibration. Studies conducted in
partnership with local agencies can guide calibration by providing insight into delivery mechanisms.
2.6 DATA GAPS
RESPEC understands that no model application is perfect and challenges along the way need to be
overcome. The following are some of the expected challenges:
/ The algal growth, respiration, and uptake of nutrients are important processes to model
accurately. Often, some parameters (e.g., chlorophyll a) that help to calibrate the model are
minimally available. RESPEC will work to ensure that attention is paid to what data are available
so that they accurately represent these important processes.

QAPP-21
19

/ Point-source data are imperfect. RESPEC has a QA/QC method to ensure outliers and point-
source issues are identified before being used in the model. Area experts will review any issues
to ensure the best available data are being used.
/ The process of filling missing point-source data is complicated. RESPEC has tools and scripts
generated to fill different types of point-source discharges (e.g., occasionally discharging
ponds versus continuously running mechanical facilities). A field expert should verify the script
logic.
/ Septic systems can make up a very large portion of the load, and assumptions for septic
systems can be poor. RESPEC will leverage the knowledge of local experts and regional studies
to ensure the best representation of septic systems in the model.
/ Nutrient loads can be greatly impacted by applying and spraying manure and it can be
challenging to understand precisely where these practices occur. RESPEC will work with local
experts and review NPDES data to understand how to represent these practices best.
/ The plan to represent boundary conditions is to use a complete flow time series and a less
complete water quality time series below the outlet of Falls Lake Dam. Upstream point sources,
high temperature-related growth, and other low-flow phenomena can impact the relation of
flow and time of the year to water quality constituents. RESPEC will review past reports related
to special load relationships to ensure boundary conditions are well represented.
/ Data availability and time to collect data are some of the biggest challenges in modeling.
Occasionally, entities with data do not want their data shared or used in a model. Receiving all
available datasets can be a restriction to water quality modeling. RESPEC will work with
shareholders to set a final date to receive datasets and will formulate a plan to move forward if
receiving specific datasets becomes a problem.
/ The most recent NLCD is from 2019. More recent land-cover layers can be introduced in the
middle of a project. RESPEC will assume that this is the most representative of the calibration
(not validation) period, representing the most recent years of the modeling period.
/ Land application in agricultural areas can be difficult to understand. Nutrient inputs can be off if
the land cover does not adequately represent cropland versus pasture. RESPEC will have
discussions with the shareholders to ensure the best representation of the data available to us.
/ Fertilizer application rates and pet waste nutrient loads in urban areas are difficult to
understand. RESPEC will use information from shareholders to ensure the best representation
of these loads.
/ A goal should be to maintain parameter consistency on land covers throughout the watershed
unless there are reasons to adjust them.
/ Parameters should remain within recommended ranges unless a plausible reason exists to go
beyond ranges.
/ Impervious cover percentages represented on different developed land-cover classifications
can be variable. These will be verified with NCDOT to ensure they are reasonable before being
used in the model.

QAPP-21
20

3.0 DATA ACQUISITION/MANAGEMENT
Time-variable and GIS data of known and documented quality are essential to the success of any water
quality modeling study and generate information to use in establishing watershed management
strategies. The QA/QC process for the HSPF modeling of the Neuse River Watershed consists of using
appropriate data, data analysis procedures, modeling methodology and technology, administrative
procedures, and auditing. The quality of a modeling study is determined to a large extent by the
expertise of the modeling and quality assessment teams.

The quality of an environmental analysis program is achieved by means of three steps: (1) establishing
scientific assessment quality objectives, (2) evaluating the program design for whether the objectives
can be met, and (3) establishing assessment and measurement quality objectives that can be used to
evaluate the appropriateness of the methods used in the program. The quality of a dataset is a measure
of the type and amount of error associated with the data. Error sources are commonly grouped into two
categories: sampling error and measurement error. These types of errors, as well as processing errors,
can affect the accuracy and interpretation of results. For various reasons, some of the data evaluated
for potential use in developing, calibrating, and testing the model may be judged as acceptable for uses
to support this modeling effort. The data acquisition procedures that will be followed for modeling
include data review and management practices that will reduce the sources of error and uncertainty in
using the data. The quality considerations involved in data acquisition and management are described
in the following sections.
3.1 DATA ACQUISITION
This project will require secondary data (data collected under a different effort outside of this project),
also referred to as non-direct measurements. Much of the data needed for this model application
reside on the NC DWR servers, EPA's Water Quality Portal, Neuse River Basin Associations, the North
Carolina Department of Agriculture (NC DOA), the NCDOT, USGS, and other federal/state sources.
These data will be downloaded from those sites to support this project. Supplementary secondary data
will be collected from government publications and databases, scientific literature, related technical
studies, watershed groups, and other organizations as needed. Data commonly required for populating
a database for calibrating watershed model applications are listed in Table 3-1. The data in the table are
exemplary and, as such, are not intended to be all-inclusive. Table 1-5 summarizes the acceptance
criteria for using secondary data in the model setup, calibration, and validation.

The primary mechanism used for data acquisition will be electronic downloading. RESPEC will adhere to
protocols developed while performing hundreds of previous HSPF applications to ensure that QA/QC
considerations are properly addressed for preventing, detecting, and correcting downloading errors.

RESPEC will include a summary of the data collection activities, data sources, and expected data uses
in a section of the model simulation plan submitted to the NC DWR. The following information will be
provided for each dataset in an outline format:
/ Data type
/ Originating source agencies

QAPP-21
21

/ Time period the data were collected (if different from the modeling period)
/ Available QA/QC documentation
/ Data gaps
/ Planned use in model setup and/or calibration and validation efforts.
Table 3-1. Secondary Environmental Data to Be Assembled for Watershed and Water Quality Modeling in the Neuse River
Watershed
Data
Type
Example Measurement
Endpoint(s) or Units
Geographic or Location Information (Typically in GIS Format)
Hydrologic Unit Code Boundaries ArcGIS Shapefile or Feature
Hydrography ArcGIS Shapefile or Feature
Land Use ArcGIS Shapefile or Raster
Topography Digital Elevation Model (meters or feet)
Population Distributions ArcGIS Shapefile or Feature/Number
Soils (including soil characteristics) ArcGIS Shapefile or Feature/Hydrologic Group
Water Quality and Biological Monitoring and Meteorological Station Locations Latitude/Longitude
Permitted Point-Source Discharge Locations Latitude/Longitude
Dam Locations Latitude/Longitude
Flow
Historical Record (daily) cubic feet per second (cfs)
Peak Flows (daily maximum) cfs
Storm Hydrographs (hourly or less) cfs
Permit Limits (flow) cfs
Meteorological Data
Rainfall inches
Temperature degrees C (°C)
Potential Evapotranspiration inches
Wind Speed miles per hour
Dew Point °C
Humidity percent or grams per cubic meter
Cloud Cover percent or grams per cubic meter
Solar Radiation watts per square meter
Water Quality (Surface Water, Groundwater)
Total Suspended Solids milligrams per liter (mg/L)
Nutrients mg/L
Dissolved Oxygen mg/L
Permit Limits (concentrations) mg/L, micrograms per liter (μg/L)

QAPP-21
22

For any identified gaps in the necessary data, RESPEC will assess the potential impact on the project
and suggest possible workarounds. A final technical memorandum incorporating comments provided
by the NC DWR will be prepared and approved before the model setup. An electronic copy of the files
developed as part of the data acquisition process will be provided to the NC DWR concurrent with the
final technical memorandum submittal.

The data acquisition plans, activities, and results will undergo additional review when incorporated into
the model approach/configuration document, which is also a required deliverable of this project. This
document has a broader scope that includes considering the approach for identifying and using data
and the approach for developing, calibrating, validating, and applying the HSPF model. In this context, all
the datasets and watershed-loading estimates targeted for use in the model calibration process will be
identified. A general list of all the data needed to run the HSPF model will be assembled and presented.
The model approach/configuration document will be the basis for a webinar/online meeting that
includes RESPEC, NC DWR staff, and partners. The audience of knowledgeable people contributing to
the data acquisition process will be expanded as a result of the webinar. This process will help to ensure
that the most recent and applicable datasets are used.

The data used for modeling will be documented in the final modeling report. The report will include a
summary of all the final data used in the setup, calibration, and validation of the model. The final data
used in the model, period of record of the data, and data source will be documented, along with any
secondary data of unknown quality, data gaps, and assumptions used in filling such gaps.

Included among the data types that the final modeling report will address are the following datasets:
/ GIS coverages used in model setup
/ Meteorological data used to drive the model
/ Point-source loading data used as model input
/ Observed data used in model calibration and validation.
3.2 DATA MANAGEMENT
Two data types will be used to support HSPF modeling in the Neuse River Watershed: GIS and time-
series data. The data types must change format as they are integrated into an HSPF model and are thus
subject to possible errors. As with electronic data acquisition, RESPEC will adhere to the protocols
developed while performing an abundance of previous HSPF applications to ensure that QA/QC
considerations are properly addressed related to preventing, detecting, and correcting electronic data
manipulation errors. The protocols are listed in the following paragraphs.

Errors in data manipulation are minimized by automating the data manipulation processes. GIS data are
projected automatically using a standard projection library. When a new type of GIS data are added to a
project, projection of those data is translated to match the projection of the project. When time-series
data are downloaded, they are automatically imported into the standard HSPF database formats.
Having these processes automatically occur minimizes the mistakes that could occur during this
process.

QAPP-21
23

When a new dataset is processed, the resulting data are checked against the data at the beginning of
the process to ensure accuracy. GIS and time-series data are visually checked. If the new dataset is
very large, the manipulation processes are automated by writing and testing software scripts. All data
are visually inspected for a selected subset of the dataset while testing the software scripts. If that
inspection yields successful results, the software is run as a "production run" for manipulating the entire
dataset.

After the production run, RESPEC verifies that the results exactly duplicate what was produced during
software testing. This verification is accomplished using comparison software such as Beyond
Compare. If that verification holds, a visual cross-check of a small portion of the data occurs. Data are
typically visually inspected for a second subset of the dataset during this phase. Then a visual cross-
check of a small portion of the manipulated data records, perhaps one per thousand, throughout the
entire dataset is performed. The entire process must be rerun if errors are found at any point. If reran,
the visual cross-check is performed again at the end of the process. When no errors are found, the
checking ceases.

Because the manipulation processes are automated using custom computer software scripts, the fixes
are accomplished by fixing the automated conversion software. After the software is corrected, the
entire visual check process repeats until satisfactory results are achieved.

Consistent data management procedures will be used during the preprocessing, model calibration, and
postprocessing stages of the project and will be stored in a project folder on RESPEC's network. Data
processing will be completed using a combination of ArcGIS, MATLAB, Python, and SARA Timeseries
Utility. RESPEC modelers will be responsible for adhering to and documenting data management
practices that ensure the quality of downloaded and/or manipulated data. Original data sources will be
documented to identify the website or contact person that provided the data, data query parameters,
and data request correspondence. Original (unaltered) copies of all the data sources used in the project
will be retained in the project folder. Metadata will be included with spatial datasets.

GIS data will be used in a geodatabase feature-class format. The projection of all the GIS data will be
consistent. When new GIS data are added to a feature class, ArcPro automatically projects the data to
match the projection of the feature class.

Because of the large amount of data required for model development, the model approach/
configuration report (see Section 3.1) will include a section that addresses efficient data management
and organization for the modeling project. The report will also address the transfer of data and other
information from agencies to RESPEC (and vice versa) through interaction with publicly accessible
databases and file-sharing sites as much as possible. As mentioned in Section 3.1, a draft of the model
approach/configuration report will be the basis for a webinar/online meeting that includes RESPEC, the
NC DWR staff, and partners. The webinar will allow for an optimal data management approach to be
identified and practiced, thereby further ensuring effective inter-organizational QA/QC practices with
respect to data management.

At the project’s completion, RESPEC will transmit a copy of all the project files to the NC DWR and
maintain a copy of the files on our network for 5 years.

QAPP-21
24

3.3 DATA REVIEW, VERIFICATION, AND VALIDATION
Data review and validation processes formalize a method for determining the usability and limitations of
data and provide a standardized data quality assessment. RESPEC's experienced professionals will
conduct the study's data review, compilation, and evaluation phases and review data entries,
transmittals, and analyses for completeness and adherence to QA/QC requirements. As described in
Section 1.7, a screening process will be used to identify data outside typical ranges. Values outside
typical ranges will not be used to develop model calibration datasets. The Modeling QA/QC Officer will
review or oversee all data related to the project for completeness and correctness. The Modeling
QA/QC Officer will report any issues of concern to the RESPEC Project Manager, who will resolve these
issues with the modeling team. General procedures for model calibration and validation are described in
Chapter 5.0.

QAPP-21
25

4.0 ASSESSMENT/OVERSIGHT
4.1 ASSESSMENT AND RESPONSE ACTIONS
The project QA/QC program includes surveillance with independent checks of data acquisition, model
setup, model application, results analysis, and documentation activities. The essential steps in the
QA/QC program are as follows:
/ Identify and define the problem
/ Assign responsibility for investigating the problem
/ Investigate and determine the cause of the problem
/ Assign and accept responsibility for implementing the appropriate corrective action
/ Establish the effectiveness of and implement the corrective action
/ Verify that the corrective action has eliminated the problem.
If quality problems that require attention are identified, the project manager will determine whether
attaining acceptable quality requires either short- or long-term corrective actions. The technical
problems that might occur can often be solved on the spot by the staff members involved; for example,
a staff member can modify the technical approach or correct errors or deficiencies in documentation.
Immediate corrective actions form a part of normal operating procedures and are noted in records for
the project (e.g., monthly progress reports). Problems that cannot be resolved in this manner require
more formalized, long-term corrective action. Examples of significant corrective actions include the
following:
/ Reemphasizing to staff the project objectives, scope limitations, adhering to the agreed-upon
schedule and procedures, and documenting QA/QC activities
/ Securing additional commitment of staff time to devote to the project
/ Retaining outside consultants to review problems in specialized technical areas
/ Changing procedures (e.g., replacing a staff member if doing so is in the project's best interest).
The RESPEC Project Manager is primarily responsible for monitoring the project activities and
identifying or confirming any quality problems. RESPEC’s staff and project manager for this project
have been working together for more than 5 years on similar HSPF model applications. During this time,
RESPEC established the practice of convening weekly internal project meetings encouraging direct,
open, and frequent communication of project issues. The project manager and QA/QC officer will
participate in the weekly meetings.

If significant problems are brought to the attention of the QA/QC officer, they will initiate the corrective
actions described above, document the nature of the problem, and ensure that the recommended
corrective actions are carried out. The project manager and QA/QC officer can stop work on the project
if the identified issue affects data quality and requires extensive effort to resolve. The NC DWR project
team leader will be notified of significant corrective actions and stop work orders. The NC DWR project
team leader can stop work on the project if QA/QC concerns arise.

QAPP-21
26

RESPEC's project manager will perform surveillance activities throughout the project duration to ensure
that the management and technical aspects are properly implemented according to the schedule and
quality requirements specified in this QAPP. These surveillance activities include assessing how project
milestones are achieved and documented, corrective actions are implemented, budgets are adhered
to, technical reviews are performed, and data are managed. QA/QC surveillance activities will be
documented in quarterly progress reports.
4.2 REPORTS TO MANAGEMENT
RESPEC will prepare monthly progress reports delivered with invoices. These reports will outline the
activities during the invoiced month, difficulties encountered, and planned activities for the following
month. The project manager will provide each report to the project staff as a draft for review and
comment. When this review is complete, the team will submit a final report to the project team leader
who will review and submit the final report to the client.
4.3 MODEL REVIEW
NC DWR quality guidance recommends independent third-party review of modeling efforts. RESPEC
recommends NC DWR obtain an independent third-party or peer review of the draft model. While
outside the scope of this contract, an independent review will provide additional confidence in the
model’s capability to address the stated goals.

QAPP-21
27

5.0 MODEL APPLICATION
The EPA [2000; 2002] emphasizes a systematic planning process to determine the type and quality of
output needed from modeling projects. This process begins with a modeling needs and requirements
analysis, which includes the following components:
/ Assess the need(s) of the modeling project
/ Define the purpose and objectives of the model and the model output specifications
/ Define the quality objectives to be associated with the model outputs.
NC DWR has developed the front-end modeling needs and requirements for this project, including the following:
/ Support the Neuse Nutrient Strategy by providing a calibrated watershed model that meets
agency standards for regulatory use and support.
/ Determine transport zones and delivery factors for point-source discharges and nutrient offset
credits using the watershed model.
/ Estimate the proportion of end-of-pipe nutrient loading from wastewater sources that reach
the Neuse River Estuary.
/ Identify other nonpoint nutrient sources and their relative impacts on nutrient loading to the
Neuse River Estuary.
/ Run at least in a daily time step and represent different forms of nutrients including nitrate +
nitrite (NO2/ NO3), ammonia (NH4), and organic nitrogen (ON), and represent time-varying
dynamic systems.
/ Include the spatial scale for the Neuse River Watershed, beginning below the Falls Lake Dam
and extending down to the estuary. The Falls Lake Watershed is excluded from explicit
modeling in this effort to avoid unnecessary duplication of existing efforts by the Upper Neuse
River Basin Association to model this watershed.
/ Incorporate a wide range of meteorological conditions and consider the availability of data and
resources that determines the timeframe selection for the modeling. NC DWR's preferred
modeling timeframe is 2004–2021.
/ Involve stakeholders from the Neuse River Watershed, including the Neuse River Basin
Associations, the United States Geological Survey, NC DOA, NCDOT, and other interested
organizations.
The quality objectives for the model follow directly from the purposes and objectives stated above. The
modeling effort generally needs to be designed to achieve an appropriate level of accuracy and
certainty in answering the central study questions. This process considers the following elements:
/ The accuracy and precision needed for the model applications to predict a given quantity at the
application site of interest to satisfy the study questions
/ The appropriate criteria for deciding whether or not the model applications are accurate and
precise enough are based on past general experience combined with site-specific knowledge
and completeness of the conceptual models

QAPP-21
28

/ The appropriate criteria would be used to determine whether or not model outputs achieve the
needed quality.
RESPEC will develop a model configuration/approach document describing how HSPF will be used to
model the Neuse River Watershed. This document will provide details on how the data will be used to
support the modeling project, assess the data’s perceived quality, and provide a roadmap and
communication tool for the NC DWR and stakeholders. The document will describe the study
objectives, available data, water quality and land uses, calibration/validation procedures, and targets.
The final modeling approach document will be submitted to NC DWR after any written comments or
comments received during a webinar/online discussion meeting with NC DWR staff and other partners
are incorporated.

The remainder of this section addresses model accuracy and precision and will be further elaborated in
the modeling approach document.
5.1 MODEL PARAMETERIZATION (CALIBRATION)
Model calibration is the process of adjusting model inputs within acceptable limits until the resulting
predictions provide a good correlation with observed data. Calibration commonly begins with the best
estimates for model input based on measurements and subsequent data analysis. Results from initial
simulations are then used to improve the concepts of the system or modify the values of the model
input parameters. The goal is to develop a watershed model to determine the reductions in pollutant
loads needed to improve water quality. Model calibration and validation should strive to minimize the
differences between model predictions and observed measurement data; hence, the availability of
abundant observed data is an essential element of successful calibration.

The experience and judgment of the modelers will likewise be a significant factor in accurately and
efficiently calibrating the model(s). The RESPEC Principal-in-Charge and project manager will direct the
model calibration efforts with assistance from competent modelers with significant experience with the
model(s) they are applying. RESPEC Principal-in-Charge Russell Persyn served as the project principal
for numerous water quality modeling and model tool development projects. Project Manager Seth
Kenner has been a technical leader in developing more than 50 HSPF watershed model applications,
with 14 years of experience in applying HSPF to represent complex hydrologic and water quality
processes. Well-developed internal protocols and customized spreadsheet tools support the
calibration efforts of RESPEC's HSPF modelers. As a backup to RESPEC's HSPF modeling expertise,
detailed guidelines for HSPF calibration [Duda et al., 2012; Michael Baker, Jr., Inc. et al., 2013] are
available and will be consulted.

Modeling procedures and model results will be routinely reviewed by senior-level modelers at RESPEC
and subjected to additional review by the NC DWR. The results will also be made available to interested
stakeholders.

The model calibration effort will be designed and performed to meet prespecified quantitative
measures of accuracy that will establish the model's acceptability in answering the principal study
questions. The model calibration process proceeds through qualitative and quantitative analyses. The

QAPP-21
29

quantitative analysis checks to confirm targets are met. Qualitative measures of the calibration
progress are commonly based on the following:
/ Graphical time-series plots of observed and predicted values
/ Flow and concentration duration plots of observed and predicted values
/ Monthly average plots of observed and predicted values
/ Annual average plots of observed and predicted values
/ Scatterplots of observed versus predicted values
/ Measured and predicted values and their deviations tabulated.
Initial model parameters will be set based on available data and literature. After the model has initially
been configured, the modelers will perform model calibration and validation. The watershed model will
be calibrated to the best available data, including literature and interpolated or extrapolated values
using existing field data. If multiple datasets are available, an appropriate time period and
corresponding dataset will be chosen based on factors characterizing the dataset, such as the related
weather conditions, amount of data, and temporal and spatial variability data. A calibration journal will be
kept to document the calibration states at certain calibration stages. Parameterization will remain as
consistent as possible with respect to accepted values for land-cover classifications and stream order.
If deviations are needed for a specific area (e.g., an area where changes in the soil type or geology are
not represented by land-cover classification or do not correspond to existing implementation
practices), the reasoning for the parameterization values adjustments will be discussed in the final
report.

Traditional sensitivity analysis involves comparing the relative differences in parameters and responses
by dividing the percent change in a variable response by the percent change in a calibration parameter,
which helps to understand how parameters impact the physical and biological processes represented
in the HSPF model. Although this analysis is valuable and inherent during the calibration process,
several thousands of parameter-variable options exist for evaluating and interpreting the results and
require a thorough understanding of HSPF mechanics and terminology. A more useful approach to
understanding model sensitivity involves using real-world terminology to evaluate source contributions.
Simulated source contributions at subwatershed outlets for selected time periods and/or flow
conditions can be determined by using the HSPF model application results for each day, source,
watershed, and constituent of concern. The source contribution analysis will be included for the outlet
of each model application as a part of the final model deliverables. A more detailed source analysis can
be performed with the SAM tool. Methods for using SAM for the detailed source analysis will be
included in the training.
5.2 MODEL CORROBORATION (VALIDATION AND SIMULATION)
Model validation is performed using a dataset separate from the calibration data. If only a single time
series is available, the series may be split into two segments: one for calibration and another for
validation. If the model parameters are changed during the validation, this exercise becomes a second
calibration, and the first calibration needs to be repeated to account for any changes. Representative
stations will be used to guide parameter adjustment to obtain an accurate representation of the

QAPP-21
30

individual subwatershed and stream conditions. The calibration and validation approach intended for
this project will be proposed in the modeling approach document, confirmed at the webinar/online
meeting, and documented in the final report.
5.2.1 MODEL PERFORMANCE TARGETS
Calibration and validation will be achieved by considering qualitative and quantitative measures
involving graphical comparisons and statistical tests. For flow simulations where continuous records
are available, both qualitative and quantitative techniques will be employed, and the same comparisons
will be performed during the calibration and validation phases. Value comparisons for simulated and
observed state variables will be performed for daily, monthly, and annual values in addition to flow-
frequency duration assessments. Statistical procedures will include error statistics, correlation and
model-fit efficiency coefficients, and goodness-of-fit tests, as appropriate. Figure 5-1 provides value
ranges for the correlation coefficient (R) and coefficient of determination (R2) for assessing the model
performance for daily and monthly flows. The figure shows the range of values that may be appropriate
for judging the model’s performance based on the daily and monthly simulation results. As shown in
Figure 5-1, the ranges for daily values are lower to reflect the difficulties in exactly duplicating the timing
of flows given the uncertainties in the timing of model inputs (especially precipitation). The model
development plan will include more information on the calibration and validation methods.

Figure 5-1. R and R2 Value Ranges for Model Performance [Donigian, 2002].
The values shown in Figure 5-1 are derived from extensive experience with the individual model
applications and the selected past efforts on model performance criteria discussed above. If
preliminary model results do not satisfy the target tolerances listed in Figure 5-1, additional efforts will
be required to investigate all the possible errors in, as well as the accuracy of, input data, model
formulations, and field observations. If adjustments in these tolerances are needed, they will be fully
investigated and documented, and QAPP revisions will be issued through the formal QA/QC process.
Given the uncertainties in model performance criteria, inherent errors in input and observed data, and
approximate nature of model formulations, absolute criteria for watershed model acceptance or
rejection are not generally considered appropriate by most modeling professionals.

For water quality calibration, various methods are used to compare simulated and observed mean
values. The sporadic observed data can be aggregated over annual, seasonal, or monthly time frames
and compared to the full range of simulated values. The simulated time series can alternatively be
sampled to include only the time periods when samples were gathered and model-data comparisons

QAPP-21
31

can then be limited to those sampled time periods. Both approaches have advantages and
disadvantages. These approaches and others will be explored as part of the model performance
evaluation.

For sediment, water quality, and biotic constituents, model performance will be based primarily on
visual and graphical presentations because the frequency of observed data is often inadequate for
accurate statistical measures. However, alternative model performance assessment techniques
(e.g., error statistics and correlation measures shown in Table 5-1) that are consistent with the
population of observed data available for model testing are investigated during model calibration.
Table 5-1. General Calibration Targets or Tolerances for the Percent Difference Between
Simulated and Recorded Water Quality Constituents for HSPF Applications
[Donigian, 2000]
Calibration
Parameter
Very Good
(% Difference)
Good
(% Difference)
Fair
(% Difference)
Sediment (Sand, Silt, and Clay) < 20 20–30 30–45
Water Temperature < 7 8–12 13–18
Water Quality/Nutrients < 15 15–25 25–35
Pesticides/Toxics < 20 20–30 30–40
Stipulations:
/ Relevant to monthly and annual values; storm peaks may differ more than the monthly and
annual values
/ Quality detail of input and calibration data
/ Purpose of model application
/ Availability of alternative assessment procedures
/ Resource availability (i.e., time, money, and personnel)
Traditional sensitivity analysis involves comparing the relative differences in the results by dividing the
percent change in a variable response by the percent change in a calibration parameter, which helps to
understand how parameters impact the physical and biological processes represented in the HSPF
model. Although this analysis is valuable and inherent during the calibration process, several thousands
of parameter-variable options exist for evaluating and interpreting the results requires a thorough
understanding of HSPF mechanics and terminology. A more useful approach to understanding model
sensitivity involves using real-world terminology to evaluate source contributions. By using the HSPF
model application results for each day, source, basin, and constituent of concern, source contributions
at all of the locations for selected time periods and/or flow conditions can be calculated. The final model
deliverables will include the source contribution analysis for the outlet of each model application (i.e., all
sources contributing to the endpoint of a mainstem river).
5.2.2 POSTPROCESSING TOOLS
Hydrology and water quality calibration are completed through an iterative process of parameter
adjustments and simulated and observed value comparisons. Scripted processes in MATLAB and
RESPEC's expert system for hydrologic and water quality calibration tool (HSPEXP+) facilitate
calibration. RESPEC has developed software applications such as HSPEXP+, SAM (Preprocessing
Application Translator for HSPF [PATH] tool), and our in-house MATLAB tools to expedite calculating

QAPP-21
32

these metrics and ensure that applications are evaluated in a consistent manner. The HSPEXP+ Regan
plots available to assist in calibration by reach include a daily time-series plots of minimum and
maximum simulated DO concentration; time-series plots of simulated phytoplankton, dissolved
ammonia, dissolved nitrate, dissolved orthophosphorus, and benthic algae; time-series plots of total
suspended solids, bed depth, and flow; and load/duration curves.
5.3 RECONCILIATION WITH USER REQUIREMENTS
Evaluating the data quality and comparing the methods and results with published data and scientific
literature, as well as the data quality objectives identified in this QAPP will determine the value of the
information generated by this project. Confidence in model predictions can be limited by several
factors, including the representativeness of calibration data, knowledge of actual nutrient inputs
(external and internal loading), and the inherent ability of the model to simulate the conditions.

Data quality indicators will be calculated during model setup, calibration, and validation. Measurement
quality requirements will be compared with the data quality objectives to confirm that the correct type,
quality, and quantity of data are used for the model setup and calibration. Computation and post-
simulation analysis results will be reviewed for reasonableness. The final report will sufficiently detail the
data sources, assumptions made, and calculations used in the preprocessing, modeling, and
postprocessing to ensure the reproducibility of the work by NC DWR.

As part of the reconciliation process, the model deliverables (e.g., modeling reports and technical
memoranda) will be reviewed by NC DWR to assess whether or not the quality requirements of the
QAPP have been met. A comprehensive review of the final model files and documentation will be
completed, and recommendations will be provided regarding the model’s effectiveness in predicting
hydrologic and water quality response.

Inherent in the model reconciliation process is the recognition that models are simplifications of the
environmental processes they intend to represent. Although no consensus on model performance
criteria exists in the literature, numerous basic statements are likely to be accepted by most
professional modelers, as follows:
/ Models approximate reality and cannot precisely represent natural systems.
/ No single accepted test determines whether a model is validated.
/ Models cannot be expected to be more accurate than the sampling and statistical error
(e.g., confidence intervals) in the input and observed data.
An accurate numerical representation of the study area is the primary goal of the model application
effort because this representation determines whether the model results can be relied upon and used
effectively for decision-making. Using the weight-of-evidence approach (a mainstay of watershed
modeling) along with the targets and tolerances for model performance presented in this document
involves extensive qualitative and quantitative measures involving both graphical comparisons and
statistical tests under a wide range of environmental conditions. More specifically, the model
reproduces a continuous record of model predictions (on an hourly basis) across wet and dry time

QAPP-21
33

periods, back-to-back storms, and so on. The targets presented for model performance are derived
from extensive experience with the HSPF model and will ensure that the model application meets the
project objectives.

These considerations must be included in developing appropriate procedures for the QA/QC of the
models. Despite a lack of agreement on how the models should be evaluated, the following principles
provide a final set of evaluation criteria for the modeling projects:
/ Exact duplication of observed data is not possible, nor is it a performance criterion for projects.
The model corroboration (validation) process will measure the model’s ability to simulate
measured values.
/ No single procedure or statistic is widely accepted as measuring or capable of establishing
acceptable model performance; therefore, combined graphical comparisons, statistical tests,
and professional judgment are proposed to provide sufficient evidence to base a model
acceptance or rejection decision.
/ All model and observed data comparisons must, either qualitatively or quantitatively, recognize
the inherent error and uncertainty in the model and observations. Where possible, model
sensitivity and uncertainty will be documented as part of each modeling study.
A margin of safety will be established as part of the modeling process to compensate for model
limitations and assumptions and to gauge the impact on the usability of the results toward decision-
based management. This topic will be addressed in the modeling approach report.

QAPP-21
34

6.0 PROJECT REPORTS
The following reporting deliverables will be prepared under this project and submitted to NC DWR for
review and approval:
/ Model Selection memorandum
/ Modeling QAPP
/ Technical memorandum summarizing data collection activities
/ Model configuration/approach memorandum
/ Model calibration report
/ Final modeling report.
The contents of the first four deliverables are addressed with adequate detail in previous sections of
this QAPP.

The final modeling report will adhere to the high-level topics and organization outlined for final project
reports in the EPA's Template for Developing a Generic or Project-Specific Quality Assurance Project
Plan for Model Applications [EPA, 2009]; however, RESPEC will customize our final project report
subtopics and lower-level organization to more closely correspond to the specifics related to an HSPF
model application. The following high-level topics will be addressed:
/ Introduction and Background
/ Purpose of Modeling and Modeling Objectives
/ Observational Data Used to Support Modeling
/ Model Configuration
/ Model Parameterization (Calibration) and Corroboration (Validation)
/ Performance Against the Performance Criteria
/ Delivery Factors for Point-Source Discharges and Nutrient Offset Credits
/ Model Use Scenario Analysis and Results
/ Deviations From the QAPP
/ Conclusions, Recommendations, References, and Appendices.

QAPP-21
35

7.0 REFERENCES
Bicknell, B. R.; J. C. Imhoff; J. L. Kittle, Jr.; T. H. Jobes; and A. S. Donigian, Jr., 2001. Hydrological Simulation Program – FORTRAN, User's Manual for Release 12.2, prepared by AQUA TERRA Consultants, Mountain View, CA, for the U.S. Environmental Protection Agency, Ecosystem Research Division, Athens, GA, and the U.S. Geological Survey, Office of Surface Water, Reston, VA. Crawford, H. H. and R. K. Linsley, 1966. Digital Simulation in Hydrology: Stanford Watershed Model IV,
Technical Report No. 39, prepared by the Department of Civil and Environmental Engineering, Stanford
University, Stanford, CA. Donigian, Jr., A. S., 2000. "Lecture #19. Calibration and Verification Issues, Slide #L19-22," HSPF Training Workshop Handbook and CD, prepared by Aqua Terra Consultants, Mountain View, CA, for the U.S. Environmental Protection Agency, Office of Water, Office of Science and Technology, Washington, DC.
Donigian, Jr., A. S., 2002. "Watershed Model Calibration and Validation: The HSPF Experience," Proceedings, Water Environment Federation National Total Maximum Daily Load Science and Policy Conference, Phoenix, AZ, November 13-16. Donigian, Jr., A. S. and N. H. Crawford, 1977. Simulation of Nutrient Loadings in Surface Runoff With the
NPS Model, EPA 600/3 77/065, prepared by HydroComp, Inc., Palo Alto, CA, for the U.S. Environmental Protection Agency, Athens, GA. Donigian, Jr., A. S. and H. H. Davis, 1978. User’s Manual for Agricultural Runoff Management (ARM) Model, EPA-600/3-78-080, prepared by the U.S. Environmental Research Laboratory, Athens, GA.
Donigian, Jr., A. S. and J. C. Imhoff, 2009. "Evaluation and Performance Assessment of Watershed Models," Proceedings, Water Environment Federation Total Maximum Daily Load 2009, Minneapolis, MN, August 9-12. Duda, P. B.; P. R. Hummel; A. S. Donigian, Jr.; and J. C. Imhoff, 2012. "BASINS/HSPF: Model Use, Calibration, and Validation," Transactions of the American Society of Agricultural and Biological
Engineers, American Society of Agricultural and Biological Engineers, Vol. 55, No. 4, pp. 1,523-1,547. HydroComp, Inc., 1977. HydroComp Water Quality Operations Manual, prepared by HydroComp, Inc., Palo Alto, CA. Kenner, S. J., 2022. Selection of HSPF Watershed Model for the Neuse River, RSI(RAP)-
W0392.22001/11-22/15, prepared by RESPEC, Rapid City, SD, for the North Carolina Department of Environmental Quality, Raleigh, NC, November 29. (draft) Krause, P., Boyle, D.P., Bäse, F., 2005. “Comparison of Different Efficiency Criteria for Hydrological Model Assessment,” Advances in Geosciences, Vol. 5, pp. 89–97. https://doi.org/10.5194/adgeo-5-89-2005

Michael Baker, Jr., Inc.; Aqua Terra Consultants; and Dynamic Solutions, LLC, 2013. APPENDICES: Setup, Calibration, and Validation for Illinois River Watershed Nutrient Model and Tenkiller Ferry Lake EFDC Water Quality Model, prepared for the U.S. Environmental Protection Agency, Region 6, Dallas, TX. Available online at https://www.epa.gov/sites/production/files/2016-03/documents/irw_report_appendices_8_7_15_0.pdf

QAPP-21
36

National Pollutant Discharge Elimination System, 2020. “Stormwater Discharges From Municipal Sources,” epa.gov, accessed May 2, 2020, from https://www.epa.gov/npdes/stormwater-discharges-municipal-sources Sutherland, R. C., 2000. “Methods for Estimating the Effective Impervious Area of the Urban Watersheds,” Technical Note #28, Watershed Protection Techniques, Vol. 2, No. 1, pp. 282–284.
U.S. Environmental Protection Agency, 1993. Guidance Specifying Management Measures for Sources of Nonpoint Pollution in Coastal Waters, EPA-840-B-92-002, prepared by the U.S. Environmental Protection Agency, Office of Water Program Operations, Washington, DC. U.S. Environmental Protection Agency, 2000. Guidance on Systematic Planning Using the Data Quality Objectives Process, EPA 600-R-96-055, prepared by the U.S. Environmental Protection Agency, Office
of Environmental Information, Washington, DC. Available online at https://www.epa.gov/sites/production/files/2015-06/documents/g4-final.pdf U.S. Environmental Protection Agency, 2002. Guidance for Quality Assurance Project Plans for Modeling, EPA 240-R-02-007, prepared by the U.S. Environmental Protection Agency, Office of Environmental Information, Washington, DC. Available online at https://www.epa.gov/sites/
production/files/2015-06/documents/g5m-final.pdf U.S. Environmental Protection Agency, 2009. Template for Developing a Generic or Project-Specific Quality Assurance Project Plan for Model Applications, prepared by the U.S. Environmental Protection Agency, Office of Environmental Information, Washington, DC. Available online at https://www.epa.gov/sites/production/files/2015-07/modelqapptemplate2009.doc

QAPP-21
A-1

APPENDIX A
FORMULAS USED FOR STATISTICAL COMPARISON

QAPP-21
A-1

APPENDIX A: Formulas used for statistical comparison
The following formulas are used for statistical comparison of observed values (O ) to predicted values (P ). [Krause et al., 2005].5 Coefficient of Determination r2:

__
2 1
22__
11
n
iii
nn
iiii
OOPP
r
OO PP
=
==
    −−  =    −−  
∑
∑∑
(A-1)
Weighted r2:

2
2 1 2
for 1
for 1
br b
wr br b
−
×≤=×>
(A-2)
Nash-Sutcliffe Efficiency E:
( )
2
1 2_
1
1
n
iii
n
ii
OP
E
OO
=
=
−
= −−
∑
∑
(A-3)
E using ln(obs) and ln(sim):

To reduce the problem of the squared differences and the resulting sensitivity to extreme values, the
Nash-Sutcliffe efficiency E is often calculated with logarithmic values of O and P. Through the
logarithmic transformation of the runoff values, the peaks are flattened and the low flows are kept more
or less at the same level. As a result, the influence of the low-flow values is increased in comparison to
the flood peaks resulting in an increase in sensitivity on In E to systematic model over- or
underprediction.

Modified efficiency E:
( )
1
_
1
1 with
n j
iiijjn
ii
OP
E jN
OO
=
=
−
=−∈−
∑
∑
(A-4)

5 Krause, P., Boyle, D.P., Bäse, F., 2005. “Comparison of Different Efficiency Criteria for Hydrological Model Assessment,” Advances in Geosciences, Vol. 5, pp. 89–97. https://doi.org/10.5194/adgeo-5-89-2005

QAPP-21
A-2

Relative Efficiency E

2
1
2_
_1
1
n ii
i i
rel
n i
i
OP
OE
OO
O
=
=
−= −−
∑
∑
(A-5)
Index of Agreement d:
( )
2
1 2__
1
1 01
n
iii
n
iii
OPdd
POOO
=
=
−=−≤≤−+ −
∑
∑
(A-6)
Modified Index d
1
__
1
1 with
n j
iiij jn
iii
OP
d jN
POOO
=
=
−
=−∈−+ −
∑
∑
(A-7)
Relative Index d

2
1
2__
_1
1
n ii
i irel
n ii
i
OP
Od
POOO
O
=
=
−= −−+ −
∑
∑
(A-8)