NC0024945_Response to Notice_20221107

From:
To:
Cc:
Subject:
Date:
Sypolt, Shannon
Bell, Wes
Pitner, Andrew; Lockler, Joseph; Bolick, Jacob; Macomber, Maggie; DeWitt, Darrell; Miller, Craig; DeCristofaro,
Andrew
[External] CLTWater - Irwin Creek WWTP (NPDES NC0024945) Event - 5 Day Report
Monday, November 7, 2022 3:54:54 PM
CAUTION: External email. Do not click links or open attachments unless you verify. Send all suspicious email as an
attachment to Report Spam.
Mr. Bell,
Please let this letter serve as Charlotte Water's (CLTWater's) 5-day report for the event that
occurred at Irwin Creek WWTP (NC0024945) on Tuesday, November 1st. At approximately 06:50,
Irwin's ORC arrived at work and began checking SCADA data. He immediately noticed that the
effluent mag meter flow was not correct at the UV channels and began to investigate the situation.
First, he went to check the effluent mag meter and determined that everything was reading
properly. After checking the effluent mag meter, the ORC proceeded to the UV station and at 07:00
he noticed that the lamps were off in channel #2, which was in service, and running in channel #1,
which was out of service and had no flow. The ORC tried to operate the manual override switches
for the UV channels, but they didn't respond. The ORC also tried to reset a high priority alarm on the
UV PLC but the UV PLC was unresponsive and the ORC could not reset the alarm. Given the
unresponsiveness of the system, at approximately 07:05 the ORC made the decision to divert all
incoming plant flow to the EQ basin.
Irwin's Chief Electrical Engineer was contacted by the ORC and he was able to dial into the PLC but
discovered that the PLC was halted and unresponsive. The ORC unplugged the CPU processor inside
the PLC to complete a hard reset of the system. This cleared the settings for the UV program
including the water level targets and dosing setpoints. The ORC and Chief Electrical Engineer then
reentered the setpoints. The UV system reset was completed and returned to normal operation at
07:23.
CLTWater's investigation into this event consisted of three main parts. First, operators were
interviewed and asked to provide their accounts of the situation. Operators had physically checked
the UV system station at 19:40 on October 31st and again at 12:15 on Tuesday, November 1st.
Operators remembered physically seeing the UV lamps on at 12:15 and no alarms had occurred at
either the UV system station or on SCADA.
Second, a review of the SCADA system determined that when the UV PLC halted no alarm was
generated due to the halted alarm being tied to an "is connected" tag in the PLC which never lost
connection. An "is connected" tag is used in the SCADA system to alert operators when connection
to the system is lost. When the UV PLC halted, the last data was sent to SCADA and was
maintained. This led the operators to believe nothing was wrong on the UV screen within SCADA.
Third, a review of the power logic monitoring of the UV channels was conducted. It was observed

that when the PLC hafted both UV channels and all the lamp banks started at 100%. This is a default
setting in the UV program. Because the PLC program was halted and could not communicate with
the influent or effluent UV flow control gates, channel #1's influent gate did not open. Based on the
power monitoring, channel #2 continued to maintain UV lamps power until the flow conditions in
the plant dropped off at 05:05. At this time, plant effluent flow dropped off and a low water
condition was met due to the effluent gate position being stuck at 83% open. The low water level
alarm is hard coded into the UV program and is one of the few conditions that will shut down the
lamps at a water level of 36 inches or less regardless of operations.
Based on operator accounts, SCADA observations, and an investigation of the power logic
monitoring of the UV channels, CLTWater concluded that the UV lamps were off from 05:05 until
07:23 on Tuesday, November 1st, and approximately 838,543 gallons were discharged to Irwin Creek
during this time. Irwin's effluent fecal sample on Tuesday, November 1st, was collected at 09:27 and
was <1 cfu/100m1.
Unfortunately, CLTWater has been unable to determine any obvious causes for the CPU halt failure.
CLTWater will be replacing the UV PLC card's CPU and is in the process of installing a timer to check
the clock on each of the PLC's. In the future, if the clock stops moving this will trigger an alarm on
SCADA and on the operator phones. If you have any questions concerning the information
contained in this report, or if you need any additional information concerning this event, please feel
free to contact me directly. Thank you.
Respectfully,
Shannon Sypolt
Water Quality Program Administrator
Environmental Management
CHARLOTTE W6TER
4222 Westmont Drive / Charlotte, NC 28217
P: 704-336-4581 / C: 704-634-6984 / charlottewater.org