FCC Commissioners stressed today the importance of preventing false alerts like the ballistic missile alert sent in error in Hawaii earlier this month (TR Daily, Jan. 16), saying that the incident there should lead to best practices that can be used by alert originators and other stakeholders. Their comments came as FCC staff reported that the alert was sent in error due to a misunderstanding about whether there was actually a ballistic missile attack.

Meanwhile, Hawaii officials released a report today that concluded that “insufficient management controls, poor computer software design, and human factors” contributed to the false alert and the delayed response in correcting it. The report echoed conclusions in a preliminary report released today by the FCC’s Public Safety and Homeland Security Bureau.

At today’s monthly FCC meeting, Commissioners received the preliminary report from the Public Safety Bureau on its investigation of the early-morning false alert, which was transmitted through the Emergency Alert System (EAS) and wireless emergency alerts (WEAs). The alert was transmitted shortly after 8 a.m. on Jan. 13 and it took officials 38 minutes to send out a corrected alert. The information provided by the bureau today was the most detailed yet from the FCC about the incident. The bureau stressed that its investigation is ongoing.

James Wiley, an attorney-adviser in the Public Safety Bureau’s Cybersecurity and Communications Reliability Division, explained how the false alert was sent during a ballistic missile alert drill that took place during a change from the midnight to the day shifts at the Hawaii Emergency Management Agency (HI-EMA). Due to a “miscommunication,” “the day shift supervisor was not in the proper location to supervise the day shift warning officers when the ballistic missile defense drill was initiated,” he said.

“At 8:05 a.m., the midnight shift supervisor initiated the drill by placing a call to the day shift warning officers, pretending to be U.S. Pacific Command,” according to Mr. Wiley. “The supervisor played a recorded message over the phone. The recording began by saying ‘exercise, exercise, exercise,’ language that is consistent with the beginning of the script for the drill. After that, however, the recording did not follow the Hawaii Emergency Management Agency’s standard operating procedures for this drill. Instead, the recording included language scripted for use in an Emergency Alert System message for an actual live ballistic missile alert. It thus included the sentence ‘this is not a drill.’ The recording ended by saying again, ‘exercise, exercise, exercise.’ Three on-duty warning officers in the agency’s watch center received this message, simulating a call from U.S. Pacific Command on speakerphone.

“According to a written statement from the day shift warning officer who initiated the alert, as relayed to the Bureau by the Hawaii Emergency Management Agency, the day shift warning officer heard ‘this is not a drill’ but did not hear ‘exercise, exercise, exercise.’ According to the written statement, this day shift warning officer therefore believed that the missile threat was real. At 8:07 a.m., this officer responded by transmitting a live incoming ballistic missile alert to the State of Hawaii. … Other warning officers who heard the recording in the watch center report that they knew that the erroneous incoming message did not indicate a real missile threat, but was supposed to indicate the beginning of an exercise.”

But Mr. Wiley noted that the day shift warning officer has refused to speak to the FCC. “Because we’ve not been able to interview the day shift warning officer who transmitted the false alert, we’re not in a position to fully evaluate the credibility of their assertion that they believed there was an actual missile threat and intentionally sent the live alert (as opposed to believing that it was a drill and accidentally sending out the live alert),” he said.

Officials immediately knew that a false alert had been sent, but it was not corrected with another alert for 38 minutes, although officials updated social media accounts before sending a corrected alert.

“Based on our investigation to date, the Bureau believes that a combination of human error and inadequate safeguards contributed to this false alert,” Mr. Wiley said. “With respect to human error, due to a miscommunication between the midnight shift supervisor and day shift supervisor, the drill was run without sufficient supervision,” he said.

“With respect to inadequate safeguards, most importantly, there were no procedures in place to prevent a single person from mistakenly sending a missile alert to the State of Hawaii,” Mr. Wiley added. “While such an alert addressed a matter of the utmost gravity, there was no requirement in place for a warning officer to double check with a colleague or get signoff from a supervisor before sending such an alert. Additionally, the State of Hawaii appears to have been conducting an atypical number of no-notice drills, which heightened the potential for an error to occur. TheBureau’s investigation so far has revealed that while other emergency management agencies use no-notice drills under special circumstances, their common practice is to schedule drills in advance for a set date and time.

“It is also troubling that Hawaii’s alert origination software did not differentiate between the testing environment and the live alert production environment,” according to Mr. Wiley. “Further,Hawaii’s reliance on prepared templates stored in their alert origination software made it easy for a warning officer to click through the alert origination process without sufficient focus on the actual text of the alert message that he or she was about to send. In contrast, the Bureau’s investigation so far has revealed that common industry practice is to host the live alert production environment on a separate, user-selectable domain at the log-in screen, or through a separate application. Other alert origination software also appears to provide clear visual cues that distinguish the test environment from the live production environment, including the use of watermarks, color coding, and unique numbering.

“Once the false alert was sent, the error was worsened by the delay in authoritatively correcting the misinformation,” Mr. Wiley noted. “The Hawaii Emergency Management Agency had not anticipated the possibility of issuing a false alert and, as such, had failed to develop standard procedures for its response. It first sent out a correction using social media, rather than the same alerting systems that it used to transmit the false alert. Indeed, the agency was not immediately prepared to issue a correction using these systems. The agency also did not maintain redundant and effective means to communicate with key stakeholders during emergencies.

“The Bureau is pleased that the Hawaii Emergency Management Agency has already taken steps to help ensure that an incident like this never happens again. It has created a new policy that supervisors must receive advance notice of all future drills. It will require two credentialed warning officers to sign in and validate the transmission of every alert and test. It has created a false alert correction template for Emergency Alert System and Wireless Emergency Alert system messages so that warning officers are more readily prepared to correct a false alert, should one ever occur again. It has requested that its alert origination software vendor integrate improvements into the next iteration of its software to more clearly delineate the test environment from the live production environment, helping to safeguard against false alerts,” Mr. Wiley said. “And finally, it has stopped all future ballistic missile defense drills pending the conclusion of its own investigation.”

But he added that “there is more work to be done. The Bureau will continue its investigation and issue a final report, including recommended measures to safeguard against false alerts and to mitigate their harmful effects if they do occur. And once we have developed these recommended measures, we intend to partner with FEMA to engage in stakeholder outreach and encourage implementation of these best practices. Among other avenues, we are considering convening roundtable with stakeholders in the emergency alerting ecosystem to discuss the lessons that should be learned from this incident as well as developing a joint webinar with FEMA to further educate stakeholders.”

“In my view, here are the two most troubling things that our investigation has found so far: (1) Hawaii’s Emergency Management Agency didn’t have reasonable safeguards in place to prevent human error from resulting in the transmission of a false alert; and (2) Hawaii’s Emergency Management Agency didn’t have a plan for what to do if a false alert was transmitted,” FCC Chairman Ajit Pai said. “Every state and local government that originates alerts needs to learn from these mistakes.  Each should ensure that it has adequate safeguards in place to prevent the transmission of false alerts, and each should have a plan in place for how to immediately correct a false alert. The public needs to be able to trust that when the government issues an emergency alert, it is indeed a credible alert.  Otherwise, people won’t take alerts seriously and respond appropriately when a real emergency strikes and lives are on the line.

“Today’s preliminary report is not the end of our work on this issue, but the beginning,” Mr. Pai added. “In the weeks to come, the Bureau will produce a final report on this incident.  And the FCC will work with federal, state, and local officials to explore appropriate actions and/or develop best practices.  We want to minimize both the chances of future false alerts being issued as well as the impact of any such false alerts.”

The Public Safety Bureau’s probe reveals “that at many levels this mistake could have been avoided and its effects could have been mitigated.  Now we need to take these facts and use them to improve our emergency alert systems across the board,” Commissioner Jessica Rosenworcel said. “We can start by considering how this agency can help develop best practices at the local, state, and federal level.  Then we need to incentivize their use through the Emergency Alert System state plans, which are subject to regular filing and review at the FCC.  While we’re at it, we should address everything from state training to improved user interfaces that reduce the likelihood of error. In addition, we should explore the viability of offering alerts to audio and video streaming services and the possibility of aligning traditional daisy chain reporting practices with newer federal alert aggregation capabilities.”

Ms. Rosenworcel stressed that “we need to act with dispatch.  We need real changes in place on an accelerated schedule.  We should commit right here, right now, to having them in place before the Summer begins — because what happened in Hawaii should never happen again.”

“The false ballistic missile alert in Hawaii should be a wake-up call for all stakeholders involved in emergency communications,” said Commissioner Mignon L. Clyburn. “We cannot dismiss this as being just an inadvertent mistake that only public safety officials in Hawaii need to address. This incident should serve as a catalyst for communities in every state and locality, to review their emergency alert processes.  Every community should be doing more, to prevent an issuance of a false alert.  But if and when a false alert is ever sent again, the technical capability to immediately send a correction should be in place, and the protocols on how to go about that, should be clearly defined.”

“Many residents of Hawaii thought that those 38 minutes were their last,” Commissioner Brendan Carr noted. “The panic and fear and heartache of those 38 minutes we now believe was due to human error but also deficient preparation and training. No one ever should have to go through moments like those — especially if basic competency would have prevented them. The people of Hawaii are justifiably livid. They demand answers. And so do we. … We will get to the bottom of this incident, and it is incumbent upon all of the relevant agencies of our government to make certain that this does not happen again.”

Commissioner Mike O’Rielly posed questions for FCC staff to highlight the fact that the Federal Emergency Management Agency is responsible for what occurs in the alerting system before providers transmit them to the public.

“The FCC is responsible for the distribution by the communications service providers,” noted Lisa Fowlkes, chief of the Public Safety Bureau, while FEMA oversees the Integrated Public Alert and Warning System (IPAWS). She noted that it is up to state and local officials to decide what alerts to issue and when. They are then forwarded to FEMA, which transmits them to providers. Ms. Fowlkes acknowledged that while FEMA has a statutory responsibility to deal with alert originators, the FCC in the past has convened stakeholders in the alerting system to identify lessons learned and best practices.

Mr. O’Rielly also noted that FEMA oversees an IPAWS advisory committee, which Congress created in the IPAWS Modernization Act of 2015 (S 1180), which President Barack Obama signed into law in 2016 (TR Daily, April 11, 2016).

During a news conference after the meeting, Mr. O’Rielly stressed that best practices, outreach, and coordination are already required by FEMA under the law.

But Mr. Pai agreed that the FCC “has historically exercised somewhat of a convening role to get all of the stakeholders in one place” on alerting matters. Mr. Pai and Ms. Fowlkes also declined to comment on whether the FCC could take enforcement action related to the false alert or even if it has the authority to do so.

Ms. Rosenworcel told reporters that the FCC’s requirement that states file EAS plans can be used “to compel states to follow best practices that we identify as a result of our investigation so that this never happens again.”

During a news conference in Hawaii today to release a report commissioned by Gov. David Y. Ige (D.), officials and the report described a dysfunctional operating environment at the HI-EMA. Officials said that the employee who sent the false alert in error was fired last Friday, an employee who was responsible for part of the protocols used to send drills and helped develop a checklist used for ballistic missile alert drills is being suspended, and two employees have quit, including Vern Miyagi, the administrator of HI-EMA and the No. 2 official.

Retired Hawaii National Guard Brig. General Bruce Oliveira, who conducted the investigation, said that because the drill was conducted during a shift change, “there was confusion” about who was “in charge and who needed to do what procedure.” He noted that the employee who sent the alert reported not hearing that it was only a drill. That employee “has been a source of concern for the same SWP [State Warning Point] staff for over 10 years,” according to the report. On at least two prior occasions, the employee “confused real life events and drills,” the report said.  One involved a fire drill and another a tsunami drill, Mr. Oliveira said. The employee had been counseled due to his poor performance, the report said.

The name of the terminated employee will be released once his appeals are complete, said Maj. Gen. Joe Logan, adjutant general of the Hawaiian Department of Defense and director of HI-EMA.

“The protocol was not in place. The protocol was very general. It was not detailed,” he added. He said that the protocol “left a lot of room for interpretation.”

The 13-page report, which said the false alert was sent at 8:06 a.m. and received on smartphones of SWP staff at 8:07 a.m., said the missile warning checklist was not detailed enough and did not have any protocol to follow in the event of a false alert, an issue that staff had raised in the past. The report also raised questions about whether employees had adequate training and said adequate training records did not exist. And it said that HI-EMA incorrectly thought it had to consult with FEMA before sending a corrected alert.

Among the recommendations in the report are installing a computer process so alerts can be quickly canceled, developing deactivation procedures, and modifying the checklist so it takes two people to send an alert and they repeat all directives and actions. The first two actions have already been taken. The report also recommends increased training for employees.

Gov. Ige noted that he signed an executive order authorizing Brig. General Kenneth Ha, deputy adjutant general, to conduct a “comprehensive” review of HI-EMA and alerting in the state. He said the report released today will serve as “a starting point” for that initiative.- Paul Kirby, paul.kirby@wolterskluwer.com

Courtesy TRDaily