PHILADELPHIA – The City of Philadelphia is providing the following update, one week after the significant challenges to its computer networks and city service delivery posed by the global CrowdStrike outage that began last Friday, July 19.
Early last Friday, an unforeseen global outage impacted the City’s CrowdStrike platforms — major anti-virus tools. By 1:30 AM that day, the vigilance of the city’s Data Center Team and IT Command Center (ITCC) had detected the issue, setting the stage for a coordinated, rapid response.
City IT experts interacted closely all day, through the weekend, and the early portion of this week with the city’s Office of Emergency Management, which coordinated and tracked the city’s emergency response and readiness during this crisis. OEM activated the city’s emergency operations at 7:30 am that day, at the direction of Mayor Cherelle L. Parker and Managing Director Adam Thiel.
City security and platform engineering teams acted swiftly to identify and implement solutions, ensuring the integrity and security of our computer systems. Meanwhile, the support services team, Deputy CIOs, IT directors and managers, networking team, original equipment manufacturers (OEM), IT technicians and analysts, and numerous volunteers worked tirelessly to restore functionality to over 6,000 affected computer systems. Thanks to all these efforts, the majority of our data servers and core systems were operational by midday Friday. By Monday morning, all City operations were fully functional, showcasing the remarkable dedication and expertise of our IT personnel.
Normal operations resumed by midweek, and the city continues to maintain careful oversight of its systems and data. This experience underscores the critical importance of teamwork, preparedness, and swift action in managing IT crises.
“I am very proud of all the work of our IT professionals all weekend and all week, under the direction of Chief Information Officer Melissa Scott,” said Mayor Parker. “I’m just as proud of the work of all our emergency management staff under OEM Director Dominick Mireles. Working as one integrated government, the City of Philadelphia responded to this crisis, and worked every day to maintain our systems. We passed this test with flying colors.”
“I am immensely proud of our team,” said Melissa Scott, Chief Information Officer for the City of Philadelphia in the Parker Administration. “Their dedication and quick thinking in the face of this unprecedented challenge ensured that we could restore and secure our systems efficiently. It is remarkable to think about where we would be without the vigilance and expertise of our entire IT department.”
Meanwhile, the Office of Emergency Management, under Director Mireles, reported that as of July 24, 99 percent of city devices were restored. The remaining 1 percent are devices that may be with individual employees and not yet seen by IT staff, and other individual issues – IT is in a constant state of issue mitigation and recovery.
No critical impacts or service gaps were reported across governmental and non-governmental sectors this week.
“I am very pleased to see that our Standard Operating Procedure for coordination across all levels of public, private, and nonprofit stakeholders in a citywide event worked well,” Dominick Mireles, Director of the Office of Emergency Management said. “While critical issues were not present at a large scale, the City had a structure in place to work quickly and efficiently as one coordinated function to problem-solve, if needed. While the impacts of this event were not as bad as they could have been, it serves as a good reminder that as a business or an organization of any size, we need to have a solid continuity of operations plan in place – and as private individuals, we need to take our own cybersecurity and backup plans seriously. Right now, is a great time to visit www.ready.gov for more information from FEMA about how to prepare yourself or your business from a cyber incident in the future.”