Major IT system outages experienced by the federal border agency this past fall, which caused significant delays at airports and highway border crossings, were initially triggered by human error, an internal review indicates.
The Canada Border Services Agency (CBSA) experienced IT system outages between Sept. 28 and Oct. 5, which affected primary inspection kiosks at major airports, inspection systems in airports and at highway border crossings, as well as passenger information systems that enable airlines to board passengers.
Additionally, client service portals and electronic data exchange systems for commercial shipments by air, marine, rail, highway and postal services were “significantly impacted,” according to an Oct. 31 report by the CBSA and Shared Services Canada (SSC) that was publicly released this month.
Following the outages, Public Safety Minister Gary Anandasangaree had ordered the CBSA to conduct an internal review within 30 days to investigate the incidents and outline steps the agency would take to prevent future occurrences.
Hundreds of travellers faced long delays as CBSA kiosks began malfunctioning on Sept. 28, impacting airports across the country, including in Vancouver, Calgary, Toronto, Montreal, and Quebec City.
The CBSA previously told The Epoch Times that the failure was caused by “an unexpected technical issue during routine maintenance,” noting the agency worked to restore connectivity “as quickly as possible.”
The CBSA internal review revealed that an upgrade to the databases that support most CBSA traveller and commercial systems was initiated by SSC and was intended to be “non-disruptive” to service.
However, a prerequisite patch wasn’t applied to the databases prior to upgrade, leading the upgrade to fail and cause “significant corruption of live traveller and commercial data,” the report said, attributing the issue to “human error.”
“Subsequent system performance issues and intermittent outages continued until October 5, resulting in delays in airports for international air passengers, a week-long backlog of commercial shipments at highway border crossings, marine ports, and air and rail yards,” the report says.
The corruption also affected primary inspection lanes at the Canada-U.S. highway border crossings, causing “significant” wait times for commercial trucks, with ports of entry in Southern Ontario and Manitoba reporting delays of hours or days for trucks between Sept. 29 and Oct. 6.
The review revealed that a separate incident occurred on Sept. 29, involving an emergency security patch on CBSA firewalls implemented by SSC that caused a break in communications with some commercial airlines attempting to access passenger information systems.
The report says the CBSA didn’t receive proper notice of the security patch, so it wasn’t able to coordinate with airlines to prepare or find a less busy time to implement the patch. As a result, international and domestic air travel was disrupted as airlines experienced challenges and delays boarding passengers for flights.
The report lists “critical lessons learned” from the IT outages, as well as a corresponding action plan to prevent such incidents from occurring again. The plan includes actions that were to be completed last month, as well as deadlines for March and October of next year.
The CBSA and SSC will conduct a joint review to find ways to improve oversight, requisite training, and experience for IT employees, the report says. The agencies will also implement measures to “improve scrutiny, review, collaboration and communication to avoid similar human error in the future.”
Additional measures include reviewing the agencies’ IT change and incident management procedures, reviewing the CBSA’s IT applications and systems, strengthening the systems to correct single points of failure and ensure backups and redundancy are in place, and insulating front-line operations from back-end service disruption and outages.
