Trusted Professional

TIGTA: IRS System Outage Caused by Obsolete Software

ComputerHell The April 17 system outage Treasury Inspector General for Tax Administration
"There was no detailed discussion regarding individual fixes and whether these defects and associated updates could affect the IRS environment," said the report. "Documentation of the monthly microcode meetings is limited to a single spreadsheet that identifies the recommended microcode updates and an e-mail sent to stakeholders containing high-level comments about the meeting. The meeting documentation lacks an agenda, meeting minutes, a list of attendees, and any relevant documentation about decisions made."

While IRS systems have now been upgraded to account for this bug, TIGTA said the agency has not taken appropriate steps to prevent another such outage in the future. In particular, TIGTA pointed to "the Tier 1 storage architecture," which includes mainframes, storage arrays, computer hardware and software maintenance and related services. Three of the storage arrays, located at the Martinsburg Computing Center, were singled out as a potential point of failure. Although they host a portion of core tax processing data, TIGTA said "there are no automatic failovers or built-in redundancies" for these storage arrays. Further, not all of the applications and system data there are fully replicated at a backup computing center in Memphis. 

TIGTA recommended that the chief information officer should: 1) document lessons learned and implement a corrective action plan, 2) formalize the monthly microcode bundle meetings, 3) ensure that decisions not to install microcode bundle updates are documented and approved, and 4) seek liquidated damages from the contractor and make modifications to the Enterprise Storage Services contract. The IRS agreed with all of the recommendations and has formalized the monthly microcode bundle meetings and sought damages from the Enterprise Storage Services contractor. The IRS plans to document a consolidated action plan, ensure that decisions not  to install microcode bundles are approved, and make modifications to the contract.