The Harvard Business School presents the case “CareGroup,” which tells the remarkable story of how the advanced information technology (IT) systems of the health-care company CareGroup suddenly crashed for three and a half days. It details the steps taken to recover from this incident and the invaluable lessons learned. On October 1, 1996, three distinguished hospitals in eastern Massachusetts merged to establish CareGroup, a collection of health-care professionals providing community-based primary care and a wide range of specialized services.
The formation of CareGroup was a response to three important factors in the healthcare industry during the mid-1990s. Firstly, many hospitals had joined forces in order to gain contracting power against HMOs, due to the intense competitive environment within the industry. Secondly, hospitals were under pressure to strengthen their balance sheets amidst a complicated price war caused by excess supply. Lastly, the merger of hospitals provided an opportunity to create integrated services that could significantly enhance healthcare quality and reduce costs. The merger that formed CareGroup had been a successful endeavor.
Despite facing challenges on the road to success, the company managed to generate $1.6 billion in revenue and emerged as the second-largest group of hospitals in eastern Massachusetts. A remarkable accomplishment was the creation of an integrated IT system that connected the entire organization. This positioned CareGroup as a top innovator in IT, renowned not only for its excellence in health care but also as a leader in any industry.
The healthcare company had superior e-mail, voice, wireless, data center, and web-based infrastructure. Their IT systems effectively managed 900,000 patient records from 1977 onwards. Additionally, 3,000 physicians and 200 staff members processed 40 terabytes of data daily. CIO John D. Halamka played a vital role in the organization. His impressive background and extensive experience made him the most suitable CIO for CareGroup. Halamka states that his success is attributed to his comprehensive knowledge of technologies, proficiency in 12 programming languages, and authorship of books on Unix system administration.
As a doctor, I possess knowledge in both the clinical domain and technical requirements. However, it is important to mention that despite my experience in wiring telephone closets and building servers and desktops, I have never worked on a project beyond a home network. This is a fact that I acknowledge. Nevertheless, I have recently taken on such a project. The weaknesses of CareGroup, which ultimately led to the collapse of their network, became evident during the aftermath of the catastrophic event. CareGroup’s previous belief that they had the most advanced network in the healthcare industry unintentionally caused them to overlook the importance of keeping their IT system updated and controlled.
CareGroup’s systems were solely dependent on one switch that was managed by one person, making them vulnerable to a single point of failure. The company lacked a dedicated team to monitor and control the ongoing IT changes and their impact on the network as a whole. However, the merger that created CareGroup presented an opportunity to combine the resources of individual hospitals and enhance the quality of care. Recognizing this opportunity, the company developed an advanced IT system.
Regrettably, CareGroup failed to effectively address their threats before the network collapse despite their success in seizing opportunities. The rapidly evolving nature of the IT industry poses a threat that no company, regardless of the sophistication of their network, should overlook. Cisco’s overlooked services for monitoring external factors were an important opportunity for the company and played a critical role in the recovery of the system. Cisco possessed a skilled engineering team capable of efficiently reviewing all IT system changes and preventing future network issues.
CareGroup’s IT organization was decentralized, with each hospital running on its own legacy systems prior to the merger. At Beth Israel Deaconess Medical Center, operations were further complicated by the need to integrate the systems from all the merged hospitals. These systems were outdated, not seamlessly connected, and incurred significant expenses. By 2002, all CareGroup hospitals had been consolidated into one system utilizing Meditech, the most advanced software available among non-medical center hospitals.
During Halamka’s leadership, CareGroup’s IT system had quickly become the most advanced in the healthcare industry by 2003. Information Week recognized CareGroup’s IT organization as the number one in America in September 2001 and had honored them as one of the top 100 companies for three consecutive years before that.
The architecture of IS, which resulted in the network collapse, was completely centralized and relied on a single main switch. Smaller networks were incrementally added, causing the network to exceed its specifications. The collapse occurred when a CareGroup researcher accidentally left his experimental software open, resulting in massive data transfers that overwhelmed the company’s main switch.
Because all networks were connected to this switch, all messages on the network started to rapidly repeat and reproduce in an infinite loop, ultimately causing the entire network to become disabled. The IT department at CareGroup faced a major challenge in diagnosing the problem as the primary issue was that nothing was functioning properly.
Initially, everyone had their own thoughts on the problem and hastily tried to implement their own potential solutions. However, this approach only made the problem more complicated and difficult to solve when their changes didn’t work. Within 24 hours, Halamka and his IT department recognized that their efforts were actually counterproductive. To address the issue, Halamka enlisted Cisco’s SWAT team, granting them full control and responsibility. On-site engineers worked tirelessly alongside global teams, offering round-the-clock focused care and attention to resolve the problem.
Another crucial decision that greatly influenced CareGroup’s successful recovery was the choice to continue using backup procedures. Halamka recognized that it was possible to operate an entire hospital effectively using paper records. This meant that the hospital staff had to duplicate or even triple their work by switching between manual and system-based processes. However, three and a half days after the devastating crash, Cisco was able to restore the network, and the interruption to CareGroup’s service for its patients was minimal. Remarkably, there were no instances or evidence of any negative consequences resulting from the network outage.
The collapse of CareGroup’s network exposed the negative consequences of not keeping up with advancements in technology and constantly seeking improvement. CareGroup had previously been successful in the field of IT but had made little effort to continue progressing. In an article by Halamka (2009) discussing the incident, he compares the situation to the “broken window effect.” He uses the example of a perfect neighborhood called Lake Wobegon, where everything surpasses average expectations. However, when a baseball breaks a window and the owner chooses not to repair it immediately, the house begins to look neglected. As a result, a neighbor decides to abandon a junked car on the street, further deteriorating the appearance of the neighborhood.
Then a bit of graffiti isn’t cleaned up. Folks let garbage pile up in yards. The disorder gives rise to discourtesy and, eventually, crime. Halamka has responded to the lessons of IT by imposing, or allowing Cisco to impose, a zero tolerance policy that the organization did not have before. By rigorously and consistently examining every incident as it happens, Halamka has developed a new IT culture that encourages constant improvement based on the sharing of experiences. This ensures that broken windows are fixed and recurrences are rare.
Is the CareGroup IT strategy too cautious? The stringent change-control board must approve all IT actions, raising this question. Innovation carries the risk of failure. CareGroup’s network collapse serves as a reminder that thorough precautions, constant review, and monitoring are necessary for innovation. Halamka emphasizes the importance of remaining vigilant to avoid stagnation and promote growth.
References
- Halamka, J. (2009). IT and the Broken Window Effect. Computerworld, 43(9), 19. Retrieved from Academic Search Premier database.
