Organizational Success Depends on Data Management
Every operation of every organization today involves data. And so, the management of that data will factor into the success or failure of the organization in its mission. This is not entirely new. Effective use of data has always played an important role in organizational outcomes.
What has changed over time is the size and complexity of data, hence the notion of big data. In years past we knew where our data was, and we knew how it was organized. It was in the filing cabinet organized by topic and alphabetized. The volume and complexity of today’s data far exceeds that of the past. As a result, many organizations don’t have well-organized and usable data. They don’t even know how it is organized, where it might be, or even what data they have. Clearly, they cannot fully utilize their potential data assets.
The volume and complexity of today’s data far exceeds that of the past. According to a study by International Data Corporation,[1] the Global Datasphere consists of 59 zettabytes that will be generated, copied, and consumed within 2020. A zettabyte is one thousand exabytes, one million petabytes, or one billion terabytes. Data growth is projected at 26% compounded.
But what to do with all this data? Will it be an asset or a burden? Will it inform or confuse us? The purpose of this book is to provide a clear path toward converting your potential data burden into a valued asset, leading to mission success.
How will this happen in the broader context of your organization? Let us take the example of military organizations. In any military conflict, an organization will win given that it has a winning strategy among its available strategies and given that it recognizes that winning strategy. So, success depends in part on having talented strategists that can recognize when a winning strategy is at hand.
What then determines the universe of all strategies available to the organization, including that winning strategy? There are five factors.
The Five Strategic Factors
1) What you have
2) What you know
3) What the opposition has
4) What the opposition knows
5) The environment
What you have includes physical assets, their attributes, and quantities. This includes personnel.
What you know is your data, its quality, quantity, and organization. Likewise for the opposition’s physical and data assets.
Finally, the environment consists of everything else in the world, not in your or your opposition’s possession, that constrains or enhances your universe of possible strategies. It could include a river, a mountain range, the weather, or popular opinion.
We tend to think of strategizing as the process of inventing strategies, when in fact all possible strategies are predetermined by the five factors. The job of the strategist is to discover more than it is to invent. He must discover winning strategies in the universe of available strategies.
So far so good and it seems quite simple. But it is really a little bit more complex because of the interrelationships between the factors, particularly between knowing and the other factors. If you have something but you don’t know it, it can’t be formulated into a strategy. If the opposition has something but you don’t know it, it can’t be formulated into a strategy. If some aspect of the environment constrains the available strategies and you don’t know it, then it can’t be formulated into a strategy.
The most surprising relationship is between knowing and itself. If you know something but don’t know that you know it, then it can’t be formulated into a strategy. This may sound flippant or even impossible. However, knowing what you know is for real and will be explained shortly.
I’ll mention one more subtlety on strategies. A strategist also plays the role of statistician. Facts are rarely known for certain, so the strategist must weigh the probability of his given data being true. He must also consider possibilities not explicit in his data at all. What other possibilities are there?
This book will not delve into statistical reasoning by strategists. We will leave that to some excellent book on strategizing. We will cover related topics, however, such as quality assurance and quality control. Nothing is worse than basing a strategy on false information.
If your organization is not a military organization, then you may wonder how applicable this framework is to you. Consider the fact that all organizations must have a strategy for mission success. All organizations have both physical and data assets, and all organizations have an environment that they operate in. For some, the only difference might be that there is not a clearly defined opposition. But even so, most organizations do face other organizations that either oppose them or compete with them in some fashion.
Historical examples demonstrate the various ways that success depends on data. I’ll use military and national security examples here due to the fact that the outcomes are unmistakable. Take for instance the battle of the Little Big Horn, between the 7th Cavalry Regiment, U.S. Army under George Custer and the Lakota, Northern Cheyenne, and Arapaho tribes. Custer was a highly successful civil war officer, promoted to the rank of general at the very young age of 23. He was not foolhardy and reckless as some would later claim but rather “meticulously scouted every battlefield, gauged the enemy’s weak points and strengths, ascertained the best line of attack and only after he was satisfied was the 'Custer Dash' with a Michigan yell focused with complete surprise on the enemy in routing them every time.”[2] During the Indian wars, he acquired an extensive knowledge of Indian customs and tactics.
However, on June 25, 1876, Custer and his men were missing what is called situational awareness, an understanding of the current size and positioning of enemy forces. It was enough to cost them the battle and most of their lives.
A more positive example comes from the June 1942 Battle of Midway, an air and naval battle during WWII. The U.S. forces were woefully unprepared in terms of equipment and tactics. Their early WWII aircraft were inferior to the Japanese Zeros, and U.S. torpedo plane tactics had pilots flying directly into Japanese anti-aircraft guns. The few American torpedo bombers that managed to avoid getting shot out of the sky ended up delivering dud torpedoes that failed to detonate on impact.
In spite of these deficiencies, the Americans had one advantage. Their cryptographers understood the Japanese codes well enough that they knew the Japanese were coming. As a result of this data, the Americans positioned their own fleet northeast of Midway Island, exactly where the Japanese hoped they would not be. In the end, Japan lost as many as 3,000 men (including more than 200 of their most experienced pilots), nearly 300 aircraft, one heavy cruiser, and four aircraft carriers, while the Americans lost the carrier Yorktown and the destroyer Hammann. The Japanese never fully recovered from their loses and so Midway became the turning point in the Pacific war.
From these two examples, the reader might conclude that having certain data is always the crucial thing. This is a common misconception as the next two examples will show.
The battle leading up to Midway was the infamous December 7, 1941, attack on Pearl Harbor. Generally considered to be a surprise attack, it was anything but. In the week leading up to the attack, the Japanese changed their radio call signs at an irregular time, Japanese ships were on the move, American intelligence had lost track of four Japanese carriers, and the Navy Department radioed officials at Pearl Harbor authorizing them to destroy secret documents at various bases. In addition, a Japanese mini-sub was sunk by Americans just outside Pearl Harbor the morning of the attack. Perhaps the most glaring indication of the pending attack was the detection of the Japanese planes on American radar. Neither the existence of the mini-sub nor the planes on radar was reported to the Army command on Oahu.
So, while the crucial data was in hand, it was not acted on because there was no plan in place to notify key actors and for those key actors to carry out pre-planned actions. Such a plan is part of a notification system, which we will discuss later. Pearl Harbor was not a case of missing information. It was a classic case of data mismanagement. All this happened prior to the zettabyte data era, which shows that even organizations with modest data assets require data management.
Now fast forward to September 11, 2001. This is an age of sophisticated computing, data collection, and analysis. Such sophistication allowed American intelligence and national security organizations to collect extensive information predicting the attack that included the Trade Center, the Pentagon, and Flight 93.
Predictive data included:
A CIA briefing to President Clinton in December 1998 that al-Qaeda was preparing terrorist attacks on the U.S. that might include hijacked aircraft.
In April 2001 anti-Taliban leader of the Northern Alliance in Afghanistan, Ahmad Shah Massoud, gave a speech before the European Parliament in which he conveyed that a large-scale terrorist attack on the U.S. was imminent. Massoud was assassinated by al-Qaeda two days before 9/11.
British intelligence leaders were aware that a major attack was coming. This was conveyed by the British to the Americans in June 2001.
June 29, 2001, the President’s Daily Brief by the CIA reiterated that attacks were anticipated near term and expected to have dramatic consequences.
July 10, 2001, National Security Advisor Condoleezza Rice received a top secret briefing on communication intercepts showing increased likelihood of the attack.
August 6, 2001, the President’s Briefing entitled “Bin Laden Determined to Strike in U.S.” It included information from the FBI indicating hijackings or other types of attack. Rice discounted the report.
Foreign sources included Saudi Intelligence, the Malaysian Special Branch, and Jordan. Jordan even conveyed the codename of the operation, “The Big Wedding,” and that it involved aircraft.
Senior terrorism official Richard Clarke wrote after the attack that “Somewhere in the CIA there was information that two known al Qaeda terrorists had come into the United States. Somewhere in the FBI there was information that strange things had been going on at flight schools in the United States... They had specific information about individual terrorists from which one could have deduced what was about to happen. None of that information got to me or the White House.” U.S. agencies were not sharing information.
The NSA and the CIA were aware of meetings between 9/11 plotters but decided it was not a matter for the FBI.
On July 13, a CIA agent assigned to the FBI’s Counterterrorism Center requested permission to inform the FBI that 9/11 actors were in the U.S. The CIA never responded.
Mid-August, one Minnesota flight school alerted the FBI about Zacarias Moussaoui, who had asked “suspicious questions.”
In July, Phoenix based FBI agent, Kenneth Williams, sent a message to FBI Headquarters and New York FBI agents on “the possibility of a coordinated effort by Osama bin Laden to send students to the United States to attend civil aviation universities and colleges.”
So, while the data was extensive and highly predictive, it failed to have any impact whatsoever. This was partly due to the lack of data sharing and data integration, best practices that we shall discuss later, but also the lack of a notification system, as in the case of Pearl Harbor. 9/11 was a classic case of an abundance of predictive data, neutralized by poor data mismanagement.
Your own organization may never face a crisis as great as 9/11, or maybe it will. Whether your organization is military, government non-military, private sector, or non-profit, the same principles apply. All organizations have a mission, they should all have a strategy, and whether they have a winning strategy available depends on the five strategic factors.
The examples of Pearl Harbor and 9/11 show that having data is not the same as knowing, at least not in the way we want to know. It turns out that there are four levels of knowing.
The Four Levels of Knowing
1) Owning data
2) Knowing what data you have
3) Understanding your data
4) Knowing what your data means in the context of other data
These are additive levels in the sense that level n includes all the lower levels.
Owning data includes both having it on servers under your jurisdiction and access for those pursuing your mission. This second part is often missing, as vendors you hire or data hoarders within your organization might withhold data. If you can’t get to it, you don’t really own it.
Level 3 pertains to understanding bits of information in isolation, whereas Level 4 provides greater insights. Level 3 is something that can be gained via something called a data dictionary, whereas Level 4 is determined by the design of your databases and data systems.
In this book, you will learn how to gain Level 4 knowledge from your data. In the case of Pearl Harbor and 9/11 we had Level 1 knowledge.
1 https://www.idc.com/getdoc.jsp?containerId=IDC_P38353
[2] Marguerite Merrington, The Custer Story In Letters. (Lincoln, NE: University of Nebraska Press, 1987).
