Organizations are like people when it comes to collecting. Consciously or unconsciously, a lot of data is stored. More and more applications and devices are creating mountains of log data that are being filed away somewhere. According to Statistica, 149 zettabytes of data were stored worldwide last year. About 65 percent of that is never used again after storage. Dark data are called that forgotten data. In organizations, this includes data from production processes, business knowledge in documentation or financial data, for example.
AI comes to the rescue
Just as sifting through my moving boxes is a time-consuming chore that I'd rather not get into for a variety of reasons, analyzing dark data was complicated and costly for years. The business case for doing it was lacking because it wasn't clear what new value could come from such a process. Now, however, there is AI that can make that job a lot easier in the digital world. With trained Large Language Models (LLMs) and natural-language processing, petabytes of data can be searched for valuable business insights in a short period of time.
Data management lowers risks
While doing so, it is important to examine whether the data that is going to be analyzed is reliable. However, there are good data management platforms today that ensure that the policies that management establishes in data governance are applied.
Dark data not only presents an opportunity, but also a risk. Uncategorized data increases the chances of violating laws and regulations, for example when it comes to privacy protection. It is also important to pay attention to dark data from a societal perspective. After all, storing enormous amounts of unused data increases the carbon footprint of organizations. Inventory of dark data is therefore not just a matter of management, but a strategic factor for responsible IT policy.
I think I’ll take a proper look through my storage myself. Perhaps I'll get new insights while sifting through my history, stored in physical objects or digital crumb trail. Perhaps I will find some compromising or inspiring correspondence. And if not, I'm sure a good clean-out will save money on my next move, which is just around the corner.