‘Big Data’ itself is not new, nevertheless how the world of technology approaches the problem is in the early lifecycle stages of propaganda, and whilst the technology vendors jostle to gain the attention of their target audience the real question of what you will do with such a technology, and more importantly who will be dealing with the intellectual underlying issues is yet to play out.
Big Data is, in very many ways, data mining and analytics, but on a much bigger scale. At its core it is a combination of Business Intelligence / Data Warehousing, and Search. None of this is new, however the technology companies make this appear so because large organisations have huge amounts (petabytes plus plus plus) of structured and unstructured data.
The objective of big data is content analysis, or trying to identify repeatable business trends and patterns in an effort to make better-informed business decisions. Many organisations feel that hidden within their repositories of data are gems that could provide significant competitive advantage, and operational knowledge.
The flip side is organising these structured and unstructured repositories, ensuring proper metadata, master data, and data dictionaries is being used: not just keywords which are relevant to the individual rather than relevant to the organisation as a whole.
A further problem with big data is that it the quantity is so enormous it cannot simply be dropped into a database or spreadsheet for queries to be run against it. It requires extraordinary computing power to undertake these interrogations, and not run on the core systems that might interfere with normal operational process.
Over the past decade Information and Records Managers were concerned about the scale of EDRM solution. Now that major corporates have identified the not getting their house in order can bring about serious commercial failures the concerns over EDRM will seem like a minor issue.
Big Data is where Corporate Governance, Risk, and Compliance meets Information & Records Management. Are Information and Records Managers ready to meet that challenge?