Big data refers to the dynamic, large and disparate volumes of data being created by people, tools and machines; it requires new, innovative and scalable technology to collect, host and analytically process the vast amount of data gathered in order to derive real-time business insights that relate to consumers, risk, profit, performance, productivity management and enhanced shareholder value. Big data includes information garnered from social media, data from internet-enabled devices (including smartphones and tablets), machine data, video and voice recordings, and the continued preservation and logging of structured and unstructured data.
It is typically characterized by the four “V’s”
• Volume: the amount of data being created is vast compared to traditional data sources
• Variety: data comes from different sources and is being created by machines as well as people
• Velocity: data is being generated extremely fast — a process that never stops, even while we sleep
• Veracity: big data is sourced from many different places, as a result you need to test the veracity/quality of the data.
Evolving technology has brought data analysis out of IT backrooms, and extended the potential of using data-driven results into every facet of an organization. However, while advances in software and hardware have enabled the age of big data, technology is not the only consideration. Companies need to take a holistic view that recognizes that success is built upon the integration of people, process, technology and data; this means being able to incorporate data into their business routines, their strategy and their daily operations. Organizations must understand what insights they need in order to make good strategic and operational decisions. The first part of the challenge is sorting through all of the available data to identify trends and correlations that will drive beneficial changes in business behaviour. The next step is enriching this organizational information with that from sources outside the enterprise; this will include familiar big data sources, such as those created and stored online. In a business environment that constantly and rapidly changes, future prediction becomes more important than the simple visualization of historical or current perspectives. For effective future prediction, data analysis using statistical and predictive modelling techniques may be applied to enhance and support the organization’s business strategy. The collection and aggregation of big data, and other information from outside the enterprise, enables the business to develop their own analytic capacity and capability, which for many years has only been available to a few larger organizations.