The last several blog posts have looked at the explosive growth of data and the challenges faced by data centres in managing it. Many of our Tweets of late have been around the topic of “Big Data” and we felt it would be of interest to our readers to talk a bit about this topic.
First off, there are a variety of differing opinions on an agreed definition of what exactly is “big data”? The best description we found was from a Wiki post which described big data as consisting of…”data sets that are too large and complex to manipulate or interrogate with standard methods or tools.” This complex environment explains why so much IT investment is now going towards managing and maintaining this big data.
When defining big data, it’s important to also understand that there is mix of both unstructured and multi-structured data within it. Multi-structured data refers to a variety of data formats and types such as those typically between people and machines such as web applications. On the other hand, unstructured data is not easily interpreted by traditional databases or data models. This type of data is not organized or structured and typically comes from text-heavy applications like Twitter tweets and other social media posts.
A logical question one might ask then is: “So…how do we make sense of this Big Data?” This will be an ongoing challenge for many years to come but by properly interpreting big data we can learn powerful new insights which might not otherwise be immediately visible. These new insights would likely be impossible to find using traditional data analysis methods. The challenge will be to focus on finding hidden trends and patterns which would be invisible to the naked eye. Not an easy task but one which is being aggressively pursued as we speak.
An example of big data in action was nicely documented in an article by the Wall Street journal back in late 2012. They showed how Netflix used big data to build out their streaming video service. The Netflix operations team were able to analyze their traffic in indepth detail to spot problem areas and help them proactively and accurately add network bandwidth ahead of their growth curve to prepare for future demand. Big data helped them gain insight into the type of content customers preferred which allowed them to make more accurate suggestions as to what programs these subscribers might like.
When talking about big data, industry analysts typically use the “V” words…“volume” (the amount of data), “variety” (the kind of data) and “velocity” (the speed of information generated). Some experts even expand to add new terms such as big data’s “veracity” and “value”. Regardless of how it is defined, it is critical for every enterprise to begin to fully understand what big data will do for them, what it means to them, how to manage it and how to maximize its potential for their business strategy.
The infographic below highlights the profound impact that big data will have across all industries from healthcare to retail to the military.