The hype called "Big Data"

“Big Data” – The hype of the next big thing in IT. We are hearing a lot about this buzzword for quite sometime now. But is it really worth the hype? What can big data give to businesses and software architecture which the traditional 2-tier/3-tier architectures or DWH can’t solve? As a developer and as an architect is it worth my time and effort to learn this technology? If so where to start, which route to take?

These are few questions this blog will cover.

So, What is Big Data?

Just to give an idea of amount of data that is generated everyday today: One study says “A full 90 percent of all the data in the world has been generated over the last two years.” Another says “we generating as much data in 2 days which the whole mankind generated till the year 2002”. That’s insane isn’t it ?!!. Where is this sudden burst of data coming from? Before thinking about it, just turn around and have a look around you – Your smart phones, Your social media, Your CCTV cameras, Your fire sensors, Your temperature sensors, your Car. Your every click on a website generates some kind of data. All these devices around you leave some kind of digital footprint about the users’ action and surroundings. And this is only expected to go up with time. With the cost of storage dropping significantly, most businesses store this information. But a lot many of these enterprises don’t realize what kind of gold mine of information they are sitting on.

Just because of the very nature of this data, the traditional way to process this ever growing data heap becomes obsolete. Forget about processing, the basic storage and retrieval of these data becomes a humongous task.

This forms the basic principle for the need for Big Data Technology. Most popularly known as the “4Vs Concept of Big Data” originally from Oracle, it clearly defines the goals to tame this every growing data and bring value to businesses in a fast, smart and cheap way.

The 4V principle covers the following:

Volume	Handling the huge amount of data generated by enterprises, storing them in a scalable distributed way
Velocity	Processing and analyzing this huge data in an acceptable good speed
Variety	Combing the data sources from the multiple inputs and processing them
Value	Create maximum value from these data to have great insight into possible areas which was not possible before

So, from an IT architecture perspective, the basic question that lingers around is how I can adapt my existing system to satisfy the 4Vs principle of Big Data. Throwing away a long running stable system for a new untested technology is never an easy decision. But when you know your existing system is going to get clogged down soon with the huge amount of data and not doing anything about it is a recipe for disaster. For businesses who claim they don’t generate as much as data which they can’t handle, big data provides much more insight into things which you haven’t seen and can add great value to your organization.

So, it’s important every businesses and software architects should have a strategy for the future. Any new application development should be thoroughly examined for its scalability. Existing applications should be eventually integrated in to the ecosystem of Big Data.

For having this strategy a thorough understanding of this technology becomes important. My goal for this is blog is to evangelize for the need for businesses, IT architects and developers to move towards Big Data technology, understand the underlying principles of this technology and develop interesting solutions to problems which were never been thought before.

More to follow…..