Saturday, February 1, 2014

My Interview for an article related to global mobility of Big Data Professionals

Even though Big Data has undeniably captured the imagination of most of the global tech companies, the industry has a unique problem. Lack of skilled manpower to take the momentum forward. The big data industry is mired by acute shortage of talent. The technology in itself is less than few years old, but the ecosystem is flooded with new tools and frameworks. With no proper books, training materials or university curriculum widely available, its mostly left with individual self learning and internal training programs of companies.

This lack of talent has forced companies to look for people wherever they are available, there by creating a truly global work force of big data professionals. This was theme of the study conducted  by the Hydrogen group.  

The Links to the article and its content are given below:



 Big data projects will become major catalysts in the creation of global tech hubs, attracting top professionals from around the world to the countries that specialise in this sector.

Dominic McNamara, Global Leader of Hydrogen’s Technology Practice, said, “Technology is being talked about much more by the wider business community, especially in the US and UK which are seeing major investment into big data companies. It is no coincidence that these two countries were also voted the top two destinations for tech professionals in Hydrogen’s Global Professionals on the Move report. The best global candidates are at the forefront of this rapidly evolving trend that is having a huge impact on business as a whole. The industry is now extremely powerful and professionals are travelling overseas to capitalise on their expertise in new markets.”
Vinoth Kannan, a big data engineer currently working in Germany who has been a truly global professional since he left India in 2007 is a great example of effective overseas capitalisation. He has worked in Italy and France and has now moved to Germany to advance his career in big data, “I have always been open minded about moving from country to country as it is a means of accessing the best jobs available in the global marketplace. Working in different countries is a great way of broadening one’s perspective through experiencing a wide variety of work cultures and meeting new people.”
This willingness to travel to wherever the best opportunities are is greatly beneficial to companies who can then be assured of reaching the best global talent for their roles. Tesco recently ran a project with IBM to cut its refrigeration energy costs. The team that undertook the research is international, with consultants from Germany, Argentina, America and Australia in the mix. John Walsh, Data Consumption Manager for Tesco UK and Ireland commented, “Now that an increasing number of technology professionals are prepared to travel all over the world for the right role, we can undertake projects of this scale in the absolute knowledge that we have the people with the very best skillsets drawn from a global talent pool.”

Friday, January 31, 2014

Real Time Big Data Use Cases for Telecommunication Sector


Today, one of my friend who works as an Analyst for a major telecommunication company asked me, how Big Data can help create value for their company. I thought this makes a good topic to make an entry in my blog.

Majoring in telecommunication systems, I have a fair idea about the problems and limitations of telecom companies. The mobile customer base across Europe and North American are already saturated. India and China once fastest growing markets for new customers are also in the stage of maturing. The fierce competition between various telecom players has forced the tariff rates to remain as low as possible. With no avenues to increase the tariff rates or customer base, compounded with the increasing operating cost has already taken a toll on the balance sheets of the telecom companies across the globe. The future doesn’t look bright for this sector.

Other than the financial owes the other common factor uniting these companies are the user and caller logs – tons and tons of it. Majority of these companies store these unstructured data and most do nothing with it.

So how can big data help process these huge unused logs to create value and new source of revenue for the telecommunication industry?

Big Data helps you achieve it in the following areas:
  • Targeted Advertising
  • Location Based Advertising
  • Infrastructure Planning
  • Customer Service Center Optimization
Know your Customer:
For adding value to the any industry, it’s important to know about the customers it is serving. Big Data helps us understand more about who the customer really is, what are his likes and dislikes, his behavior pattern and recommends/predicts products and services which suits his needs better on real time.



From the given data in hand we can see how big data can help Telecom Company to understand its customer. The caller logs generally contain information given in the table below. It’s usually who called whom, at what time, the duration of the call and from which location (usually the mobile tower number or GPS coordinates)

Caller Number
Caller Location
Receiver Number
Receiver Location
Time of Call
Duration of Call

The customer database has the information about the caller such as the age, sex, profession, income etc. With just these two datasets and applying the right data science techniques it’s possible to extract the detailed customers profile of each of the caller.

If you know who your customers are and what they want, the possibilities are limitless.

Targeted Advertising
Mobile based advertisements can be better targeted to the right person at the right time if we know the customer need and preference

Let’s consider this example:

Peter, a 20 year old guy calls Dominos pizza and orders a pizza for home delivery through his mobile phone on a Sunday afternoon.

The caller logs will have this data. Since the Dominos pizza telephone number is publicly available, we know whom he called and at what time the call was placed.

From this single log we can deduce the following information about the customer.

o   Male Customer aged 20
o   Likes pizza
o   Lives in Area X
o   Prefers pizza on a Sunday afternoon
o   Prefers Dominos Pizza

Consider a million such call logs of all consumers who ordered for pizzas through various outlets across the city which can be processed with the power of Big Data. The customers can effectively be found, categorized, grouped and profiled based on their age, locality, likes and preferred brands. This data can be used with advertisers to market their product in a targeted manner.

Now, consider if Dominos Pizza in Area X wants to run an SMS advertisement campaign through the telecom provider to advertise a new pizza it is introducing in that month or a promotional discount.
The SMS/Mobile App based advertisement campaign is bound to be successful

·         If it is viewed by the 20 year olds than the 60 year olds.
·         If it is viewed by the consumers who are in Area X
·         If it is viewed by the consumers who likes pizza
·         If it is viewed by the consumers who likes Dominions pizza
·         If it is viewed by the consumers who like pizza in their preferred day and time.

Similarly its possible to categorize users based on their financial needs, shopping needs, travel needs and recommend products and services just in time. This has already been done by social media companies like Facebook and Google to track the users online digital usage and do targeted advertisement. Effective big data integration into the telecom industry can replicate the success of social media in telecom too.

Real-time Location Based Advertising
Telecom providers are better connected to their customers than any one else at the moment. Even social media cant match the access to their customers like the telecommunication companies. With the fixed mobile towers the telecom provider can effectively know their customers by location and target advertisements based on their current location.


Lets consider the same Dominos pizza example:

A user who likes pizza is travelling in his car near location X. The Dominios pizza in location X will have better chance of the user visiting the restaurant if the advertisement reaches him when he is near by the location.

But since the user is on the move it becomes important the message reaches him on time. Unlike the previous case when profiling is done in bulk and messages are sent in bulk but in a targeted manner, in real time location based advertising timing is critical. The information has to be fed to the system and decision has to taken and acted on at real time.  

Real time big data solution can help us realize such a system in a fast, scalable, fault tolerant and really smart way. The ecosystems of Hadoop and Storm give us the perfect tools to build such a System.

Infrastructure Planning
Big Data can help the companies plan their infrastructure better. From the user logs, it’s easy to identify the general usage of the network i.e. when the network load is at maximum and minimum. The network capacity can also be determined for a particular location and during a given time of a day.

This can help the operations to plan their infrastructure better. With proper machine learning algorithms we can predict network overloads and possible cases of crashes in advance. This in turn translates a stable infrastructure with less downtime and better customer satisfaction.

Customer Service Center Optimization

When you know your customers, its easy to determine the problem they might face. With the predictive data modeling it’s possible to determine cause of problem of the customer before the customer service executive can pick up the call.

This can help company to resolve the issue to the earliest by connecting to the right person without much delay. With some good analytics on the customers issue, its possible to foresee the problem in advance and rectify them even before they occur.

Conclusion
To conclude, real time big data systems can optimize and improve the existing process in the telecom industry and add more value the sector. Making use of the existing data can give a huge boost to the performance, revenue and open more avenues for new business.


More to continue..

Monday, January 27, 2014

The hype called "Big Data"



“Big Data” – The hype of the next big thing in IT. We are hearing a lot about this buzzword for quite sometime now. But is it really worth the hype? What can big data give to businesses and software architecture which the traditional 2-tier/3-tier architectures or DWH can’t solve? As a developer and as an architect is it worth my time and effort to learn this technology? If so where to start, which route to take?
These are few questions this blog will cover.

So, What is Big Data?

Just to give an idea of amount of data that is generated everyday today: One study says “A full 90 percent of all the data in the world has been generated over the last two years.” Another says “we generating as much data in 2 days which the whole mankind generated till the year 2002”. That’s insane isn’t it ?!!. Where is this sudden burst of data coming from? Before thinking about it, just turn around and have a look around you – Your smart phones, Your social media, Your CCTV cameras, Your fire sensors, Your temperature sensors, your Car. Your every click on a website generates some kind of data. All these devices around you leave some kind of digital footprint about the users’ action and surroundings. And this is only expected to go up with time. With the cost of storage dropping significantly, most businesses store this information. But a lot many of these enterprises don’t realize what kind of gold mine of information they are sitting on.

Just because of the very nature of this data, the traditional way to process this ever growing data heap becomes obsolete. Forget about processing, the basic storage and retrieval of these data becomes a humongous task.

This forms the basic principle for the need for Big Data Technology. Most popularly known as the “4Vs Concept of Big Data” originally from Oracle, it clearly defines the goals to tame this every growing data and bring value to businesses in a fast, smart and cheap way.



The 4V principle covers the following:

Volume
Handling the huge amount of data generated by enterprises, storing them in a scalable distributed way
Velocity
Processing and analyzing this huge data in an acceptable good speed
Variety
Combing the data sources from the multiple inputs and processing them
Value
Create maximum value from these data to have great insight into possible areas which was not possible before


So, from an IT architecture perspective, the basic question that lingers around is how I can adapt my existing system to satisfy the 4Vs principle of Big Data. Throwing away a long running stable system for a new untested technology is never an easy decision. But when you know your existing system is going to get clogged down soon with the huge amount of data and not doing anything about it is a recipe for disaster. For businesses who claim they don’t generate as much as data which they can’t handle, big data provides much more insight into things which you haven’t seen and can add great value to your organization.

So, it’s important every businesses and software architects should have a strategy for the future. Any new application development should be thoroughly examined for its scalability. Existing applications should be eventually integrated in to the ecosystem of Big Data.

For having this strategy a thorough understanding of this technology becomes important. My goal for this is blog is to evangelize for the need for businesses, IT architects and developers to move towards Big Data technology, understand the underlying principles of this technology and develop interesting solutions to problems which were never been thought before.

More to follow…..