The traditional data warehouse is the single repository of non volatile, historical enterprise data snapshots used across a range of analytical and reporting purposes. Data from different Transactional data source systems get extracted, cleansed and transformed to build the enterprise data warehouse. The EDW acts as a centralised data hub from which different analytical reports are built to answer specific analytical need of an organisation. The analytical capability of an EDW system is built based on facts as gleaned from structured data sets and often limited in generating predictive and indicative analytics. The critical reason for this failure is the lack of near-real-time and expanded information beyond structured data. 10-20 years before, it was considered an analysis of day-old data as a good unit of measurement for analysis, which is no more true for today’s fast paced Internet world. Internet traffic trends can very hour to hour or even minute to minute with so much wide spread internet users spread across the globe. To get more accurate results, the more topical data has to be included for analysis. Also study says that only 10-20% of data in an organisation are structured, which is very minimal to be considered for major business decision making process. These are the two main important reasons behind the need of including Big Data into Traditional Data warehouse system. Big data analytic enables better analytical insights by integrating more voluminous data of varying complexity and timeliness into one structured output.
Big Data is the new Industry buzz word, just like Cloud Computing a few years before. It is massive-volume, increased-velocity and greater-variety information assets which if effectively utilised can bring opportunity of doing more real-time analytics and help in effective decision making.
Mostly Big Data boils down into 3 categories
1)Transactional Data – includes Data from invoices, payment Orders etc.
2)Machine Generated Data – include data generated from Industrial equipment such as real-time data generated from sensors on your smart phone, Heart-Rate Monitor, smart grids, CCTV cameras etc.
3)Social Data – These data coming from social networking site such as face book likes, tweets or you tube video views etc.
These data are either semi-structured or unstructured and in many cases these in its own meaningless. However if combined effectively with traditional structured data could generate business Insights, decisions and actions which standalone data warehouse often fail to produce.
It is a matter of fact that the concept of big data is not altogether new. Enterprise organizations have been working with extremely large data sets for decades, combing through the mountains of information contained within their repositories to gain insight into customers, their buying habits and demographics.However the topic “Big Data Analytics” is relatively new idea, which industries are interested to tap into it in terms of understand its significance, benefits and challenges in implementing the same.
Why is Big Analytic is indispensable?
Big data hype is certainly around for quite some time, but the clarity is still not well understood by many executives on how to leverage on this 3 V’s (Volume, Velocity and Variety) of this big data and augment the value of data for their organisations.
With the advent of new technologies every minute details data can be captured continuously which brings large mass of data exploding at an alarming speed. Also Industries are slowly moving from existing reactive approach to more forward looking and proactive analytic to survive in the fast paced world with more competition.
In a conventional data warehousing approach, BI solutions provide standard business reports, OLAP, analytic and even alerts and notification by analyzing the static past records, which has its purpose in a limited situation. However in big data analytic with exposure to most-recent and big variety of data sets, it can help business to identify trends, spot weakness or determine conditions to make more informed decisions by doing various statistical analysis, forecasting, text mining etc. etc which was not in scope of the Traditional data warehousing phenomenon. So Big Data Analytics is more of a culture change – An imperative need to do business in a new and efficient way with sound knowledge and insights in advance to gain more competitive advantages in the industry. It can truly change operations, prevent frauds, gain competitive edge, retain more customers, anticipate disease outbreaks or run unlimited budget simulations – the potential opportunities in Big Data Analytic are endless.
Benefits of Big Data Analytic
The expansion of big data into business Intelligence platform is nothing but Big Data Analytic, which will enable BI professionals to leverage expanded analytic and create 360° perspectives. It can offer valuable outlook on customer behaviors, sentiments, interactions and utility usage trends from a business-to-consumer model in CRM, efficient energy management scheme and better safeguard against major grid failure in utility industry and better vision on a patient’s medical background field to bring more accurate diagnostic results in Medical Health care system. I will limit my scope on 3 important areas, where augmenting big data analytic to the existing BI solution will be a wise investment.
CRM Analytic
The Traditional data warehouse based out of transactional call center data helps business to determine the customer behavior and choices to the extent it can retrieve from the transactional data entered by call center agents for each issue raised by the customer himself. However it is beyond the scope to visualise on certain important parameters such as customer sentiments, wish lists, actual customer response data from different online surveys or from social networking sites about a product review etc. These data could be collected from different blog sites, forums and social media platforms, which are more recent and large in size and can give much more fruitful insight about the customer than the conventional business intelligence capability. Big-data analytics in CRM often focus on social media. And usually that means listening to customers to see what they're saying good or bad about your company or brand. Businesses, for instance, might scrape data from social sites, or have human analysts monitor these services to detect customer sentiment. And hence, it gives you a prospect to retain the valuable customers with customized opportunities that either encourages them to not leave, to buy more, or to feel more engaged with your brand.
Health Care System
Another popular area where big data analytics can be used in big scale is health care.
The large volumes of data generated by the healthcare industry are growing at a rapid speed. This data takes many forms, from electronic medical records (EMR), lab and imaging test results, and physician notes to medical correspondence and insurance claims.
Most healthcare organizations use structured data from ERP system and build business intelligence solution with much focus on financial side. But much of the information generated by other departments lies fallow due to the inability of companies to integrate and consume it in a meaningful way. It has been estimated that as much as 80 percent of healthcare data is unstructured, however clinically relevant. Now new demands are being placed on the healthcare system, resulting in a fundamental shift in how health-related organizations are managed. Stakeholder pressures are forcing healthcare companies to adopt new business models, patient care methods, and revenue drivers. Healthcare providers are moving from a payment-for-service model to a results-driven methodology. Big data analytics is helping healthcare providers to corral all of the relevant data available to gain an enterprise-wide, rather than departmental, view of the business. Among the challenges big data analytics can help address are improving patient care, cutting costs, and making the best use of staff.
Smart Grid Management
Many utilities companies are moving to smart meters and smart grids as part of a long-range plan to ensure reliable energy supplies, incorporation of distributed generation resources, innovative storage, a reduced need for new power plants and customer control over their energy use. When fully operational, a smart grid generates huge volumes of high-velocity data. The deployment of smart meters alone, often the first step in the journey to a smart grid, represents a 3,000x increase in data generated.
Hidden inside all of this grid and meter data are valuable insights utilities can use to better understand customer behaviour, detect outages, fraud or theft and more accurately forecast energy demand. Extracting this insight from the data generated by smart meters and smart grids requires solutions capable of managing high volumes of data and detecting patterns – big data and analytics solutions, to be precise.
Big data and analytics solutions are greatly enhanced by an integrated view of the data and alignment across disparate operational groups and lines of business. Utilities that integrate and analyze this data can gain insight into their operations and assets, enabling them to anticipate events instead of reacting to them as mentioned below.
1) Optimize software to execute unit commitment algorithms
2) Run predictive analysis
3) Manage data and accelerate business intelligence queries
4) Present or visualize analysis results
5) Detect anomalies and correlate network events and failures in real time
Challenges and Risks ahead
There are certainly hurdles on the way to implement Big Data Analytics in organisations as mentioned below.
Executive Sponsors are lacking the knowledge of Big Data Analytics Applicability
There is a famous quote which says ‘Implementing Big Data should be a business decision rather than IT decision’. Analytic projects are most successful, when approached from a business perspectives rather than IT perspectives. Executives needs to have adequate knowledge on big data and should be able to visualise the requirements, which big data analytics can fulfil to achieve certain business goals. Business teams must own the big data project or program. This requires the CMO, CSO, CRO, CFO, or CCO to become the executive sponsor of the initiative.
To gain such sponsorship, teams need to build a robust business case, providing the depth of data analysis required, why the enterprises lacks such depth in the current environment, as well as the key metrics and analytics that can be implemented or extended with a big data project. The business case must also provide a clear ROI period and repeated ROI cycles in the system life cycle. A clear and concise business case will be the first step towards getting project approval.
Big Data Projects are more Complex to plan and execute
The complexity stems from your need to perform data discovery before you can document a single user requirement. If you lack clear business requirements, you cannot plan the remaining project logistics, including team, skills, execution steps, rollout, and training.
Lack of availability of business subject matter experts (SMEs) or Data Scientist
The success of big data project is very much dependent on the expertise of data scientist. Various survey reports says that, there is acute shortage of professionals who have both data and institutional knowledge as well as a command of statistical knowledge and logic. Since Big Data analytic approach is relatively new in the industry, so the skill sets required for these projects are scarce. This is one of the most important reasons, why Organizations often shy away from big data initiatives.
First, create a powerful team that can set up a platform to explore big data. This team will work with business and data analysts to create the road map for further execution. Critical success factors include:
1)Availability of IT resources to build and configure the selected big data platform
2)Availability of business SMEs with data and domain expertise
3)Availability of resources with BI expertise and deep statistical knowledge
4)Implement a technology centre of excellence to provide big data infrastructure support
5)Extend other BI best practices, including data governance, MDM, metadata management, analytics definition, and visualization, to include big data
6)Ensure adequate training for users to understand the new data and its integration into the analytical and reporting platforms
When it comes to people, having a combination of individuals mentioned above will create a team that can leverage each other's skills and create a unified vision for exploring big data.
Conclusion
Big data is heading towards the same path that business intelligence and analytic did a decade ago -- to become the cream of enterprise data management. It will be the most important project to be executed by any enterprise in the years ahead. This growth and adoption of big data will be aided by continued technology growth in Hadoop, NoSQL, MapReduce, and Mahout Algorithms, among others.The journey is long and complex, but with guided navigation from concept to adoption, big data will continue the dominance into the foreseeable future. Definitely it is going to be the game changer in the industry the years ahead.
