Enabling data discovery: Big data’s ability to solve bigger problems

Industry leaders are debating the co-existence of big data & traditional business intelligence. One group strongly believes traditional business intelligence will be washed away in the big data tsunami, while another group discounts big data as a big hype and vouches for traditional business intelligence as an organization’s bastion.

Big data will continue to be the buzz word at least for a few more decades, as the industry going all out to create interesting solutions. With third party solution providers fighting for their share of pie, Apache incubator will continue grooming tons of new frameworks & tools. New players will emerge and existing niche players will be consolidated – either by acquisitions or by getting lost in a fierce battle. Big data is here to stay for a long time, growing in stature and adding value to customers.

Defined in simple terms, business intelligence is to use reports and dashboards to answer a set of formulated questions. Today, big data solutions are augmenting the data warehouse eco-system; solving problems that are not optimally solvable otherwise due to the sheer volume, velocity and variety of data and other factors.

Big data solutions can do much more to help businesses. True potential arises with the ability to ask bigger questions by comprehending data beyond the realms of the traditional data warehouse. Data discovery starts with asking questions that are not asked today to know the unknown … Voilà!

Big data can catalyze organizations to mature the curve from business intelligence to data discovery. An emerging area in big data is graph database and allied graph analytics. Graphs are not new, they were documented as early as 1736 on the famous Seven Bridges of Königsberg problem presented by Leonhard Euler and have evolved since then. Today graph databases are imperative in the NoSQL arena, when the relationship among the data is as important as the data itself. The data is stored in a schema-less manner as nodes and edges, with ability to crunch humongous data in memory at astonishing speeds. Graph analytics provide the ability to solve the most complex of problems in a simple yet efficient manner by applying graph theories.

The methodology of overlaying a subset graph pattern over the dataset towards discovery of matching data patterns makes data discovery engaging as well as an easy activity.

Majority of the product recommendation engines for online retail today have products hardwired at the platform configurations. These recommendations are usually pre-calculated weeks ahead based on customer’s historic purchases and may be irrelevant today. Graph analytics can provide dynamic recommendations based on real-time or near real-time buying patterns using pattern matching algorithms which not only uses past purchases but also product catalog search patterns and social media activities.

Data discovery’s impact on enterprises bottom line is immense and big data solutions can aid getting there faster. Life science and financial industries are early adopters where solutions are helping drug repurposing, detecting financial fraud patterns and money laundering.

Virendhar Sivaraman

Architect - Big Data Solution and Technical PM, Virtusa. Virendhar is a seasoned Data Warehouse and Business Intelligence professional with over a decade of experience. Over the years, he has designed and delivered many traditional data warehousing projects. At Virtusa, he has been playing a key role in creating and implementing technology solution including big data. His core competencies include churn analytics, customer profiling & loyalty management solutions and sentiment analysis. He holds a Apache Hadoop Developer Certificate from Cloudera. Virendhar received his Post Graduate Diploma in Management (Marketing) from Loyola Institute of Business Administration, Chennai and his Bachelors in Engineering (Computer Science) from Crescent Engineering College, University of Madras.

More Posts