In a previous blog post, I discussed the benefits of the convergence between Business Process Management (BPM) and Master Data Management (MDM). The basic concept was to leverage BPM as the data governance mechanism to ensure data quality, consistency and unification within the enterprise. The point really was that it is hard to optimize your core business processes or improve the end customer experience without having a unified view of your key data – whether that’s customers, products, or any other key operational data.
Continuing on that theme, there seems to be no reason why BPM can’t also be leveraged to drive more insights within an organization’s big data. If you listen to most companies’ today, they are overwhelmed with data so there seems to be a big opportunity to derive more value from this data. And those that have tackled the big data processing problem are still struggling with ways to effectively action these insights. In some ways, MDM (i.e. a unified view of your customer) is almost a pre-requisite to leveraging big data insights. It’s really hard to use these additional insights to their potential if you don’t “know” your customer. Customers expect you to know the “whole me”.
The basic challenge that big data presents for many organizations is its unstructured nature, high volume, variety of sources and speed at which it is produced. While traditional data sets have some of these characteristics, the complexity and scale of big data makes it much harder to process, organize and store in a cohesive manner that makes it easy to analyze. This has all been aptly captured as the 3 V’s of big data (Volume, Velocity and Variety). Because of this, many organizations are struggling to keep up with the data that is now being produced by their internal systems, customer transactions or social networks. The bottom line is that with the amount of digital information being produced, companies need processes that can help them make better sense of data.
Compounding the problem, is the fact that much of the insights needed for understanding customers, markets, products, promotions and sales lies in the correlation and analysis of that data. Historically a lot of this analysis would happen in the back office and typically be produced as monthly reports for executives to analyze and react to. But the world has changed and the expectations of the end consumer are now driving more real-time analysis of transactional and social data feeds.
So how can BPM help? BPM is traditionally used to drive operational efficiencies from a process management perspective. BPM excels at explicitly defining and automating any business process. It is able to handle both straight-through as well as process exceptions. BPM is responsible for routing and tracking work as it flows through the process and manages all the necessary steps, tasks and activities required to accomplish the process outcome (i.e. the work). Once this has been operationalized, the real benefits of BPM can appear as processes can now be monitored, streamlined, further automated or enhanced to increase more straight through processing. This drives both increased quality as well as lower processing costs.
So how does this all apply to processing big data? If you consider that a lot of the processing big data requires can be rather tedious and repetitive in nature, BPM tools can certainly help automate and guide these data tasks.
The first step is to understand where all this “big data” is coming from. First there is the onslaught of social networking data with millions of unstructured data records generated every second. There is mobile data, generated from location-based services, mobile commerce and other mobile application data. There are also huge amounts of digital images, videos and other multimedia information, that are now produced as routine data sets in industries such as medicine, education, entertainment, transport and other public services. There is an enormous amount of sensor data being produced in our infrastructure, from railways, electric grids, water supply, etc. And also machine to machine interactions on the Internet are also on the rise. This is all in addition to typical business generated data such as documents, emails, transactions, etc. that are produced as part of normal business operations.
There is an opportunity is to turn these volumes of data from passive information to active information using the capabilities of BPM. Integrating BPM into the point of origin of the data can help really help drive/guide the processing approach on the backend. BPM can also play a role in the downstream analytics.
So here’s how BPM can specifically help:
- Automate the Processing. Big data frequently needs Meta data to be added to the data to effectively enable downstream analytics. The tagging of these data streams can be tedious so automating then with BPM by defining a set of rules and/or classifications for how tagging happens can improve the quality and speed at which data may is processed.
- Simplify the Integration. Data integration is something BPM deals with in any implementation. Big data has similar issues in that there are typically many data sources that need to be aggregated and processed. Feeds can come from transactional systems, social networks or B2B channels. Using BPM as a way to pre-process, pre-aggregate or direct these integrations can accelerate the time to analysis.
- Bring Context to Big Data. Rather than processing a stream of data and not understanding the context for which the data is being analyzed, BPM can tailor the analytics of the data by directing exactly what is needed. Some examples of understanding context would be knowing if it is a cross-sell opportunity, a fraud investigation, or a customer service request. Having context can help drive exactly what information can enhance a given interaction or process.
- Bring Big Data into the Mainstream. BPM can help big data enter the mainstream. Many of the big data applications that are being created today tend to be on the peripheral of mainstream applications or the core business processes. Big data is currently thought of as an adjunct these core platforms. In many cases, BPM has already evolved to an enterprise standard and as such is well positioned to more tightly integrate big data in mainstream processing.
- Drive Organizational Change. As more and more “big” data becomes available, organizations will need to consider if they are functionally structured to best take advantage of the insights in the their data. To truly become a customer centric organization, companies need to create a “Customer Data Hub” that not only provides a unified view of the customer but also integrates real-time big data analytics to drive proactive customer interactions.
The more organizations continue to tightly link their Data Management and Process Management initiatives the more effective they will become. Whether it is MDM, big data or other information management initiates, integrating these are the key to acceleration. I think Clay Richardson from Forrester Research coined it best, “Big Data” needs “Big Process”. Well said.
So, big data is not something to be frightened of, despite all the hype and fear that is being created. It’s something that should be harnessed for competitive advantage. Data has always been fundamental to running efficient, high growth businesses. Nothing has changed except that we have a lot more of it to process. But let that not be the inhibitors as distributed technologies such as the well-publicized Hadoop coupled with elastic computing power offered by Cloud infrastructure providers now offer innovative ways to tackle the big data processing problems. And once you have tackled the processing of big data, integrating it with BPM to leverage the insights to drive behaviors and actions…that’s unleashing the real power of big data!