Overcoming Legacy Processes to Achieve Big Data Success

For large companies with legacy infrastructure, changing to a Big Data focus isn’t easy — but AMEX proves it’s possible.

For large companies saddled with complex legacy environments that were developed over many decades, finding the opportunity for disruption can
be a daunting challenge.

Unlike new economy firms, which have had the benefit of being able to
build their businesses and infrastructure from scratch in a “green-field” environment, most large corporations are saddled with disparate and
fragmented operational and analytical environments and processes,
characterized by business and data silos that limit the ability to operate
with agility, flexibility, and insight across customers and lines of business.

While institutional reputation, customer reach, and operational scale
provide many advantages to large corporations, these larger firms cansometimes be challenged when it comes to innovation and
responsiveness. Yet it’s still these large corporations that continue to be
the bulwark of the economy, accounting for the bulk of business and
consumer transactional activity. Now, Big Data approaches that were
developed by the new economy firms, which enabled the flexibility and
rapid growth of these firms, are being adopted by mainstream
corporations to overcome legacy challenges and introduce greater
corporate agility and speed.

A 2014 survey conducted by NewVantage Partners of 125 senior
corporate executives, representing 59 Fortune 1000 companies, showed
that more than two-thirds of executives reported that their organization
had a Big Data initiative in production. These same executives reported
that investments in Big Data are projected to grow dramatically in the
coming years — with 75% of executives reporting investments greater
than $10 million, and 28% reporting investments exceeding $50 million
by 2017.

For these corporate giants, operational process optimization can have
a huge business impact.

One such firm that’s benefiting from new Big Data approaches is
the financial services giant American Express (AMEX), which is
harnessing the power of its data to innovate and to streamline complex
operational processes. AMEX has opened its own new technology hub
in Palo Alto, led by former Google and PayPal executive Nik Sathe.
The new tech center “will focus on innovations in Big Data, cloud
computing, and mobile infrastructure,” according to AMEX CIO Marc
Gordon.

AMEX has taken steps to optimize and streamline its operational and
data processes end-to-end by migrating many traditional processes
from legacy mainframe environments to Big Data processing
environments, resulting in dramatic improvements in speed and
performance, significant reductions in cost, and notable increases in
responsiveness to customer needs.

Ash Gupta, chief risk officer and president for Risk and Information
Management for AMEX, comments, “Fact-based analytics have always
been in our company’s DNA, yet I have never seen such an analytical
leap as the one we have made with Big Data.” He cites several areas
where AMEX is focusing its Big Data efforts, notably in addressing
service excellence, generating billings and receivables growth, and risk management. A few examples:

- Using Big Data platforms, AMEX has been able to dramatically reduce
the time it takes to process the thousands of variables and billions of calculations needed to match customer card offers to card member
interests. It previously took years of processing time just to sift through
these massive volumes of data, but now it can be done in hours.

- AMEX has made similar strides in connecting merchants to card
members on personalized offers, taking a 3-day process and reducing
it to 20 minutes on their Big Data platform.

- American Express uses Big Data analytics to detect and prevent fraud
in milliseconds and with greater precision, using machine-learning
models that leverage billions of historical transactions across millions
of card members, resulting in instant fraud alerts for customer protection.

Leading companies can learn from the example of AMEX and other
mainstream corporations that are adopting Big Data solutions and
approaches with success. Here are a few suggestions:

- Assess opportunities to migrate operational business processes from
mainframe environments to faster and cheaper Big Data approaches.

- Create a Big Data Center of Excellence or Analytical Sandbox as a
testing ground to identify suitable applications for Big Data.

- Adopt a “data first” approach that focuses on rapid identification of
data insights, rather than investing in large data warehouse initiatives.

- Develop a “culture of data” staffed by a next-generation of analytics
professional, whom are conversant in new Big Data approaches and
analytics techniques.

Mainstream companies will drive the future of Big Data investment. But
they must demonstrate a willingness and flexibility to tackle their
complex legacy environments and data infrastructure. For those firms
that make the commitment, as evidenced by the example of American
Express, the payback is likely to be high. Their ability to compete for
customer success in the coming years may depend on it.

SOURCE: MIT Sloan Management Review

How Time-to-Insight Is Driving Big Data Business Investment

For many corporations, obtaining insights rapidly is the true value of Big Data.

They say that time is money, but Fortune 1000 executives polled in the
fourth annual Big Data Executive Survey conducted by NewVantage
Partners have boldly confirmed that reducing time-to-insight rather than
saving money is the primary driver for their Big Data business investment.

Conducted in November and December 2015, and published on January
11, 2016, the survey confirms that Fortune 1000 firms believe that Big
Data will deliver competitive advantage by enabling their firms to act
faster when it comes to analyzing data, gaining insights, making critical
decisions, and bringing new capabilities to market. The survey reflects
the evolving perspectives of chief data officers, business presidents,
chief information officers, and the heads of Big Data initiatives for
nearly 50 prominent Fortune 1000 firms.

Survey participants included Fortune 1000 top 50 mainstays such
as CVS Health, JPMorgan Chase, Bank of America, and Johnson
& Johnson. Large financial services firms were heavily represented,
and as an industry group, have long been at the forefront of investments
in data management solutions.

As measured by investment and business adoption, it has taken just four
short years for Big Data to assert itself as an essential component of the
corporate mainstream. Among the Fortune 1000 firms surveyed by New
Vantage, 62.5% reported having Big Data initiatives in production or operationalized across the enterprise — nearly double the 31.4% of firms
at this stage in 2013. While only 5.4% of firms reported Big Data
investments in excess of $50 million in 2014, the number of firms that
project investments in Big Data of greater than $50 million leaps to
26.8% by 2017, a steep and rapid increase. For the first time, a majority
of firms (54%) reports having appointed a Chief Data Officer, up from
just 12% in 2012, providing further corroboration that data has become
a corporate priority.

What is driving the sharp increase in Big Data investment? According
to the NewVantage survey, a clear pattern has emerged. Organizations
feel a need to learn quickly and act faster. While only 5.6% of firms
identified cost savings and operational reductions as the primary driver
of Big Data investment, 83.5% of survey respondents named factors
relating to speed, insight, and business agility as the primary reasons for
Big Data investment.
Of this total, 46.5% firms pointed to factors aimed at increasing speed and
reducing the time-to-insight. This is illustrated in the chart showing a
breakdown of Big Data investment factors relating to time-to-insight.

Big Data Time-to-Insight

So, how will organizations respond and leverage Big Data investments to
accelerate the time in which it takes to capture and analyze data, identify correlations, derive insights, and validate their insights in the market?

Accelerate Time-to-Answer through
Test-and-Learn Processes

Business analysts have long been bound by the time it takes to capture,
organize, and make data available to non-technical users. Big Data
processes have consolidated the time it takes to engage in analytics by
reducing up-front data engineering and putting data into the hands of
business users faster. By starting with smaller sets of data, business
analysts can engage in iterative processes such as test-and-learn to
identify patterns and correlations that allow them to focus on the most
useful data quickly. This ability to accelerate the process of insight is
alternately referred to as time-to-answer, time-to-analytics, or
time-to-decision. The net result is the realization of greater insight faster.

Accelerate Speed-to-Market with
Data Discovery Environments

Organizations are employing new approaches to traditional data
management. These approaches include the deployment of analytical
sandboxes, Big Data labs, data hubs, and data lakes. All of these
approaches are designed to introduce greater flexibility and agility into
the process of taking data and transforming it into business insights.
The big breakthrough of Big Data comes from enabling firms to deploy
rapid analysis environments that facilitate data discovery. These more
nimble environments produce faster insights, which enable organizations
to move rapidly to action and accelerate the speed with which they can
bring new product and service capabilities to market. As a driver of
Big Data investment, “speed-to-market” experienced the greatest uptick
from previous years. Firms are looking for measurable results, ratified
in the marketplace.

With the emergence of a digital economy over the course of the past
two decades, leading firms have learned quickly that they must act
faster to respond to customer needs and competitive dynamics. Firms
can no longer wait days or months to analyze indicators of customer
interest and sentiment, or detect and respond to security threats or credit
breaches. Market leaders will not wait while their competitors uncover
critical insights that drive new product and service capabilities. Fortune
1000 firms have come to the conclusion that the ability to act faster
correlates with market survival and success. The need for faster
time-to-insight will be the driving force behind Big Data investment
for the years ahead.

SOURCE: MITSloan |Management Review

Big Data and Flexible Ethics

Few of us would doubt the value in the proper use of big data collection. Healthcare is perhaps most important, but better law enforcement and product safety improvement come in close seconds. Then there’s the intelligence value of collected data to find and neutralize growing threats to our nation.

So how can we take advantage of the value of big data without suffering the distrust and damage caused by its unauthorized collection and use?

Data, Data, Who’s Got Your Data?

In today’s connected world, you might say anyone and everyone. We’ve all heard that nothing is sure but death and taxes .... You can add the collection and use of big data to that list.

Everywhere we look, corporations and government agencies are relying on big data stores to pursue their relationships with you. If you shop online (and who doesn’t, these days?) your inbox and smartphone are full of marketing messages from retailers that seem to know what you have been buying, from whom and how long ago. Even your health records, driving records and a host of other things about you are being digitized and kept for analysis. In a culture that amasses big data that way, the deluge we are seeing is unlikely to abate anytime soon.

It’s become virtually impossible to know for sure who has your data or what are they doing with it. This is especially true as the “data sharing” phenomenon has grown – organizations collecting data and then sharing it with other organizations in return for their data. Now, even organizations you may trust with your information often don’t retain it and usually don’t control how it is used once they share it.

And with the rise of a highly sophisticated global data theft industry, even big data stores held by organizations with no intent to abuse or share them are vulnerable to unauthorized use – often ending up back in the marketplace as the criminals “fence” them back into legal channels.

With so many ways your data is being collected and with so many groups collecting it, or stealing it from someone who has, everything we do with any connection to the Internet is probably on someone’s digital shelf being used for … who knows what?

Big data can seem — and is often touted as — an unmixed blessing, taking us toward a brave new world of information transparency. But, like the “man behind the curtain” in the Wizard of Oz, there is a side to the big data phenomenon that we are meant to “pay no attention to.”

So, our question: How can the collectors of big data guarantee that our information won’t be used in ways we don’t approve of?

The answer: They can’t.

Fighting an Old War

All this isn’t to say we aren’t doing anything, but what we’re doing is way behind the times.

Several years ago, as technology began its rapid rise and data collection grew with it, privacy concerns (at a much lower level of jeopardy) drove the industry to adopt several protective techniques in an attempt to assure people that their data would be safe and used only for purposes they were OK with:

1. Purpose limitation, the careful restriction of collected data to only those elements critical to its intended use
2. Data minimization, the collection of only that data absolutely needed to perform the intended functions
3. Anonymization (a.k.a. “de-identification”), separating personal information from the statistical data needed for analysis
4. Software and hardware barriers to prevent data repository compromise

These worked (sort of) for a while, often creating more impression than substance of safety. But, like the aphorism about generals always fighting the previous war, while the growth of technology has rendered these techniques less and less effective, industry continues to depend on them to deal with today’s new world of big data.

The use of big data by legitimate firms for legal uses can be controlled to some extent by the use of “opt-out and opt-in” relationships between firms and their clients: Unless you give me permission to use or share your data, I agree not to do so. We see this on a growing number of websites.

SOURCE: CMSWiRE

Why companies wrestle with the 360-degree customer view

One Friday afternoon, Rob Rich's flight was canceled as he was trying to return to Boston from a business trip in Washington, D.C. There was bad weather, and the line at the ticket counter to rebook snaked across the terminal.

Rich, managing director of TM Forum Insights Research, a global trade association that offers market research and consulting to enterprises, decided he might save time by calling customer service directly to rebook. As a top-level flyer on the airline, he thought a service rep could make a quick change to get him home.

But the offshore agent on the other end of the line first suggested a flight that left Monday -- three days later. Then, when Rich suggested looking at nearby airports for flights, the agent said she hadn't known those airports were close.

"She didn't have the information she needed to be helpful," Rich said, also one of the heads of the customer experience management (CEM) practice at TM Forum. "She was trying, but she didn't know."

Technology has made it possible for companies to collect massive amounts of information about customers -- from location, to social media interactions to buying preferences. Most companies now know they need to have what's known as a 360-degree view of their customers to provide a better, more seamless customer experience. This involves adopting customer experience management best practices to track, oversee and organize every interaction with each customer. This often entails being able to integrate data from multiple communication channels as well, such as customer communication on the phone, in email, on social media, in live chat and so on.

Now we can have a meaningful conversation with customers when they walk in the door.

James SlocumCIO, OneUnitedBank

But having the ability to collect data doesn't mean that data is easily accessible or that it's being used to the fullest extent to improve the customer experience, analysts said. In an ideal world, for example, Rich's agent would have immediately known which airports were nearby and that he was a top-level flyer.

Analysts say companies are making progress toward this 360-degree view of the customer. But most, including the giants, have a long way to go, they said. The technology is not the problem -- it's readily available-- but issues like cost, employee buy-in, a poor understanding of the customer experience, and siloed data get in the way.

"Right now there are very few companies that have that complete picture of their customers. It's hard to do," said Megan Heuer, vice president and group director of SiriusDecisions, a global business-to-business research and advisory firm. "All of this data is available, but there often aren't the skills in place for using the data to make day-to-day decisions. Even data scientists trying to make use of rich information have a hard time getting their internal customers to buy in."

Getting to that 360-degree view

Boston-based OneUnited Bank was well-versed in the problem of data silos across complex systems, according to James Slocum, the chief information officer. Bankers needed to shuffle between multiple systems to get information about customers' deposit accounts, loans, investment accounts and online banking. Branches couldn't collaborate and share information, he said, creating a disjointed customer experience.

The solution for OneUnited, a 23-branch bank, was a customer relationship management (CRM) platform from Salesforce. They also brought in a cloud-based customer service application and an employee collaboration app. Bankers now have access to a single screen that gives a 360-degree view of a customer and all a customer's interactions with the bank, said Slocum, and all branches have access to the same information.

"When you have all these silos of data, you end up with a disjointed experience in your call centers and in your branches," he said. "Now we can have a meaningful conversation with customers when they walk in the door."

Though it's a long-term goal, the bank has not yet integrated customers' social media profiles into its CRM platform. There are plenty of available products to do that, Slocum said, but it's an issue of cost and resources.

Kiva, a nonprofit that matches low-income entrepreneurs throughout the world with microloans, also brought in a CRM platform and employee collaboration tools to improve its customer experience.

The organization has three sets of customers -- individual donors, entrepreneurs and the banks that handle the microlending, said Gerard Niemira, director of product management. Kiva needed access to complete, though varying, information about each set of customers to tailor the company's interactions with those customers, he said. With lenders, for example, having access to their lending history is key.

"Someone who has done 1,000 loans -- we don't have to give them basic information. They've probably heard it 1,000 times," Niemira said. "Now customers get better answers because we have a more complete picture of how they've interacted with us in the past."

However, analysts say technology alone is not enough in achieving this 360-degree view of customers. Companies need to develop a strong understanding of who their customers are, said Corinne Sklar, chief marketing officer for Bluewolf, a global consulting firm. They also need to develop a culture of being "customer obsessed," she said.

"Technology is just an enabler," Sklar said. "Figuring out who your customers are has nothing to do with technology."

When a client comes to Sklar to get a 360-degree view of customers, her first question is, "Why?" Companies have to identify a goal first, she said. Are they trying to acquire new customers, retain existing customers, cross-sell or up-sell, or reduce costs? Then, there needs to be a detailed analysis of the current customer experience to figure out what can be improved, she said.

Data silos are often a problem, Sklar said. Data integration is usually a priority, as is identifying any data that is missing and deciding which pieces of data are most important, she said. Sklar also recommended companies become more digital and cloud-focused because of the speed and flexibility a cloud environment provides.

Rich, the TM Forum director, said behavioral and cultural hurdles far overshadow technological issues when it comes to creating a 360-degree view. Organizations need to convince different departments to work as a team. Companies need to focus on the value of a 360-degree view to the organization, employee buy-in, training and making sure there is support from top management.

Most executives realize they need to move in the direction of a 360-degree view, Rich said. In a few short years, customer satisfaction has eclipsed financial issues as the most important priority in TM Forum surveys. But a 360-degree view isn't going to happen overnight, he said. Companies that try to make the transition all at once are more likely to fail than those that think strategically and move one step at a time.

"The key is thinking big and starting small," Rich said. "There's really no big bang in all of this."

SOURCE: SearchCRM

Don't Let Data Blind Your Sales Team

Not too many years ago, the concept of driving sales through the use of data drew ridicule from sales professionals. “Data-driven sales” was viewed as an oxymoron, since so much of the selling process depended on the talents and instincts of sales people.

But supremely talented sales people with keen instincts are in limited supply. The rest of us need all the tools we can bring to bear to succeed at selling. And as the amount of data we can collect about the selling process has grown, sales pros have changed their tune. Now even the mega-talents of the selling world are tapping into the vast torrent of prospect data available to them.

Using the Past to Guide the Future

The pendulum can swing too far. An emphasis on data-derived selling insights can blind sales pros to the one thing their talents have provided consistently: a wide-ranging understanding of the context of the deal in its entirety, including emotional cues from buyers. Emphasizing the data often means examining data points outside of the context of the deal, which can lead to misleading conclusions and a negative impact on sales results.

How do you avoid this? First, be careful about your data sources. What data you’re examining and how much weight you’re giving that data can distort your view of the prospect, especially if you’ve picked the wrong metric and made incorrect assumptions about its value to closing a sale. A good example of this comes in lead scoring — taken out of context, repeated visits to product pages might suggest customers who are on the verge of a purchase. Without a deeper examination, these “leads” may be passed to sales, only to reveal that the visitors are students (their failure to look at a pricing page might have tipped marketing off to this) and have thus wasted your sales team’s time. Choose the type of data you use to trigger sales activities very carefully.

As you’re adding data sources to your arsenal, don’t ignore the data you’re already collecting about sales. Your compensation system is a fantastic source of historical data on how sales has performed on an individual, team and company-wide basis. Using this data with a predictive platform can offer a window into future performance that can help with forecasting, coaching, planning and identifying areas where changes to the sales process can boost sales performance. 

Unfortunately, few companies realize the value of this data, and fewer still employ it to look forward rather than backward to merely understand what has happened in the recent past.

Don’t get me wrong — there is genuine value in understanding what has happened in the immediate past. It’s the key to forecasting what will happen in the future. But without a forecasting component to the way you look at this data, you’ll be stuck forever in a reactive mode. Past performance is a good indicator of future behavior, but a better indicator combines past history with an examination of the potential impact of improved training, new incentives or changes to sales territories.

Keeping Your Sales Limber

One side effect of the big data era is that it will force smart companies to also enter the era of the agile business. If the demonstrable facts point that your sales operation must go in a new and surprising direction, you must be ready to change — and, perhaps more important, be willing to change. 

Just as we’re seeing some companies dismiss the benefits of insights revealed through data, there will be companies who take the first steps toward becoming more data-driven but then find their new processes are just as ossified as their pre-data processes were. Don’t allow today’s use of data impede the evolution of tomorrow’s sales techniques — they must drive each other, and that means the way you’re harnessing data is very likely to change regularly. Be prepared for that change, and have a process for change management in mind.

Avoid dismissing startling insights when the data present them. Sometimes, confirmation bias toward our established ideas can cause us to dismiss or even disparage data that contradicts them — “If your data is so good, then why doesn’t it confirm my assumptions?” The very reason we’ve come to depend on data is to find aspects of the selling process we can improve. 

If the data provides a refutation of something critical to your sales process, investigate it — make sure that the insight is from the right data, and not from your misuse of the data — and if you find it to be valid, use that insight to drive change.

And be aware of the swing of the pendulum. In many companies, the pendulum is still strongly pitched toward sales reps’ talents and sales managers’ intuition. In other companies that are more tech savvy, the pendulum swings too far the other way, overemphasizing analytics. 

Your task is to get the pendulum centered, bringing in the technologies that augment the A-players’ sales skills and filling in the gaps in knowledge for the B players, while also capitalizing on the human talents sales brings to the table.

SOURCE: CMSWiRE

How to determine if big data analytics tools are right for you

The growing availability of big data platforms and big data analytics tools is helping organizations gather and analyze data in search of valuable business information and insights that can help them improve their products and services.

Big data analytics tools are used to analyze collections of data and for developing predictive and prescriptive models. Embedding these models within modernized business process applications can improve business productivity and value. At the same time, big data analytics tools are designed to easily scale to employ the resources that are typically available in a big data platform.

In general, the techniques that big data analytics tools frequently provide aren't new; however, it's only recently that the power of data mining algorithms has been adapted to empower the mainstream business user with predictive and prescriptive analyses of data sets featuring a combination of large volumes, a variety of data types and variable structures.

From a user perspective, big data analytics is still an emerging enterprise capability, with inherent risks and requisite time investments to achieve the expected benefits. How can you determine whether big data analytics is appropriate for your organization? Here, we examine how the business use cases dovetail with fundamental data drivers such as volume, variety, etc., to highlight how using big data analytics tools can yield measurable value.

Types of big data to analyze

There's one clear notion that differentiates big data analytics from other forms of analytics: the volume, scale and diversity of the data to be analyzed. In the past, analytical models often were built and then trained through a test-and-refinement process using sample data sets pulled from very large databases. However, today, with computing platforms providing scalable storage and computational ability, there are few limitations on the volume of data that can be analyzed. This suggests scenarios in which real-time predictive analytics and access to large volumes of the right data can lead to improved business performance. The opportunities lie with the ability to blend and analyze the different kinds of big data, such as:

Transaction data. A big data platform provides the ability to capture greater volumes of structured transaction data spanning much greater periods of time. This enables the analysis of a broader array of transaction types, extending beyond point-of-sale or e-commerce purchases to include behavior transactions, such as those captured through logging of Internet clickstream data by Web servers.

Human-generated data. Unstructured data contained in emails, documents, images, audio files, video files, and data streamed through blogs, wikis and -- especially -- social media channels provides fertile ground for using text analytics capabilities to enhance analyses.

Mobile data. As Internet-connected smartphones and tablets become ubiquitous, the apps deployed on these devices are capable of tracking and communicating numerous events. These range from in-app transactions (including recorded events such as a product search) to demographic or status reporting events (such as reporting a new geocode when there's a change in location).

Machine and sensor data. This includes data that's created or generated by functional devices such as smart utility meters, intelligent thermostats, factory machinery and network-connected home appliances. These devices can be configured to automatically communicate with other nodes in a connected mesh as well as a centralized server that can support analysis of the data. Machine and sensor data are prime examples of data emanating from the emerging Internet of Things (IoT). The data streaming from the IoT can be used to build analytical models that do continuous monitoring for predictive behavior (such as identifying when sensor values indicate a problem) and offer prescriptive directives (such as alerting a technician to service a device before it actually fails).

Business use cases

With the different types of data in mind, big data analytics tools can be a reasonable investment if your organization is considering any of these applications:

Customer analytics. This includes analyzing customer demographics, behaviors and characteristics to develop models for segmenting customers, predicting churn and making next-best-offer recommendations to help with customer retention.

Sales and marketing analytics. There are two types of marketing use cases. The first involves using analytical models to improve how customer-facing applications make direct recommendations to the customer. Examples include better identifying opportunities for cross-selling and upselling, decreasing abandoned shopping carts and generally improving the accuracy of integrated recommendation engines. The second type is more reflexive in that it's intended to show the performance of the marketing group's processes and campaigns and recommend adjustments to optimize that performance. Examples include analyzing which campaign addressed the needs of identified clusters or segments, or the success ratios for motivating the campaigns' call to action.

Social media analytics. The content that streams across social media channels provides ample opportunities for analyzing customer sentiment and identifying brand risks when negative information is promulgated about a company's products.

Cybersecurity. Massive cybersecurity attacks on companies such as Target, Sony and Anthem highlight a growing need for businesses to rapidly recognize when cyberattacks occur. Identifying potential attacks involves building analytical models for monitoring massive amounts of data reflecting network activities and corresponding access behaviors to identify suspicious patterns that might indicate a breach.

Plant and facility management. As more devices and machines are Internet-enabled, organizations are able to collect and analyze streaming sensor data indicating continuous measures of power usage, temperature, humidity and contaminant particles, among a myriad of potential variables. Models can be developed for predicting equipment failures and scheduling pre-emptive maintenance to keep items in working order without interruption.

Pipeline management. Energy pipelines are increasingly fitted with sensors and communications capabilities. Continuous streams of sensor data can be analyzed for local and global issues that indicate a need for observation or maintenance.

Supply chain and channel analytics. Analyzing warehouse inventory, point-of-sale transactions and shipments via a variety of channels (e.g., trucking, rail, shipping) results in predictive analytical models that can help with pre-emptive replenishment, inventory management strategies, logistics management, route optimization and notifications when delays imperil timely deliveries.

Price optimization. Retailers looking to maximize overall profitability for product sales may develop analytical models that combine a variety of data streams, including competitors' prices, sales transactions across many geographic regions (to review demand), and information on production, inventories and the supply chain (to monitor supply). The resulting models can be used to dynamically adjust product prices up when supplies are low, demand is on the increase and competitors are unable to deliver, or down when inventory needs to be cleared as seasonal demand shifts.

Fraud detection. An adjunct to the growing risk of identity theft is the growth of fraudulent activity and transactions. Financial institutions analyze billions of transactions to identify patterns of fraudulent behavior, and the analytical models can also trigger alerts to customers when a potential fraudulent transaction might be taking place.

All of these share similar characteristics in that the analysis involves both structured and unstructured data, the data is being accessed or streamed from a variety of sources, and the volumes are potentially massive. In turn, the results of the analyses yield analytical models that can be used to identify patterns from the same sources and streams of data in real time.

Taking advantage of the lowered barrier to entry

Although the capabilities provided by big data analytics tools aren't entirely new, three key factors have helped to lower the barrier to entry and enabled a broader spectrum of organizations to consider adopting this type of technology:

1. Cost. The availability of tool suites with pricing models that make adoption economically feasible contrasts with early vendor products that had high price tags and offered expensive after-sale services to integrate and deploy.
2. Simplicity. These tools are increasingly designed for and targeted to a non-expert community of users. Early vendor products were embraced by statisticians and mathematicians who not only built models, but also understood the details of how they worked. Today's products don't require the user to have an advanced scientific degree to gain business advantage from the resulting models.
3. Performance. Scalable platforms can accommodate the data volumes and computational needs for big data analytics. Today, there are open source platforms enabling massively parallel processing over distributed storage frameworks deployed on commodity hardware whose price/performance ratios are orders of magnitude lower than what has been available in the past.

Integrating big data analytics into the enterprise

The barriers to adopting big data analytics software have fallen, enabling forward-thinking organizations to rapidly try out and integrate these tools within the enterprise. These organizations share some common characteristics in that they have:

• A culture of being data/analytics-driven where business sponsors recognize the potential of information in creating corporate value;
• An environment in which the key stakeholders are aware there's a wide variety of data sources -- some static, and many others dynamically streamed -- that may contribute to analytics processes;
• An environment conducive to proofs-of-concept along with agility in making decisions for adopting technology.

In other words, if your company exhibits some or all of these characteristics, it may be poised to take advantage of integrating big data analytics as part of the enterprise technology landscape.

Once you've determined that your site could benefit from big data analytics tools, the next step is to establish your company's specific needs and prioritize the specified criteria that will be used to evaluate selected vendor products. You can then map those needs to the types of characteristics provided by big data analytics tools, and use those as evaluation factors for a request for information (RFI) or request for proposal (RFP) to be shared with the product vendors. The responses to the RFI/RFP will refine the selection of vendors and influence your choice of big data analytics tool.

SOURCE: SearchBusinessAnalytics

How big data analytics tools can help your organization

The analytics process, including the deployment and use of big data analytics tools, can help companies improve operational efficiency, drive new revenue and gain competitive advantages over business rivals. But there are different types of analytics applications to consider. For example,descriptive analytics focuses on describing something that has already happened, as well as suggesting its root causes. Descriptive analytics, which remains the lion's share of the analysis performed, typically hinges on basic querying, reporting and visualization of historical data.

Alternatively, more complex predictive and prescriptive modeling can help companies anticipate business opportunities and make decisions that affect profits in areas such as targeting marketing campaigns, reducing customer churn and avoiding equipment failures. With predictive analytics, historical data sets are mined for patterns indicative of future situations and behaviors, while prescriptive analytics subsumes the results of predictive analytics to suggest actions that will best take advantage of the predicted scenarios.

In many environments, the processing and data storage demands of advanced analytics applications have limited their adoption -- but those barriers are beginning to fall. The growing availability of big data platforms and big data analytics tools has enabled environments in which predictive and prescriptive analytics applications can scale to handle massive data volumes originating from a wide variety of sources.

What does big data analytics mean?

In essence, big data analytics tools are software products that support predictive and prescriptive analytics applications running on big data computing platforms -- typically, parallel processing systems based on clusters of commodity servers, scalable distributed storage and technologies such as Hadoop and NoSQL databases. The tools are designed to enable users to rapidly analyze large amounts of data, often within a real-time window.

In addition, big data analytics tools provide the framework for using data mining techniques to analyze data, discover patterns, propose analytical models to recognize and react to identified patterns, and then enhance the performance of business processes by embedding the analytical models within the corresponding operational applications. For example, massive amounts of shipping delivery data, streaming traffic data, streaming weather data and historical vendor performance data can be analyzed to devise a model for optimal selection of shipping subcontractors within geographic regions to limit the risks of late delivery or damaged goods.

Big data analytics tools can ingest a wide variety of data types: structured data with defined and consistent fields, such as transaction data stored in relational databases; semi-structured data, such as Web server or mobile application log files; and unstructured data, encompassing things like text files, documents, emails, text messages and social media posts.

Powering analytics: Inside big data and advanced analytics tools

A Google search for big data analytics yields a long list of vendors. However, many of these vendors provide big data platforms and tools that support the analytics process -- for example, data integration, data preparation and other types of data management software. We focus on tools that meet the following criteria:

• They provide the analyst with advanced analytics algorithms and models.
• They're engineered to run on big data platforms such as Hadoop or specialty high-performance analytics systems.
• They're easily adaptable to use structured and unstructured data from multiple sources.
• Their performance is capable of scaling as more data is incorporated into analytical models.
• Their analytical models can be or already are integrated with data visualization and presentation tools.
• They can easily be integrated with other technologies.

In addition, the tools must incorporate essential characteristics and include integrated algorithms and methods supporting the typical suite of data mining techniques, including (but not limited to):

• Clustering and segmentation, which divides a large collection of entities into smaller groups that exhibit some (potentially unanticipated) similarities. An example is analyzing a collection of customers to differentiate smaller segments for targeted marketing.
• Classification, which is a process of organizing data into predefined classes based on attributes that are either pre-selected by an analyst or identified as a result of a clustering model. An example is using the segmentation model to determine into which segment a new customer would be categorized.
• Regression, which is used to discover relationships among a dependent variable and one or more independent variables, and helps determine how the dependent variable's values change in relation to the independent variable values. An example is using geographic location, mean income, average summer temperature and square footage to predict the future value of a property.
• Association and item set mining, which looks for statistically relevant relationships among variables in a large data set. For example, this could help direct call-center representatives to offer specific incentives based on the caller's customer segment, duration of relationship and type of complaint.
• Similarity and correlation, which is used to inform undirected clustering algorithms. Similarity-scoring algorithms can be used to determine the similarity of entities placed in a candidate cluster.
• Neural networks, which are used in undirected analysis for machine learning based on adaptive weighting and approximation.

This is just a subset of the types of analyses used for predictive and prescriptive analytics. In addition, different vendors are likely to provide a variety of algorithms supporting each of the different methods.

The advanced analytics market

The market for advanced analytics tools has evolved over time, and the types of tools that are available vary in degree of maturity and, consequently, in capability and ease of use. For example, there are tools with relatively long histories from some mega-vendors like IBM, Oracle and SAS. Other large vendors have acquired companies whose tools have a more recent history, such as those provided by Microsoft, Dell, Teradata and SAP.

A number of smaller companies provide big data analytics products, including Angoss, Predixion, Alteryx, Alpine Data Labs, Pentaho, KNIME and RapidMiner. In some cases, companies have developed their own suite of algorithms. Others have adapted the open source statistical R language and provide predictive and prescriptive modeling capabilities using R's features, or use the software from the open source Weka project.

A third category of products are those available as open source technologies. Examples include the previously mentioned R language, the Mahout software distribution that's part of the Hadoop stack, and Weka.

In some of these cases (particularly with the mega-vendors), the big data analytics tools are incorporated into larger big data enterprise suites. In others, the tools are sold as standalone products. In the latter case, it's the customer's job to integrate with the big data platform being deployed. Most of the tools provide a visual interface to guide the analytics processes (data mining/discovery analysis, evaluation and scoring of models, integration with operational environments), and in most cases, the vendors provide guidance and services to get the customer up and running.

Who uses big data and advanced analytics tools?

While some individuals in the organization are looking to explore and devise new predictive models, others look to embed these models within their business processes, and still others will want to understand the overall impact that these tools will have on the business. In other words, organizations that are adopting big data analytics need to accommodate a variety of user types, such as:

• The data scientist, who likely performs more complex analyses involving more complex data types and is familiar with how underlying models are designed and implemented to assess inherent dependencies or biases.
• The business analyst, who is likely a more casual user looking to use the tools for proactive data discovery or visualization of existing information, as well as some predictive analytics.
• The business manager, who is looking to understand the models and conclusions.
• IT developers, who support all the prior categories of users.

All of these roles would typically work together in the model development lifecycle. The data scientist subjects a swath of big data sets to the undirected analyses provided, and looks for any patterns that would be of business interest. After engaging the business analyst to review how the models work and evaluate how each of those discovered models or patterns could potentially positively affect the business, the business manager and IT teams are brought in to embed or integrate the models into business processes or devise new processes around the models.

From a market perspective, though, it's interesting to consider the types of businesses that are embracing big data analytics. Many of the early users of big data technologies were Internet companies (e.g., Google, Yahoo, Facebook, LinkedIn and Netflix) or analytics services providers. Each of these companies relied on operational and analytical applications requiring fast-flowing streams of data to ingest, process, analyze, and then feed the results back to continuously improve performance.

As appetites for data expand among companies in more mainstream industries, big data analytics has found a place in a more general corporate population. In the past, the cost factors for a large-scale analytics platform would have limited the adoption to only the very largest businesses. However, the availability of utility-style hosted big data platforms (such as those available via Amazon Web Services) and the ability to instantiate big data platforms such as Hadoop on-premises without a large investment have reduced the barrier to entry. In addition, open data sets and accessibility to fire hose data feeds from social media channels provide the raw material for larger-scale data analyses when blended with internal data sets.

Larger businesses may still opt for high-end big data analytics tools, but lower-cost alternatives deployed on cost-effective platforms enable small and medium-size businesses to evaluate and launch big data analytics programs and achieve the desired business improvement results.

Now that we've examined the different types of tools and their uses, the next step is to determine how these tools could benefit your company. By taking a look at the various use cases for big data analytics, you will begin to see where a general big data analytics capability can be leveraged for creating and enhancing value.

SOURCE: SearchBusinessAnalytics