The Stages of Evaluating New BI and Analytics Solutions

Part 1: Analytics Goals

Consider Your Use Cases

Analytics use cases fall into two main categories: internal or external. For both categories, the ability to scale is crucial for success.

• Internal Analytics

Internal analytics includes use cases that aim to answer questions regarding the health of your business — also known as business intelligence (BI).

BI is an umbrella term that encompasses any measurement of business performance. These use cases can relate to churn rates and prevention; customer lifetime value; customer segmentation; risk modeling; customer service metrics; system performance and optimization; and more. Processes can be measured whether they have a direct impact on revenue or not. After all, with the right tools and skills, something can be learned from data surrounding nearly anything.

One implementation approach is to determine which business-critical processes must be tracked and measured, and then to build out your analytics strategy. Another method is to identify weak areas of the business, followed by analyzing the relevant data to combat bottlenecks, inefficiencies, or broken systems.

• External Analytics

External analytics, also known as distributed analytics, can be client-facing or partner-facing.

Client-facing analytics are data products that help your clients assess the value they’re achieving or delivering — due to using your products and services, or by leveraging your data sets. This is especially common for marketplace businesses, such as leading European retailer Zalando; banks or payment providers; transportation industry resources; customer service platforms; and others.

Most clients likely will access your products or services via some sort of online portal or dashboard. It’s also possible to develop analytics that sit directly in this portal — also called embedded analytics. Embedded analytics are directly integrated into the client’s workflow, thus minimizing barriers to usage and providing a purchase incentive.

Data products can be developed and delivered as a standalone product, as well, which may be a simpler option for providing analytics as an add-on product — or as the main product itself. Either method can provide tiered access, where clients pay an additional cost for advanced analytics or for more robust information.

Partner-facing analytics can provide significant benefit to supply chain partnerships or any other codependent business relationships. These use cases can help both parties determine their respective strengths and weaknesses, and subsequently make their relationship more cost-effective and efficient.

Consider Your User Needs

Different use cases will fulfill needs for different audiences — each with unique skill sets and terminology. The key to a successful analytics implementation is to provide a solution that meets the needs of each user group.

• Business Users

Business users will be interested in understanding how the business is performing and whether their key performance indicators (KPIs) are being met. They’ll likely also look for metrics on customer stickiness, selfservice functionality, shareable reporting, consistently reliable and clean data, and easy-to-understand metrics and terminology.

• Technical Users

Technical users, such as developers, will value capabilities that allow them to support analytics initiatives without placing excessive demand on their bandwidth. These features might include a fault-tolerant data pipeline, multiple data integration options, easy code maintenance, and automated data refresh and distribution.

• Data Analysts and Scientists

Since data analysts or scientists will be responsible for maintaining the backend of your dashboards, these users will prioritize the ability to be able to write their own metrics, implement data models, easily mitigate risks and comply with security directives, and speed up time to market.

• Accessibility and Permissioning

Another key point to consider is how each audience segment will interact with different types of data. Do you manage sensitive data or personally identifiable information? Does your organization need tiered access based on role level? Or, do you want to provide anonymized benchmarking data to your clients?

In each of the aforesaid examples, the BI and analytics solution that you adopt will need to provide mechanisms for segmenting audience access and defining metrics appropriately for each user group.

• Data-Driven Business: Cost and Revenue

There are two primary avenues to drive growth with data: selling data products and services, and achieving business efficiencies with analytics. True data-driven businesses can do both.

The right tool for your business is one that will grow with you. As such, it’s critical to choose a solution that makes growth easier — not one that acts as a bottleneck.

The old adage “every company is a technology company” has morphed into something new: Every company is a data company. All businesses are (or should be) collecting data. The next step is determining how to manage and use it.

Part 2: Evaluation Process

Now that you’ve considered your use cases, user needs, and types of distribution, it’s time to start evaluating solutions. The evaluation process can be broken down into three general steps, followed by your purchase decision:

• Research and discovery
• Product demos
• Proof of concept

Note that steps one and two will likely overlap with each other, because you may find yourself conducting deeper research on a product after attending a demo. But, for the sake of clarity, we’ll discuss the two steps separately in this e-book.

Step 1: Research and Discovery

Overview of considerations

Each stakeholder group will focus on a different aspect of the solution based on its goals. The chart on the next page provides a brief overview of what each group should consider.

Make sure you involve all key stakeholders — people from IT and security, data management, finance, and any line of business leaders who will need access — to choose a solution that meets all the necessary requirements and top priorities.

Data sources and integration

Especially for business users, be sure to work with your tech-side peers to understand whether the available deployment and data integration options support your data strategy. The analytics provider should offer a set of tools that connect to your data sources and ensure high performance.

Data sources can include cloud data warehouses, file storages, messaging queues, and others. Ideally, the provider will also offer out-of-the-box data integration options that will cut down on the time needed to deliver your first data insights.

Data modeling

Does the solution provide flexible data modeling capabilities with a powerful semantic layer? Your platform’s data model will impact its scalability. A logical data model provides a better user experience in the form of faster query formation and more reliable analytics. A semantic layer translates source data into business terms for easy-to-understand insights.

Distribution

Flexible distribution options will help you scale analytics beyond a small group of users. As explained in the callout on page 5, depending on your business goals and use cases, you could distribute analytics in a variety of ways: standalone tools or dashboards, static reports, one-off data visualizations, embedded dashboards, and more.

Customization

Regardless of your distribution choices, customization is an important part of your user experience. Also mentioned in the callout on page 5, white-label products create a unified feel for your products. Data and analytics products can also offer customization for your interface and data visualization color schemes. Developers may have some options in the form of component libraries, dashboard plug-ins, and API availability.

Governance

Outstanding governance functionality is mission-critical for large-scale scalability and security. Talk to your shortlisted BI providers about available tools for efficient and automated customer, user, and data management. Operating and maintaining a complex solution poses many security risks. Without governance automation tools, human error always poses a threat.

Monetary costs

There’s always a budget to work with, of course. But for a solution that is used by multiple parties within your organization (e.g., product teams, other lines of business, partners, and clients), it’s crucial to understand each solution’s pricing structure.

Many solutions will be priced by user seat, which makes scaling incredibly expensive. Some solutions, like GoodData, price by workspace, which can be thought of as a flat rate for a group of users. Others may charge by data volume or storage and usage.

Resource costs

Additional costs to consider are time and employee resources. For example, how long will it take to develop a customized solution, roll it out, and onboard your users — and what demand would that place on your internal teams? “Time is money” may be a cliché, but in the business sense, it’s true. As you wait to implement or overhaul an analytics solution, consider the cost of each day that you do not have the final solution in place.

Once the solution is rolled out, it’s also important to understand what the solution’s maintenance requirements are, and how many of those requirements will fall to your internal teams. Does your current team have enough bandwidth to manage the solution? Will you need to hire additional headcount, or will you have to purchase services from the vendor? Factor these into your total cost of ownership (TCO) calculations.

Success stories

Unless you’re one of the first prospects of an earlystage startup, the vendors in question should have case studies or success stories available on their websites.

Look for a case study that matches or resembles your use case, whether that is by industry, business impact, or type of analytics. It’s also good to look for any quantifiable impact the vendor has achieved for its clients, such as cost savings, adoption rate, customer retention and acquisition, or growth rate.

Step 2: General Demos

Once you’ve gathered a shortlist of solutions, it is worth taking the time to evaluate their general demos to ensure they meet usability needs for each of your users or user groups. Most vendors will have a general demo available on their website or via a scheduled meeting led by their sales or engineering team. As stated above, this step typically overlaps with step one (taking place alongside your research and discovery efforts).

Some top things to consider include the following.

• Interface design: Is the interface intuitive for your non-technical users?
• Self-service usability: Does the solution allow users to create their own visualizations, rearrange dashboards, or create reports — all without the help of a tech or data specialist?
• Out-of-the-box tools: What and how many out-ofthe-box functionalities does the solution offer? Look into data connectors, data visualization, embedding, and others.

If the general demo is held live, this is also a good opportunity to ask any outstanding questions about features, integrations, and capabilities.

Step 3: Proof of Concept

If you have already narrowed down your shortlist to two or three solutions, this is a good opportunity to bring in a stakeholder for both the tech and business audiences. Hold an internal sync to decide what remaining requirement questions you want to have addressed during the proof of concept (POC) stage.

Then, with each candidate, request a POC, in which the vendor will prepare a demo that either uses a sample of your data, or shows how they could fulfill your primary use case and what that would look like. You can also request that they answer any questions in advance of the session; that way, the time can be used effectively, and the vendor knows which features and capabilities to highlight.

When requesting the POC, note to the vendor any specifics that you’d like to see or understand, such as:

• Customized interfaces or visualizations
• Query production and metric-building
• Security and accessibility limits
• Specific calculations (e.g., benchmarks, forecasting, real time)
• Data loading and integration methods

The POC stage typically lasts a couple of weeks, during which you’ll have access to your customized prototype. Use this time to thoroughly validate any key areas that you want the solution to fulfill. Review and get familiar with the platform mechanics, and take advantage of access to the vendor’s engineering team (who can assist whenever needed).

If you’re looking for analytics to meet the needs of an external use case, you may even go a step further and involve one or two customers in the evaluation process. Talk to them and extend them access to the prototype. This will help you gather valuable feedback; for the customer, the incentive includes influencing the design of the analytics solution in its favor.

What’s more, try to gain insight into future roadmaps and anticipated capabilities, as well as to clarify any pricing questions. Dig into what kind of support and training the vendor offers; what its onboarding process looks like; and the TCO — which includes services required by the solution, but not necessarily provided by them.

Part 3: Purchase Decision

All things considered — from the perspective of each decision-making stakeholder — which solution is best for you?

How to calculate a decision

Your team will likely incorporate both quantitative and qualitative information when making this decision. To prevent a stalemate, it may be helpful to approach the decision process with a rating and weighting system — similar to Forrester’s Wave calculations, if you’ve seen them.

Build a spreadsheet that lists each feature and capability requirement. Across the purchase team, agree on a weighting percentage that indicates how important each feature or set of features is to the overall purchase decision. Then, have your key stakeholders rate their satisfaction with the solution’s fulfillment of those needs with either a 1, 3, or 5 (1 being “not fulfilled,” 3 being “partially fulfilled,” and 5 being “completely fulfilled”). This method will hopefully produce a score that determines the quality and fit of each solution. If there is a tie, move on to qualitative discussion.

About GoodData

GoodData is on a mission to break data silos. Realtime, open, secure, and scalable, GoodData’s leading composable data and analytics platform provides a single source of truth across organizations and to their customers. To this day, GoodData has helped more than 140,000 of the world’s top businesses deliver on their hosted or cloud-native analytics goals and scale their use cases — from self-service and embeddable analytics, to machine learning and IoT — all the while maintaining the performance, cost-efficiency, and easy change management of such a central and integrated solution.

GoodData has teams in the U.S., Europe, and Asia, with customers including leading software companies (SaaS), global financial and payment institutions, and multi-brand e-commerce platforms.