Qlik & Databricks | Advanced analytics & machine learning

Businesses are increasingly looking for ways to leverage advanced analytics and machine learning (ML) to gain insights and drive decision-making.

Integrating Qlik with Databricks offers a powerful combination that brings advanced analytics and ML within reach for organisations, enabling them to harness the full potential of their data.

The Integration: Qlik and Databricks

Qlik is renowned for its data visualisation and business intelligence capabilities, providing an intuitive platform for data exploration and insight generation. Databricks, on the other hand, is a unified analytics platform that simplifies big data and AI through its collaborative, scalable environment. Integrating these two platforms allows users to perform advanced analytics and machine learning in a seamless and efficient manner.

How End Users Consume ML and Advanced Analytics Outputs

So how do end users currently consume machine learning and Advanced analytics?

The first way is static outputs. The model is run (probably on a schedule), the output is now just static data, like any other data. it’s stored somewhere, and it’s read from the store into a dashboard, where it is consumed by an end user (just as they would any other data).

Now, the model may be run with high or low frequency, but at any one time - the output is static

Examples are things like a customer churn score, or possibly product recommendation, predictive maintenance, any sort of classification algorithm or text analysis.

The second way is dynamic outputs.

This starts with an operational question whose answer changes based on an unpredictable user-defined variable or selection.

The end user drives this process by making their input, which is then processed, and the output is consumed.

So if you take static examples above, you’re now able to perform what-if analysis on churn, predictive maintenance, and so on.

But you can also create guided, decision support tools, translations, LLM/chatbot interactions, or generate new visualisations.

Challenges in Deploying Dynamic ML/AA Models

When a dynamic, interactive model is needed, organisations can go two ways.

The first leaves the model as a code-based asset, requiring technical coding knowledge and probably a reasonably detailed understanding of the underlying data science.

This means it is not widely accessible, and all access is mediated through a technical gatekeeper - probably the person who developed it.

The model is probably configured to run on one person’s local machine, and is not governed or shareable.

The second way puts it in the hands of operational staff, who may not be technical. The challenge here is that their access is usually provided in some ad-hoc way with a custom web front end.

The interface - and any governance and security - is done on an ad hoc basis, and is likely clunky, unattractive and unmanageable.

‍

So, to democratise ML, we’ve gone and done this….

Qlik, meet Databricks

Using Qlik Sense’s advanced analytics connectors (AAC), organisations can extend Qlik’s computational limits.

For example:

• Dynamic Repartitioning: Compute catchment areas on-the-fly using Databricks, enabling dynamic operational planning.

• Text Processing and Decision Support: Interact with Large Language Models (LLMs) within Qlik for interpretive/narrative extraction and whatif analysis.

• Enhanced Visualisations: Access new charts – and even animations – via R/Python without writing Qlik Extensions.

• Offline geo-analytics and routing: Implement functionality such as custom routing for offline Qlik deployments.

• Selection Logging: Relevant for high-audit environments like government departments.

Complex calculation

In this example, the catchment areas are recomputed on-the-fly, using a serverless Databricks endpoint for an MLFlow model running Python code.

The areas are selection-dependent, so they cannot be precomputed.

‍

‍

Text processing and decision support

Qlik’s AACs allow end users to interact with LLMs, chatbots, translators, etc. They can be used to expose data to GenAI for interpretive/narrative extraction. AAC also allows users to perform ‘What-if?’ analysis, leveraging complex underlying data models.

‍

‍

Animation and extension

AAC can be used to trigger selection- or variable-dependent charts using R/Python.

These charts can be interactive and even animated. This approach opens up a pathway for embedding into Qlik Sense an immense library of R/Python visualisations, without writing Qlik Extensions.

‍

Offline processing

AAC can be used to implement functionality otherwise not available. Examples include custom/offline routing for Qlik Client Managed deployments.

‍

Applicability

Now the examples above use the advanced analytics connectors to push complex computation out to Databricks, Azure functions and external APIs.

As far as we know, Notitia are only people who’ve pushed this capability this far.

A bit about the technologies we’ve used so far:‍

Databricks because it is powerful and growing so fast (plus, we’re Databricks partners!)
Azure Functions because it’s easy and cheap (plus, we’re Microsoft partners!).

And our REST API wrapper opens up everything the custom connectors do, and so much more - from all the hyperscalers’ ML offerings to tiny single purpose APIs.

But the know-how we’ve got from this exercise can be applied to anything with a REST API, or using Qlik’s custom connectors for the following: