How ontologies drive proactive insight discovery
Most data teams need to balance two opposing priorities in the tools they provide business stakeholders: accuracy of insights and breadth of data availability. Typically, you can not maximize one without sacrificing the other. The holy grail of the “text2sql” and “chat with your data” space is to break the dichotomy and allow full data democratization with full confidence in accuracy of insights. With the latest advances in foundation models, I believe we’re getting close to being able to do this. I also believe that the ability to “chat with your data” is only the beginning of the AI revolution in the data space. The next frontier is proactive insight discovery; the ability to answer questions you didn’t know you wanted to ask.
I’ll illustrate what I mean by starting with a business intelligence asset that’s existed since the 90s: the KPI dashboard.
KPI Dashboard: Answer one question well
Picture your typical KPI dashboard. Many key decisions hinge on accurate reporting of KPIs, and presenting the KPI in a central and trusted place is much more important than enabling ad-hoc analyses around it. It has a very high accuracy requirement, and a very narrow data availability requirement. When users view the KPIs, they might have a bunch of followup questions: Why did the number go down this week? How does this quarter compare to the last 4 quarters?
You send these requests to the data team, for them to prioritize and tackle with a days-to-weeks timeline. You don’t satisfy them within the KPI dashboard. In the KPI dashboard, flexibility is traded for clarity to the end user. They might have a hundred other questions they want to ask about the KPI. Curiosity is a virtue! But the KPI dashboard is designed to answer the KPI well and just the KPI, not to satisfy curiosity.
Realistically, unless there really is a burning curiosity in the heart of the user, many of those questions disappear into the void. Hopefully someone else at some point catches those lost whispers of curiosity and pursues them with more vigor. You don’t know what you don’t know and these questions sometimes unveil critical information relevant to making better decisions.
Chat with your Data: Answer any question well
Contrast the KPI dashboard with the emerging “chat with your data” experiences that AI-forward data teams are starting to provide. The “chat with your data” experience matches the users curiosity where it is**.** The user does the creative work and has a set of questions they want to ask, so they go to a chat interface to ask questions and get answers.
Whatever their heart desires, the magical AI fulfills. Providing a reliable “chat with your data” experience is worlds better than providing your team with a set of static dashboards. It enables them to move at the pace of their curiosity and imagination. To draw an analogy to the consumer content space, this is a similar experience to Google Videos or the classic YouTube experience. The primary interface to get the videos you want is the search bar, where you come with your own curiosity about what videos you want to see, and you’re satisfied once you complete a search and find a cool video.
Ontology Building: Answer questions you didn’t know you had
There’s an emerging category of ontology building tools, which enable teams to automatically build business-relevant structures on top of their sprawling raw data. The ontology building experience is one which stokes the end users curiosity. It answers questions they didn’t they even had.
When you build an ontology, you model raw data in terms of the objects and events that users actually think about. You map data to their real world. With every object and event in the real world as the rooting structure, an ontology generation AI like AstroBee populates your data model with useful properties and statistics that are within your raw data, and are sensibly relevant. These insights are useful, verifiable by data experts, but are not direct answers to questions any of your employees have ever asked.
Think of ontology building as hiring a team of senior data scientist who are experts in your particular industry and have complete visibility into your data. Such an employee would come to your team knowing what questions to ask, and would munge through your data to figure out the answers to those questions. They would then provide those insights to you on a platter.
To complete the analogy to the consumer content space, this is similar to modern TikTok-esque feeds. The user is presented with content they didn’t know existed and they didn’t know they wanted to see, because the underlying algorithm understands what is trending and what the user specifically is interested in seeing. Generated ontologies offer a chance to do the same for insights on top of enterprise data. By understanding the underlying data, and understanding the objects and events relevant to the users real life workflows, tools like AstroBee can discover and push important insights to the people who need them.
AstroBee is in early beta. We’re gearing up to launch in the coming weeks. If you’re interested in assisting in pre-launch testing and gaining early access discounts on launch, reach out to us at galen@astrobee.ai!