ModelOps Is Just The Beginning Of Enterprise AI

Most of this year, enterprises have been reviewing the lessons learned in the past few years from their Enterprise AI initiatives, i.e., what has worked, what hasn’t, and how to move forward to modernize their infrastructures and take full advantage of AI. According to Garner’s recent research report, from 2018 to 2020, only around 47% of projects in enterprise organizations are in production. The rest are stuck in the pre-production phases. Many enterprises are still trying to get their AI projects into operation and contributing to the business.

What’s holding these enterprises back?

Last week, I spoke to Stu Bailey, the Co-founder and Chief Enterprise AI Architect at ModelOp, a company trying to help enterprises implement ModelOps, the key component in operationalizing enterprise AI. We caught up following a roundtable that Stu moderated in September that featured many industry leaders along with Erick Brethenoux, VP Analyst with Gartner's, and the lead for their AI research. Erick Brethenouxc introduced Gartner's Enterprise AI framework, and the panelists highlighted key challenges that Enterprises are facing.

A Disciplined Approach

Enterprise AI spans different departments, and along with new AI initiatives, it often incorporates existing AI or Big Data implementations from the last decade. Enterprise AI presents a unique opportunity for CIOs to consolidate existing data warehouses, data analytics, and business intelligence applications from various departments to find company-wide use cases that will heavily impact the bottom line.

Each department and business unit will have its unique applications and use cases for AI and needs to freedom to use the most appropriate tools and techniques, which in the AI world are changing rapidly. But at a corporate level, there's a need for unified approaches to governance and operations that necessitates a centralized, disciplined approach to modelops.

Stu Bailey, Co-founder and Chief Enterprise AI Architect  of ModelOp says, “For almost all participants in our roundtable, everyone agreed that ModelOps lies at the center of the broader enterprise AI strategy.”

A recent Gartner webinar explains that successful organizations do two things that allow them to be more disciplined in their Enterprise AI approach.

1. They start to think about production-sizing AI in the first phase of the project. The first phase is always “Understanding the business.” In this phase, they spend a lot of time pulling in essential people such as SMEs, IT, application developers, ml engineers, and data scientists. This team helps to come up with the use cases and the models. More importantly, they define key performance indicators of the models and plan out how the model will be moved into and maintained in production.  
2. Governance is the other critical discipline that requires special attention in AI projects. In regulated industries, good governance is essential. Enterprises that successfully operationalize their AI projects often have a better handle on data governance, model governance, and application governance than their peers. But, more importantly, models have life cycles. Unlike conventional software, models are not static. There’s the concept of data drift, mathematical drift, and business drift that necessitates continuous monitoring and updates to models. Keeping track of model behaviors, measuring them, and testing their production effectiveness are critical steps to successful Enterprise AI.

Bailey says, “Every model represents a very unique piece of intellectual property derived from the company's data. Models are created from data but they are quite different from datasets because they have complex relationships with the business structure.. As data change and as the business evolves models must change.  They don’t fit into the patterns that have evolved to manage software because they encode the organization's most valuable proprietary information.”

ModelOps Can Inspire Business Accountability

Enterprise AI projects touch many functional groups across the enterprise, including the data scientists who create models, the data teams that manage the data, the development and operations teams, the governance organization, and the business unit that sponsors the model's development to meet a business KPI. These different groups tend to operate in silos, which creates friction and slows the process of moving models from the lab into production.

The role of ModelOps is to provide an efficient and transparent framework for operationalizing AI across the enterprise. It offers the opportunity to manage business accountability, share it across groups, and bring stakeholders such as compliance, finance, operations, and other business unit functions closer together to impact business outcomes.

Bailey says, “One thing the panel was very clear about is that ModelOps is more of a business accountability capability than a technical capability. It’s certainly is very technical, but it has to allow for model life cycles, integrate into existing compliance programs, and be a part of the risk management program. For example, there have to be audits, approvals, governance functions for models to be in production in the banking industry. So, it’s clearly a business accountability function that reaches across the enterprise..”

Changing the Game With Regime Changes

For large enterprises that operate in ever-changing business environments, implementing enterprise AI means learning to change with the business environment. The critical competitive advantage in any market is speed. If a firm can quickly change their model adjusting to the business environment, there can potentially be exponential business advantage just being the first in the marketplace to adapt.

Bailey says, “Many of these models have to be refreshed quite regularly, sometimes on a daily basis in terms of new data that the markets have seen. Being responsive to regime changes such as when the conditions in the market change, and starting the process of replacing the models according to that change is very useful. It is a case for automation when you need the speed, and there can be potential legal implications. You want it to be automated.”

Two New Roles Emerging in Enterprise AI

Along with data scientists and machine learning engineers, two roles emerge as central to bringing enterprise AI into production: The Enterprise AI Architect and the Model Operator.

Model Operators are people who monitor hundreds or thousands of models in production. They have SLAs, and they have benchmarks. They take care of problems and escalate according to procedures. Their job is to make sure that models run smoothly in production.

On the other hand, the Enterprise AI Architect is becoming the central liaison for different stakeholders to receive the big picture perspective on Enterprise AI projects. They are the people who are designing the life cycle for every model. They understand the business impact, stakeholder needs. They can translate these impacts and needs into sound model risk management plans in production. They are the go-to for data scientists, machine learning engineers, dev-ops, and data ops groups to define processes and systems that enable them to collaborate easily and effectively.

Bailey  says, “The role of the Enterprise AI Architect is really new. The EAIA is responsible for orchestrating the team around critical assets including models, their production life cycles, their KPIs and metrics, approvals, regulatory reports, retraining and refresh requirements, and operational requirements for, say, deploying a model into a particular cloud infrastructure. The Enterprise AI Architect articulates life cycle, either as a series of documents or as a set of functional automations using a tool like our ModelOp Center.“

Given its central importance in operationalizing AI initiatives, this Enterprise AI Architect is becoming a highly visible role as enterprises strive to be AI-driven.

The CIO Becomes the Chief Innovation Officer, Too

Due to the business impact of enterprise AI initiatives, the CIO is increasingly able to impact the business bottom line, not by reducing costs, but by developing a competitive advantage.

In many organizations, the information technology group has played defense, focusing on cutting costs while the business units take the lead in driving innovation. With enterprise AI initiatives replacing legacy systems and consolidating data for governance, innovation becomes the information technology mindset. A new generation of CIOs are emerging to take on the challenge. They are people who are more creatively minded and can navigate disruptive changes to drive business advantages.

Bailey says, “Information is increasingly the core proprietary asset for any company. When the CIO’s organization can automate operations and governance and focus more on the top line, then it’s a huge opportunity.”

ModelOps Is Just The Beginning

Our ecosystem of algorithms has dramatically expanded. Our AI intelligence will become more and more adaptive. Therefore, ModelOps is just the beginning of the journey. Gartner's webinar outlines the future for Enterprise AI, which will comprise AI that involves CompositeAI and GenerativeAI eventually. At each of these stages, there will be challenges and complexities that enterprises will need to overcome.

For instance, at the next level of complexity, with systems that can produce new decisions and provide insights by themselves, there will be a new set of metrics that will measure these systems' effectiveness and govern these models.

Compared to the complexities of future challenges, implementing a company-wide operational flow for models is similar to when companies instituted KPIs to measure IT service deliveries using ServiceNow products. It merely takes mindset changes across businesses, tools, and processes to enable the operating environment.