Wednesday in the Woods · episode 002
Goals and Metrics in Machine Learning Maturity Models
13:29youtube · Licensio, LLC
description
You wood have to admit that there were some production issues last week with the first episode, so I decided to aim for a higher level of production. Unfortunately, as happens so often when you branch out into something new, I forgot to leaf something out of the shot. I promise I'll tree-t you to the perfect fall production next time.
But back to the content - because how else are you going to fit a micronap into your day. To be fair, as Jillian Bommarito pointed out, 13 ½ minutes is more of a power nap (you can delete Calm and Headspace from your phone now). Anyway, last week, we discussed goals and metrics and I left off by promising a real example of goals and metrics across the maturity scale. Today, we discuss the machine learning maturity journey and what it might look like for an online dress shop.
Maturity is obviously a journey, but how can you figure out where you are on the path? When we work with organizations, we tend to focus our discovery on evidence of certain activities or concepts. These hallmarks of maturity exist in more than one dimension, but when you put all the clues together, the picture helps identify where an organization is today, how and why it got there, and where and how it might improve.
Our maturity assessments generally cover a range of dimensions, some of which depend on the context or industry, but almost every client we work with gets to talking about goals and metrics. These two dimensions of an analysis are, in my opinion, the most important, which is why we cover them in detail to start our series.
Goals begin with, well, nothing; in the beginning, these ideas are unknown to an organization. You could think of this as maturity level zero, as there isn’t really any activity or idea to assess.
At the first stage of maturity, companies become aware of a concept like data science or machine learning and decide that they want to associate with that label. There’s nothing actually going on yet - no capabilities, no dedicated resources, no purposefully-designed systems - but there is a spark of intention directed towards a goal. Sometimes this spark comes from the top, lit by an airport magazine or HBR article, and sometimes this spark comes from the bottom, with a curious and engaged employee. Either way, it needs fuel to grow.
At the second stage of maturity, the flame begins to burn and goals begin to form around capabilities. Goals sound like specific techniques, resource roles, or vendor types. For example, management might seek to hire a data scientist or machine learning engineer, or a self-driven employee might want to build a clustering model. The capabilities are possibly connected to discrete tasks in their day-to-day business life, but are generally fragmented. Data collection is ad-hoc, pipelines are not replicable, and systems for research and productionization are in the Pre-Cambrian stage - but there is life in the world.
In the third stage of maturity, organizations begin to align activities with operational goals. Goals start to mirror overall processes or financial metrics, especially at the department or business unit level. We see multiple types of data science roles, including an understanding of the difference between data collection and annotation, data governance and management, research and development, and productionization, though some of these functions are typically still externalized to outside vendors. During this stage, organizations typically experience the most immediate and demonstrable financial impact from their activities. Sometimes, these activities even take on *too* much life of their own, as the capability to use machine learning has outstripped the organization’s governance or compliance of that capability.
Finally, in the fourth stage, organizations can focus on strategic goals, including areas like customer experience and brand, business sustainability, or governance. Organizations in this stage have typically established dedicated, internal roles and teams, including management, for those from stage three. More importantly, these organizations dedicate resources towards compliance and ethics related to their activities, such as incorporating analysis of these activities into risk and ESG programs or monitoring models for training data bias. Capabilities around regulatory research, lobbying, and policy communication feature at the top end of the scale.
Psst - wake up. Micronap time is over. You earned it!
For Tomorrow and Friday, we’ll be posting two more example illustrations - one for an online event management and ticketing company and another for a manufacturing company. And as always, we hope you enjoyed a moment of serenity if you made it through (or 12:38, for those of you who are just here for the forest soundtrack)!
transcript
~1,834 words · showhide
from the human-edited captions published with the video. timestamps jump to that point in the video.
- 0:00
(birds chirping) (clothes rustling) - Good afternoon. And welcome to the show today on this week's Wednesday in the Woods, we'll be talking about hallmarks of organizational maturity in the context of machine learning. So last week we talked about goal setting and metrics and what they have to do with machine learning. And this week we'll talk about those two same topics, goal setting and metrics with three specific examples of businesses and what we tend to see in terms of the factors or things on the ground, in an organization at each maturity level.
- 1:33
So remember we had four maturity levels that we tend to use for an assessment, unlike CMMI, And today we're going to be looking at the goal setting that we see at each maturity level and the metric that we see at each maturity level for three different types of businesses. So the first business that we'll think about as a dress shop an online dress shop. And the dress shop, like many organizations might go through a journey where it begins with some first level, largely awareness of a concept and a desire to be able to apply a label or descriptor to itself.
- 2:09
So it might say something like, "I want AI. I want to be able to build AI into my business". It doesn't really know what that means. Just beginning to develop an awareness of the concepts. Typically no resources on the ground, and maybe some interest organically from resources, maybe top-down and definitely no real capability development, let alone execution. The dress shop might then, either through bottom-up organic skill development or through acquisition of resources from the market, begin to move into this second stage of that journey, the second level of maturity, where the organization focuses on capability development.
- 2:48
So they've established an awareness and they know that they want this descriptor to apply to themselves. They focus on capability development, something like "I want to build a prediction model so that I can tell a customer what kind of dress they might like", right? This typically begins at, at level two with simply knowing that they want to build a prediction model. So they're taking a step from describing themselves as a type of company to understanding what capabilities are required for that descriptive to be true.
- 3:21
And at that stage, they're really focused primarily on acquiring those capabilities. Again, whether it's staffing or training, or simply just pedaling enough on the bicycle that they can stay up for 30 seconds. At the next stage, which hopefully will occur for most organizations, they begin to look more at operational goals. So somebody said they wanted to build a prediction model, right? And they want that to be used for something, right?
- 3:50
Maybe as a dress shop, they want to be able to predict which dress does a customer like. And so they're moving now from the building a prediction model as the goal to building a prediction model so that they can recommend a dress to a specific customer so that they can do something like lower the cost of procuring materials and decrease the time to fulfill that material to the customer. So a good example is maybe I sell dresses online, but I don't just sell the dresses.
- 4:20
I take some interests from the user, I asked them to upload their favorite outfits from their wardrobe, and then I go and I find dresses from a catalog of designers or producers that I work with. And I could take a just in time model, right? I could wait for a customer to say yes to a product that they see in my catalog. And then I could purchase that and sell it to them. So purchase from the supplier designer, sell to the consumer, and I'm going to have to go through a in time supply process for that, not likely to get economy of scale and purchasing, not likely to be able to do anything from a logistics network in terms of optimization.
- 5:01
So whatever FedEx or ups or, or ground freight or air freight, depending on your fulfillment model, you're paying what you're paying, right, from wherever your supplier or designer location is. If at this stage they're focusing on operational metrics, they're going to get a lot out of that, right? They're going to ideally be able to determine that there are, for example, a hundred people in a region that might be interested in a certain outfit.
- 5:28
They might be able to purchase those outfits in bulk or secure some kind of discount. And then they might be able to proactively locate those dresses at a regional warehouse or some kind of third-party or for, for party optimized logistic warehousing network. And so the move from, I know that I want to be able to build prediction models, to having a specific operational goal, like reducing time to deliver it to the customer after they order or decreasing the cost of acquiring the dresses is a big one.
- 6:05
It's where much of the balance sheet measurable in the short term at least wins are won. At the fourth stage of maturity, on the other hand, we begin to connect the dots, right? So yes, it's good that we have these metrics. Yes, it's good that we're focusing on discreet operational goals, but at the end of the day, we might want to impact the company at a more strategic level. Right? And so the focus as we move into the fourth, and typically highest level of maturity organizations, is on actually affecting strategic outcomes through our use of these tools.
- 6:43
So for example, the dress shop might say, "we want customers to associate us with what we want when we want it". Right? Very simple mantra, which is we want somebody to not know that they needed a product until we told them or showed them that they needed it. And to have it basically at their fingertips, because we'd predicted that they'd wanted it. And we put it close enough that we could fulfill it rapidly or cheaply.
- 7:11
It's their dream, right? When we build businesses like this, to be able to move from simply a marketplace matching just in time fulfillment, to predictive recommending systems with decreasing costs and increasing satisfaction. It's a journey. So hallmarks at each of those four levels, from a metric perspective, those customers often demonstrate a similar level, but not always the same, right? So some cases organizations might actually start with a very discreet operational goal.
- 7:44
They might start with a goal like the one we described to predict which dresses a customer might want to purchase, but they may not necessarily have the same level of maturity from their metrics, right? Either in some senses, they may not yet have a true accounting system. They may not have full visibility into the logistics costs. They may have one-off purchases or not enough systematization to understand either retroactively or proactively the cost of acquiring those dresses.
- 8:09
Or they may simply not have the technical understanding as a team to implement what they otherwise would have access to from a data perspective. So just to again, walk through the stages from level one to level four and what kind of metrics we'd see in the dress shop, examples that we said the first stage, they just know they want AI, right? There's typically no metrics on the ground because in many cases there aren't even resources who would be able to implement a metric.
- 8:34
At the second stage of maturity, we typically see resources on the ground who are using a metric, but not necessarily the right metric from both a technical perspective or a holistic business perspective. So at that stage, the goal of the organization is to predict which dresses a customer might like might be an imbalanced problem. You may recommend 20 dresses to somebody before they actually say that they liked that one.
- 9:02
And that might be a good outcome relative to a baseline or a benchmark. The machine learning resource for the data scientists might unfortunately at an organization like that, use something like accuracy or another metric that is not really very good at giving you useful information in the context of an imbalanced problem. At the third stage of maturity, where organizations are focused on operational metrics, we often see both a maturation of the technical knowledge, as well as a connection of the technical with the financial.
- 9:36
So an organization might start to move towards looking at a true positive, right, in that context where someone that we recommended a dress to actually purchased the dress, they might look at that based on demographics or other information about their customers. And they might connect that to whether or not the customer purchases, subsequent dresses. And so those are important things that organizations often do, right, as they take increasing technical maturity and they connect it with other forms of data in the organization, typically on an ad hoc or case-by-case basis.
- 10:15
Again, moving finally into the phase where organizations are using these tools and using these programs to affect strategic outcomes for the organization, a journey not everyone takes, we start to see much more holistic measurement and metrics programs, right? So at this point, the metrics have become a program in and of themselves. There may be complete dashboards. Some of those may make it into investor reporting.
- 10:42
And in this case, we talked about customers associating our brand with the right product at the right time, right? You show me dresses I didn't know I needed, and I have them the next day or the same day, right? That's what we want customers to associate us with in this context. And so simply put, we can measure the actual brand outcome from a strategic perspective here as part of our program of measuring the impact of our machine learning strategies.
- 11:13
It's a big journey, right? From nothing into a simple identifier, right? Knowing that we want to have AI to actually connecting all the way through our supply chain and, and cost accounting and an operational details into an improvement in our brand or a better association of our brand with our desired signaling to the market. It's a dream. So not every organization is going to be able to do this. Like any maturation, it involves often fits and stops and starts and stops and starts.
- 11:54
And sometimes these happen organically, internally. In other times, there are points in time where others from from peers or for counterparties or consultants, or, or bottom of training are needed to really end up driving the change. So I have two other examples that I'll be sharing as images this week. And I hope you enjoyed this week's Wednesday in the Woods. Bye bye now. (clothes rustling) (birds chirping)