Wednesday in the Woods · episode 007
Why do modeling projects fail? From R&D to IRL risk
11:40youtube · Licensio, LLC
description
Plenty of projects fail. For example, if someone told you to clean a whole forest using nothing but a child’s rake, you might think twice before accepting. Of course, if you were really smart, you’d accept the challenge, build a lean-to, hide it with a pile of leaves, and then take a nap in the woods. Not that this was my plan...
Anyway, projects fail. Modeling projects are no different. The whole world seems to have rediscovered this fact in the wake of Zillow’s short-lived home-buying strategy But instead of carrying on the punditry, I thought I’d focus on common risks that all #modeling and #datascience projects encounter.
We like to break these risks down into two categories: R&D and "real world."
We’ve talked about a number of R&D risks before: ‣ Is your modeling project a real project? Is there a goal?
‣ Is the project feasible? - Technically? - Budgetarily? - Timing? - Data collection? - Legal? - Political and social?
‣ Have you defined an org structure/RACI to support the project?
‣ Do you have the right people to execute? Have you allocated enough of their effort at the right time? - Subject matter experts - Data collection/annotation resources - ML/stats experience - Compliance - Engineering/ops
Even when all of these risks are addressed, things still happen after launch. These “IRL” risks are different from R&D but no less important.
For example, many issues trace back to your data. ‣ Has something changed since the data was collected? ‣ Was the sample not large enough to capture normal variation, trends, seasonality, etc.? ‣ Have rare events or truly novel events occurred?
Next, many projects face pitfalls when asked for continuing justification. ‣ What expectations were set when the project began? ‣ Are the benefits or costs of your project variable/uncertain? ‣ Are the benefits from your project financial? ‣ Are the benefits from your project direct or indirect?
It’s critical for the responsible and accountable stakeholders to manage these issues from the outset of a project.
Lastly, did you consider the impact of your own model on the model? Yes, you read that right.
When your model becomes a part of the “real world,” the training data may be immediately invalidated. Because the model is now endogenous in the world it was modeling, unforeseen dynamics can emerge...
For example, your model might... ‣ Change your own behavior. ‣ Cause others to front-run your model. ‣ Cause others to copy your model. ‣ Cause others to exit the market. ‣ Cause material changes to the liquidity or price dynamics of your market.
At the end of the day, the important thing is to learn, both from what has and has not succeeded. Zillow has probably learned a lesson, and I hope that others can think critically to learn from their mistakes to.
(PS: the answer isn't to stop modeling.)
transcript
~1,367 words · showhide
from the human-edited captions published with the video. timestamps jump to that point in the video.
- 1:04
Good morning and welcome to today's episode of Wednesday in the Woods. This week, I want to talk about why modeling projects fail and they fail in two ways, in my opinion. The first way is that they fail in research and development and this begins with the idea that a modeling project should be a project. So, in some cases we have what's really more of an exploratory exercise, right? There's no defined purpose, there's no defined goal, there's no completion or success.
- 1:38
In this case, we haven't really started a modeling project. Not to say that there's not value from something like this, but to say that it's not a project and if we expect to measure it in a traditional sense, then it will be difficult to see the ROI. The second way that a project fails is if it's not actually feasible. So, in many circumstances, yes, there may be a purpose to a project and it may be defined, it may be a real goal, but it may not be feasible for one or more reasons.
- 2:08
It may not be technically feasible. So, it may just be beyond the possibility of current models to capture information or predict. There may be an intrinsic degree of randomness or entropy that prevents us from predicting something. It may be infeasible because of the economics. So, you might not have the budget for it or you may just not have a positive ROI after the cost of investment. It may be infeasible because of the timeline, so, you may not have enough time to complete the project, even if it were otherwise feasible and economical.
- 2:48
You may, even if all of those things are true, simply not have the legal right to do so. Or even if you do, it may be politically difficult or socially infeasible from a public relations perspective. So, in many ways projects can have a defined goal can have some partial feasibility, but one of these alone can be enough to shut down a project. Many, many, many projects fail at this stage. And so it's important when you start out on a modeling project to ensure that you've done a thorough review of all of these different types of feasibility before you invest further.
- 3:32
So, let's assume that we have a project with a defined goal with feasibility analysis performed. Now, this is a little bit endogenous, but we need to make sure that we have the right stakeholders assigned here. So people like to to frame org structures and stakeholders in lots of different ways. Let's just assume that you've determined your RACI or your ARCI: the Responsible, Accountable, Consulted, and Informed parties in your organization, or potentially outside your organization.
- 4:01
Failure to properly define stakeholders or have an org structure in place is another common source of failure for modeling projects. A project may have gotten this far and they have stakeholders who have taken part in a process of assessing feasibility and those stakeholders may have assessed the feasibility in the context of a defined goal. Next, we need to resource the project and even organizations that may have the budget, financially have the budget, don't necessarily allocate the right people at the right time with the right skills or for long enough.
- 4:42
And so, organizations can have it all in the sense that they have the budget, they have the the problem, it's clear to them. They define the problem, they assign the stakeholders, but if they don't keep the right resources with the right skills on the project for the right amount of time, then they will frequently see failure as a result. That's research and development. That's the process of defining the project to researching and then developing the project as the assigned resources follow our CNAFFM-MMM portion of the mnemonic from two weeks ago.
- 5:19
But, as you'll remember, the last part of our CNAFFM mnemonic is P-P-P for production, where our risks are not research and development risks, they're real world risks. And real world risk take a number of different forms. So, let's assume that the team assigned to develop the model, did a good job of selecting a model, of testing the model with the data they were given by stakeholders, assessing hyper parameters, they properly docked the model in production, the online training or retraining framework of the model is implemented appropriately.
- 5:59
The test data may simply not have been the right test data for a variety of reasons. The stakeholders or the SMEs engaged may have not properly obtained the data in that it might not be representative of the real world that the company operates in. The data might simply not have had enough variation in the sample, or it may not have captured rare events or exogenous factors that either do occur infrequently or that obviously might not have occurred at all in the historical sample.
- 6:35
Common reasons for failure in the real world when a project goes live. The other reasons that projects often fail involve failure to measure. So, have we measured the project in the sense that do we have a way of measuring its benefits and its costs? Frequently, we may be able to measure the direct costs of operating a project, but it may be difficult to measure the benefits. The benefits may sometimes be direct and financial, in which case they're measurable through some effort, or they may be indirect benefits, or they may simply not be financial benefits, In which case, stakeholders, depending on their incentives and alignment, may simply pull the plug on a modeling project in the real world in production because its benefits can't be clearly shown to outweigh its costs.
- 7:33
It happens and it's a shame, and so it's important, again, going back to the beginning of the process, that the stakeholders responsible for the modeling project, identify that risk up front and work with any other parts of the organization or external stakeholders to ensure that that's an expectation that's set from the outset. The last reason that modeling projects fail is arguably the most interesting and one that I think is relevant potentially for the Zillow in the last two weeks.
- 8:02
Sometimes you do everything right, but you yourself, by participating in the market or by taking your model into production, in changing your own behavior, change the very world that you modeled and built your model on. So, you may change your own behavior, your own organization's behavior. By implementing a model, you may change the way people work in a way that affects the model's ability to appropriately predict.
- 8:32
You may change others behavior - so others may observe your behavior. For example, in financial markets you might get front run. In other markets, your behavior may cause others to copycat or others may simply retract from the environment that you're operating in if it becomes clear that you have an unfair advantage. Those two examples I think are the most difficult to address and frequently require not just a traditional machine learning or statistical modeling approach, but also some type of qualitative or quantitative simulation of the actual implementation of your model into a real world environment. Because a failure to understand how your production usage of the model may impact others can invalidate everything else that you do along the way.
- 9:29
Critical to think through that last fact. And so, as we get ready for the end of the fall here and we continue to clean up, I think the last thing to remember is that no matter why a project fails or even if the project succeeds, it's important to take what you've done in the project - again, failure or success - and to learn from it to, if you will, compost your nutrients of that project into the future learnings of your organization, so that next time your projects are more likely to create value and more likely to succeed.
- 10:10
And with that, I hope you all enjoy the fall here and I need to get back to raking up the forest. See you next week.