The Data Science Workflow. Cnaffm. Mmm. Ppp.

from the human-edited captions published with the video. timestamps jump to that point in the video.

1. 0:43

   Hello and welcome to this week's Halloween Wednesday in the Woods. This week, we're going to be bringing back an old friend, cinnamon, because we'll be talking about a mnemonic that we use to help us remember the data science workflow. It goes a little bit like this: Cinnamon (C-N-A-F-F-M) is M-M-M, P-P-P. Very simple. You'll see in a moment. So, cinnamon. Right. We have to do this because this shows you how old I am.

2. 1:14

   [old man does the cinnamon challenge all wrong - #cringe]

3. 1:20

   C-N-A-F-F-M. M-M-M. P-P-P. What does it have to do with the data science workflow? Well, you see Cinna- is short for C-N-A, where the C stands for collect, the N stands for normalize, and the A stands for augment. In this context, we're talking about collecting the information not just from something like a data subject but also like an internal system. There are a number of different topics we'll come back to there later.

4. 1:51

   The second topic is normalization. So, when you hear things like data warehouses, data lakes, data platforms, this is the stage at which representations of information are, hopefully, normalized into similar shapes or standards or formats. And then the third piece of C-N-A, the A, has to do with augmenting or generating synthetic data. So, in many contexts we might have less data than we want or there may be issues with the training data that we have.

5. 2:20

   And so, to the extent that we're aware of these issues or we know we need a larger sample, we augment our dataset. Next, we have F-F-M: F, F, M. So the F and the F both remind us of feature. The first F stands for feature engineering, where we take the information as it's stored in our original data and we try to engineer or combine or create derivative aspects of that information. It could be something as simple as adding two numbers together, dividing one number by another, or could be something much more complex, having to do with external statistical distributions or other kinds of transforms.

6. 3:06

   The second F in F-F-M stands for feature selection.. In some cases this may be captured in the models that we choose down the line, but, in many contexts, we actively as humans or through some choice of feature selection method, reduce the number of variables that might be passed down the line into our model. Sometimes, feature selection has to do with, let's say, removal of bias or statistical properties that we know we don't want.

7. 3:33

   In other cases, it's performed simply by something like an algorithm that takes mutual information between variables and some other target of interest and helps us weed out things that don't appear to contribute. So that's the second F. The third letter, M, stands for model selection. At this point, we choose one or more types of models that we'd like to use. This could be, in the old days, something like a decision tree or Support Vector Machine or Random Forest, and today, this could have to do with selecting an architecture for a neural network as more and more modeling activities have to do with deep learning.

8. 4:14

   That's through CNAFFM: C-N-A-F-F-M. The first half of the workflow is important, but what's to come is where the actual usage occurs. So, we've selected a model. We have some features. But we need to train the model. So the next M-M-M is focused entirely on models. First, we train a model which means, depending on the model type, we have to pick something like a loss or penalty or some other strategy or means of taking some training data and our model that we've selected, so some kind of parameter set, and fitting the training data to the model. This is where we actually build or fit the model.

9. 5:01

   The next M stands for testing the model. So, it's important obviously to test the model Critically, the idea here to remember is that our training data should not be the same as our test data for a variety of reasons. And, our test data may be different than the cross-validation strategy that we have, that might also have some other kind of data or distribution of data or metrics that we used to test. Two M's. M-M. The third M is for model hyperparameter optimization.

10. 5:37

    So, in this context we're talking about models where there's a clear distinction between the model that we select in a singular sense and the space of potential models that, let's say, share a class. Simple examples would be the number of trees in a random forest or something like a learning rate parameter in a model. Obviously many, many model specific hyper parameters and hyperparameter optimization is very much in the weeds.

11. 6:09

    But again, it's the important step that is often forgotten or missed in data science workflow. So, at this point we've gone through C-N-A, collect, normalize, augment, and F-F-M, where the FFM stands for feature engineering. feature selection, and model selection. And, we just covered the M-M-M in CNAFFM MMM. In the M-M-M, the Ms stood for a reminder model training of model testing and cross validation, and model hyperparameter optimization.

12. 6:44

    At this point, we should theoretically have a model and typically this is where research and development or data science teams cut a line and hand over the model to P-P-P: production. So, there's three Ps. And you could divide the production life cycle in a number of ways, but we've chosen three Ps in our workflow. And the first P stands for the most obvious, which is implementation. In many, many situations - I would say most, historically - the research and development model was never intended for production use.

13. 7:16

    Whether for performance reasons, access to data, debugging output, programming languages, a myriad of reasons result in the production usage of a model requiring a porting or re-implementation of the model from research and development. And the second P reminds us that machine learning models, like any software in any controlled organization, ought to have testing and monitoring. So, when you're done implementing your model, you're done with the first P, you need to test the model to dock it, to confirm that the results you expect from research and development are also the model results in production.

14. 7:59

    So, when your model is running, it's like any other software process in a controlled organization. It's critical that you monitor the up time, the response time obviously, but also the quality of the results. Are our probability estimates changing, is a K-S test indicating some kind of critical phase change in in our actual production model? And the third P stands for periodic or online retraining. So, in many contexts, model is not just a trained once and implemented production model, but it's a model where there is continuous or continual act resulting in online or periodic retraining.

15. 8:45

    And this could be something as simple as users clicking a thumbs up or thumbs down an application, or it could be something much more complex like going through the entire C-N-A-F-F-M process on a daily basis, including the acquisition of new data sources or materially different model hyperparameter optimization regimes. So, the three Ps at the end, often a different team highlighting the scope and responsibility change.

16. 9:13

    But nonetheless, we don't build these models for fun. We don't build them for pure research and development perspectives. We build them because we hope or intend them to go into the P-P-P: Production stage of our C-N-A-F-F-M M-M-M P-P-P mnemonic. So, I hope you all have a happy Halloween and get out there and have a little fun. And, please, don't do the cinnamon challenge. See you later.