Wednesday in the Woods · episode 017
Real Talk about Reproducibility
5:44youtube · Licensio, LLC
description
Yes, I’m hiding underneath the cloak next to a log pile and freshly-bent EMT. No, I’m not playing peek-a-boo. And technically, it’s called a focusing cloth, but “cloak” definitely vibes with the enigmatic angle, so I won’t stop you.
Anyway, what am I talking about this time? There’s a hint in the camera and lens.
Unlike most of my cameras, I assembled the one I’m using today. Other than the fabric bellows, aluminum rail, and ground glass, all of the parts were 3D-printed from STL files. And even the rail has its own CAD diagram, which you could use to generate M and G-code to CNC a replacement.
And the lens? Well, Andrew Wollensak first started assembling shutters in a barn in Rochester, NY around the turn of the 20th century. By the time 1941 rolled around, the Wollensak company had produced more lenses for aerial reconnaissance than any other firm in the world. If you’ve seen aerial imagery from the Dust Bowl or World War II, you’ve probably been looking at silver halide activated by photons that passed through a Wollensak lens and shutter.
So what’s the point? Replication, repeatability, reproducibility, whichever word you pick, is critical to achieving the outcomes we want. Our lives have been transformed in the physical world over the last two centuries by engineered, mass-produced products - the original replication. As our society and economy transitions to an increasingly digital reality, why should this new phase be any different?
The importance of reproducibility is even more important when it comes to activities in “public” academia and in regulatory contexts. If measurements are not repeatable, calculations are not repeatable. If calculations are not repeatable, then processes and decisions are not repeatable. If processes and decisions are not repeatable, then how can we learn from them, manage them, or attest to them? This is true regardless of whether a human or GPU is executing the neural network.
Sometimes, reproducibility is as easy as documenting a process or setting the seed in a pseudo-random number generator. Other times, reproducibility is plagued by intrinsic sources of randomness in our environment or our measurement thereof. But even when controlled measurements or calculations are stochastic, documentation and measurement are still better than none.
So, next time you start on a project - whether it’s collecting data for a machine learning model or developing a manual underwriting process - I hope you take a little bit of extra time to make it reproducible. In the long run, that little bit of extra time will probably more than pay you back in dividends.
PS: yes, I developed the sheet. And yes, you’ll get to see it tomorrow. Shot 1/50th @ f/11 on Ilford FP4 at box ISO, developed in 450g water, 63g Ilford DD-X, 50g Ilfosol 3 @ 21C for 5 minutes, fixed in TF-4 for 5 minutes.
transcript
~639 words · showhide
from the human-edited captions published with the video. timestamps jump to that point in the video.
- 0:40
Hi there. How are you? It's Wednesday in the Woods. And today, we're going to be talking about replication and repeatability. You see, one of the most important things that we can do in any kind of scientific endeavor, whether it's in academia or industry, is to ensure that our measurements and our calculations are repeatable. And one of my favorite historical examples of our advances in this area is the camera. So, this is a camera that's actually 3D-printed. All the parts other than the aluminum are 3D-printed, and even the aluminum has a CAD diagram, a schematic or a reproducible set of instructions with an M-code file, that would allow you to machine the aluminum and to 3D-print the parts, so that we can take a single design and produce the object over and over again.
- 1:46
Similarly, this lens is a World War Two aerial surveillance lens and the lens allows us to capture a footprint of the earth beneath us in some kind of wavelength or wavelengths. And this allows us to take repeatable measurements instead of hand sketching something. We can fly an aircraft up at a known altitude and take a picture under specific conditions, for example, aperture and exposure time, under constant lighting conditions.
- 2:27
This allows us to take repeated or replicable measurements of some kind of environment or some piece of land. Let's talk through that last example. One of the important things is not just to create devices that can support measurement that can be replicated, but also to document, for example, the settings if you will, the parameters of our model, the order in which we took pictures or any of these things that could be translated very easily into a data science workflow. So, for example, if we collect data in order to build a model, well, collecting that data is just like taking pictures with this lens.
- 3:09
So, it's very important that when we collect the information, we record our position or where we were, where we got the data including, for example, the complete provenance that our organization needs or any licensing information. It's also important that we know the settings. So, for example, if you think of the aperture and exposure time here as the settings or the pre-processing on our data when we collect it or initially do some kind of transformation to it, then it's critical that we record that information as well.
- 3:43
Let's say you're scraping a website - a very simple example - or capturing information from an API - if you don't know what parameters you pass to the API, you don't know what day you retrieved the data from the website, you don't know what terms of use the website has, or you remove some outliers from the information but don't know how you removed the outliers, then it's going to be very difficult for you to replicate whatever comes downstream from that information.
- 4:10
Whether it's just some kind of reports or whether it's a machine learning model, anything like that. So, replication is key not just to science broadly, but also to any of our applications of computation or measurement in industry. So it's such an important point, and when we do things the right way, we unlock, for example, the ability to build incredible models, both in the physical world and in the digital world. So, I hope when you start out on your next project, you take this message to heart and you, from the very beginning, ensure that what you do is replicable.
- 4:52
There's a lot of ML Ops - machine learning ops - frameworks these days. Maybe we'll talk about those sometime later. But anyway, I hope you're having a nice day, staying warm, and we'll see you next week on Wednesday in the Woods.