Wednesday in the Woods · episode 025
Hustle and Data Flow: Pump it up
4:30youtube · Licensio, LLC
description
It’s a common misconception that networks are just a series of pipes. In reality, it’s more of a tubular racetrack full of horse-driven carts, like a Hippodrome in the subway 🐴🌊
Just kidding. But I am here this week on Wednesday in the Woods to talk about how success - in farming, war, and tech - often comes down to moving things from point A to point B.
In the world of war, we call this logistics. In the world of farming, well…it’s basically farming. And in technology, it’s just transfer.
No matter what you call it, it starts with a data strategy and some planning or prediction. If you don’t know how much of what to move when, then it’s going to be chaos. How many MREs per unit-day? When do I plant the barley - and at what rate or spacing of seeds per acre?
But once organizations have moved past this phase of maturity - once they know what data they want to collect and what they want to do with it - logistics becomes king. In many organizations, an overwhelming majority of effort and investment goes into data logistics at this stage.
Billions of dollars are spent every year on bigger pipes and more horses - in the form of network capacity, stream processing throughput, or related infrastructure. Dozens of dedicated roles have emerged in the last two decades within companies that scaled data science and machine learning capabilities. We’ve come a long way since people first tried retraining models nightly on equity tick data, but many organizations still find themselves struggling to productionize machine learning models due to data volume or transfer rates.
These logistical bottlenecks don’t just occur in value creation contexts, however; they have also been a key component of risk management programs such as BC/DR plans. For example, if you know that you need to restore 10TB in 2 hours, but you can’t restore more than 1TB an hour…well, you might need to change the BC/DR plan or look at investing in better infrastructure.
The good news is that this mostly comes down to adding and dividing. Sum up the amount of data you have, make some assumptions about growth or compression ratios, benchmark a transfer, and then do some division. Give yourself a healthy buffer. And re-check these assumptions regularly.
As a leader, it’s really one of the most important things you can do: make sure your logistical capabilities exceed your current needs. Test assumptions to make sure that they’re really true. And don't skimp - these issues silently destroy time and value fast.
PS: If you like this week’s hat, I got it from @shoesandstocks Instagram store; 100% of proceeds go to UNICEF’s support of displaced Ukrainian children.
transcript
~689 words · showhide
from the human-edited captions published with the video. timestamps jump to that point in the video.
- 0:25
There's a saying you may have heard in both farming and in war that 99% of success is just moving things from point A to point B. I've heard it paraphrased as many ways as there are seasons and I think it's just as true of farming and war as it is of technology. So, in most of our applications, we require the right data to be at the right place at the right time. And to some degree, this is about putting together a data strategy and identifying the data in the first place.
- 1:05
But for many organizations, especially once they've passed this point, once they're operating, as they start to climb the maturity ladder, the next most common issue that they face is that even though they know the data they need and they know what they want to do with it, they simply don't have enough horses to move the right data to the right place at the right time. And so, if you look at the way that most these organizations spend their time, the capabilities they try to develop, the things they invest in from an infrastructure or personnel perspective, a lot of it is this simple - trying to move something like this big pool - Maybe it's a lake,m maybe it's a swamp, I don't know - but it's some data, from one place to another.
- 1:53
For example, I've got some trees over there. They don't have enough water right now. This - too much water in one place. So, it'd be nice if I could move this water over to where it's needed. Too much here, too little over there. To do that, I need to just think about a pipe, just like a data pipeline. And sometimes we need those pipes for things like data science. Other times we might need those pipes, for things like backup or recovery.
- 2:23
So in all these circumstances, the easiest thing to do is, is count, add up things and divide. So, for example, if you want to build a model every night based on customer records like sales records or transactions, then you need to make sure that all of the data that came in today is able to be moved to the place where you're going to build the model by the end of the day. And to do that, you can count up how much data comes in, compressed or uncompressed, transactions, average sizes, whatever, and see if you've got a big enough pipe to move that data to the right place by the time you want to build the model.
- 3:03
If you don't start there, if you don't check that the answer is "yes, I have enough bandwidth," then doesn't matter what you do, you're in for trouble. And until you address that issue, you're going to have problems. Similarly, if you've got an RTO and an RPO in your your BC/DR plan for a system and you know that you have more data than the pipe can move in the amount of time that you've written down in that plan, then you've got a problem and you're going to need to figure out what to do with it.
- 3:34
And at the end of the day, all of this stuff typically comes down to looking at things like diameters, rates, and volumes. It's not that hard, even when you don't really have hard numbers and you just use assumptions, it can go a long way to making sure that when push comes to shove, when you really need it, or when you're done building that research and development model and you're ready to put it in production, things actually go well. Hopefully, it's for creating value and not just for responding to risk.
- 4:04
But either way, the point stands: Make sure that you have the pipes and the horses needed to move things from point A to point B. For me right now, that means more water and more seed and more diesel. For you, I hope it's something a little bit more fun. Either way, See you next week!