Wednesday in the Woods · episode 012
Data Lake or Data Swamp?
6:28youtube · Licensio, LLC
description
Does your #datalake smell more like an interminable #data #swamp?
Sorry. I know that’s offensive; “swamp” is typically used as a colloquial term with negative connotations. Real wetlands are treasures in all their varied forms, serving critical roles across multiple natural systems. Your metaphorical #dataswamp, on the other hand...
Well, technically speaking, what you have is probably closer to a data bog or mire. It feels like your data just sits there, letting off a strange odor. They say that a rolling update gathers no moss, but your systems are covered in the stuff. But other than the moss, not much grows there due to the fact that all of the nutrients that accumulate are not bioavailable. Maybe your team is too salty or acidic, but other than data preservation, it’s not clear what they’re up to.
Do let me know if you can recreate the iodine notes of a good Islay though. I’m always interested in liquid investments.
So if you want a more fertile data wetland, what should you strive for?
My opinion is that you don’t want a data lake. Lakes are mostly good for storing water, but relative to wetlands, they aren’t very productive relative to the space they take up. And clearly, unless you’re in the record retention business, you don’t want a data bog.
You want something like a data marsh. Marshes are a type of wetland that normally occurs where rivers and streams meet lakes and oceans. Unlike bogs, all of the nutrients in a marsh are available to support growth. And grow they do, the fauna and flora in marshes.
Now, some people might say that you do want a data swamp. Swamps, unlike marshes, tend to support the big boys - large perennials like deciduous trees. But in my experience, these “big” perennials - large, rigid systems like old-school OLAP/BI implementations - can suck up all the light, leaving little room for new or varied growth.
Marshes, on the other hand, have higher biodiversity and support a mix of annuals and more adaptable, easily-harvested perennials. Through rice cropping, marshes provide the primary staple food for more than half of the world’s population. It took our species a little while to figure out how to cultivate rice, but boy oh boy, did we make some progress once we understood how to model the tides and seasons.
In reality, the ideal situation is probably both swamp and marsh. What can you do to help set the stage to differentiate between these ecosystems? Well, it often comes down to soil composition, as Liam said on one of our last episodes! More on the nature and properties of soils later…
How far can we keep this analogy going? Well, once we’re done with soils, we’ll be back to talk about how you might assert your riparian and littoral rights with your data neighbors. The analogy factory runs 24/7, folks.
transcript
~709 words · showhide
from the human-edited captions published with the video. timestamps jump to that point in the video.
- 0:37
Good morning and welcome to this week's Wednesday in the Wetlands. Today, I want to talk to you about data lakes or, as I think when I get up close to most organizations' data lakes: data swamps. Except that's offensive. A swamp is a colloquial term for most people with negative connotations, and, in reality, wetlands can be incredibly productive and critical parts of our ecosystems, both natural and corporate. So, what are what most organizations have?
- 1:11
I think most organizations don't have data lakes or swamps. They have boggs or mires, which are a type of wetland where material tends to build up without producing any real growth. So in a bog or a mire, for example, a peat bog, we have a lot of material that might build up and fail to decompose on an appreciable timeline. Either due to pH or due to water conditions or the type of soil, we have a situation in which stuff just collects, but nothing really grows on it.
- 1:47
It's not broken down into valuable resources at a rate that's appreciable. There aren't organisms, plants or animals, that survive on it. And there isn't really a lot of appreciable value in an immediate sense from a bog. They have certain use cases, for example, a data preservation business. But for the most part, I don't think organizations who set out to create lakes are really happy with a bog or a mire. Instead, what I think we want to work towards is not a lake, because lakes for the most part are good at storing water, but per square foot or hectare of surface area. They don't necessarily create as much value. They don't unlock as many nutrients.
- 2:30
They don't support as many organisms, as, for example, marshes do. Marshes and swamps are types of wetlands where all of the nutrients that you might find in a bog, all of the data that you collect, is made available. Either due to the type of soil or the pH conditions or the salinity or simply the kinds of organisms, plants and animals, that are available in the ecosystem, given the data that you have. So, a marsh and a swamp vary in the sense that swamps tend to support more trees, more of these large woody perennials.
- 3:09
And I like to think of our old-fashioned OLAP, our online analytic database systems, and our more modern but ossified BI frameworks as large woody perennials. They're good. They serve a purpose. Organizations, especially the larger they are, need to have large support structures like trees in these wetlands, but sometimes too many trees can choke out innovation. So what we might want is a combination of swamps with large mature trees, as well as marshes, where we have more annual and perennial grasses, things that are constantly recycling and adapting and changing over time.
- 3:56
And this gives us a nice balance between tools that are production-ready tools for organizations to work from, like in a traditional data warehouse that's been productionized into actionable insights for the C-suite, but also with a portion of the ecosystem that's able to adapt, that's able to support experimentation, that has a higher level of biodiversity, for example. And it doesn't take so much effort in time to create appreciable structures, right.
- 4:26
Cottonwoods like these and some of these red maples can take, especially given slow growing conditions around wetlands, 50-100 years to grow to an appreciable size, whereas some of these sedges, perennial and annual grasses, we can have eight-foot grasses that are ready to harvest in a year. So, the analogy might be a little bit stretched but I don't think you've seen the end of it. I hope that you guys are having a better time than I am here today cleaning up our wetlands, but again, very important for your organization to nurture this kind of environment where all of the important nutrients are collected in a sink, and where organisms are allowed to thrive and adapt and evolve and really convert these nutrients that might otherwise seem like waste into something of value for the organization.
- 5:21
Anyway, I hope you enjoyed today's Wednesday in the Wetlands, and we'll see you again next week.