Jake Warner + Bret Fisher

Jake Warner: [00:00:13] Hello and welcome to the 12th episode of Cycles podcast. Today we have Bret Fisher, who is a docker captain and also a Udemy instructor. And Bret, it'd be great for you to introduce yourself and share a little bit about what your day to day looks like, what you're working on with our audience.

Bret Fisher: [00:00:33] Hi, Jake. Glad for being here. This is our second time hanging out. So I'm a docker Captain, you mentioned that. That just means I'm a kind of a community leader. Volunteer to talk about all things containers. And I'm also a consultant and a trainer. So most of my week is usually working on my courses, which I think we're working on our fifth now, course on Kubernetes, Docker, all the container topics on Udemy. And then I do a weekly live show on YouTube called DevOps and Docker, and then I do a podcast out of that and I have a nice little team that we've built. I'm also a multiple company founder over the last 20 years, so this is like our sixth attempt at something that works and it actually works this time. So we have a nice little company that we focus on the training aspect and videos. I think we've made over 600 videos or something at this point. So yeah, that's my week.

Jake Warner: [00:01:23] Yeah. And so as you already alluded to, one of the we and maybe a number of people who are watching this have seen but we were I was on your show back in January of this year and we had we had spent quite a bit of time diving into a demo Cycle at the time. But my hope with with bringing you on to our show at the moment is being able to dive into not just the technologies around containers is always an interesting thing to dive into, but more of potentially the business case and the value behind this is so, so not being in the weeds of the tech, but more of the how is this tech changed how businesses are operating? What have we done? What have we done wrong in the industry? What have we done right in the industry? And more of a high level just kind of fundamental perspective and looking at containers as a whole as opposed to the individual technologies that make it up. Right. And so I know you have your show, you have a lot of people that are coming onto your show to show off really cool technologies and things. But besides your show, I know that you also do a lot of consulting. Can you dive into that briefly?

Bret Fisher: [00:02:39] Yeah, well, this all started, you know, shortly after Docker was invented and you know where we were. Basically nine years ago this month, at least in March, that's when Docker was sort of announced to the world. And it's amazing. I think back then I was thinking this is at least a ten year evolution that we're about to enter into. So my story kind of goes around 2014, 2015. I was in a tech startup that I co- created and we were using Docker to solve a lot of problems even back then in its infancy. And it was really all about speed. So when you hear me talking about containers a lot, I'm really focused about the speed of business objectives. And that's really if you look back over the 30 or 40 years, I have an increasing amount of gray in my beard. So I've done these technology evolution. Pivots. I guess you could say many times before in the nineties we were going from mainframe to PC from, you know, Then we went about a decade later we went from PC to virtualization or we went from a hardware based single OS per piece of hardware to a mini OS per piece of hardware virtualization world and was was a part of a big government agency that sort of was leading I was leading the charge on virtualization. I was very passionate about that change. But everyone thought it was heresy, right? Everyone thought it was crazy to put all these OS on one machine. Won't it be really slow if you fast forward a few years later? We're now outsourcing that hardware responsibility to the cloud. The cloud is invented, so that's a huge shift and we're kind of in that same shift now with moving our workloads into a it's not a layer of virtualization, but it's an encapsulation or a a layer of isolation.

Bret Fisher: [00:04:20] A lot of times that allows us, again, more business speed, the velocity of deploying code, updating code, reverting to old versions of code because we have to roll back like you name it, containers have basically improved or sped up those processes. And I think that's what got me motivated in 14 and 15. When I was first starting to use it. I pivoted my entire career at that point, which I've never done for a single technology, and I decided this is such a huge change. I'm going to do nothing but this. Now. I'm only going to consult on container implementations and container optimization solutions and all that. I'm only going to talk about it and I'm and I'm now going to make start making courses about it. Eventually that happened like a year later. So and that's, you know, here we are five, six years, seven years later. And that's exactly what I'm still doing, is the container revolution, I think is something that's underplayed a little bit. We're all enamored right by that latest tool that somebody has announced. But really at the core of all this is the idea that we're taking our workloads and isolating them not in a virtualization layer, but in simply an OS construct that Linux and Windows happen to support. So I think that idea still holds a lot of weight today, and I'm excited for the next decade of how we're going to even make it better and faster.

Jake Warner: [00:05:34] So and a lot of what you just said resonates with me, especially, you know, you pivoted your entire career around the containers. And I'm thinking back to right around that same time frame, it was late 2014, early 2015. Cycle was in its early, early stages at that point. But Cycle wasn't a container platform we were building. I'm sure you're familiar with Squarespace, you know, these website builder type frameworks. And so we had we'd spent years well, I'd spent years building a dev shop where we were building websites and things like that, building early SaaS products for other companies. And but as we started, as we kept building these SAS products for companies, we had to always redo so many of these infrastructure components every single time we started the new project when we had that dev shop that it was kind of there was a point where it was so obvious that we need to we need to build something to make this easier. And so we had started building Cycle and I didn't know what containers were at the time. This was again, this was right around that start of that, that being a thing. And so we were we were probably about a year and a half ish into building Cycle, and it was a platform that at the time was built in.

Jake Warner: [00:06:49] Php Right. I mean, we were, we were pushing PHP to do things that we shouldn't be doing, right? Right. We're writing like TCP layers in PHP like it was the stupidest idea, but at the time it seemed smart. But the issue that we kept returning to as we were building that out was the idea that we we kept building out this framework. We had so many of these companies that we were working with that kept saying, Hey, we have another dev team somewhere else that wants to be able to pull in code to this framework and things like that. And the hardest part for us was how do we allow people to run third party code inside of this, this existing framework without them having to know exactly how we structured the language structure, how we our use of PHP and things like that. And so we were sitting in our office at the time it was me. And I think for developers at the time we were sitting there and we came across a tweet on Twitter obviously and it was about DockerCon. And this was, I guess, DockerCon I guess it would've been DockerCon 2014 or it was 2014 or 21 of those two years.

Jake Warner: [00:07:54] And we were sitting there watching it. We had it on the screen in our office, and next thing you know, they start diving in. And it was there was a demo that they did about containers, and it was that day. Those of us were sitting there watching this that we went. I remember I stood up. I went over to what I raced everything that we did and said, we're changing the company today. So as you talk about pivoting your career, that one day of being introduced to containers was a pivot to the entire thing that is Cycle today where we had pivoted from. We're going to be a website platform to we're going to be a container orchestration platform. And so I think getting back to the point here of the you know, what you what you already said of the the revolution of containers, the adoption of containers sometimes is is lost amongst so many of these new technologies and things that we don't realize or maybe we forget or gloss over how much it changed how businesses and developers operate today. Yeah. And so just really the that single year impacted both of our careers so much in that capacity.

Bret Fisher: [00:09:07] But well and I think that's that's the challenge of the of that's the that's when I draw this out. I basically I start with the seventies and eighties and I talk about major evolutions in infrastructure because that's kind of what Docker is to me, right? Like it is a software idea. It doesn't necessarily directly talk to hardware, but it is an abstraction layer that we have all created. And it is it creates technology momentum. And so when I think of these major evolutions of, like I said before, like mainframe the PC, you know, PC to virtualization, virtualization to the cloud, cloud to containers like these, none of those things really have anything directly in common with each other. But the idea is, is that at every point it was always about speed. I mean, we we obsess over computer, CPU gigahertz nowadays. Maybe we're not near as much as we did in the 2000s where it was like all about single processor reaching the limit of thermal dynamics. But all of those things were always about speed. But there's more than just CPU and memory speed, right? It's really at the end of the day, it's about the speed of business. And so these ideas, if they're not implemented properly, what's funny is about every one of these evolutions, if you did it wrong, it actually hurt the company like it would if it didn't have clear business objectives and goals and metrics, which is something I talk about a lot in the DevOps world is metrics.

Bret Fisher: [00:10:25] If you're not measuring the performance of the teams and the in the organization in terms of the from the time the company has an idea to the time that software is in front of customers, like that is the DevOps landscape, right? That's why we had that huge affinity symbol on all the logos of DevOps things, is that that's the continual loop of infinity that we're all dealing with. And inside that somewhere is the idea that we need to take the code, get it tested, get it packaged up into an artifact that's exactly the same on every system we want to run it on and then get it on those systems to run as fast as possible. And containers is actually the fastest way we have come up with to do that. And people have to say, Well, what about serverless? Well, I'm thinking, well, serverless frame, you know, functions as a service. Those are just running in containers too. So you're just running in in someone else's container, not your own. And it's not really anything different other than just you're changing the language a little bit and breaking things down to even smaller than microservices. But often I'll also get the question like, what is the future of containers? And I don't I don't think anyone's come up with anything other than just iterating on what we're doing yet, and we're just making containers faster to build, smaller, more secure, isolating things better, better secure defaults like that seems to be the theme of a lot of the the tools and technologies that we're all using.

Bret Fisher: [00:11:47] And not like we're going to have to undo containers and go to this whole new idea. I don't see that happening anytime soon. I think really the the idea of isolating a workload on an operating system is really kind of if you think about our phones, our Macs and PCs, like everything we're using, is all improving that technology with every update. There's always something about isolation, security updates that don't affect the entire system. But just one little part. And that's really where we came from. And not to go down too far, that rabbit hole. But if you look back, back when we started the PC revolution, the idea was actually that the hardware got smaller and cheaper and that the operating system was unit like was all encompassing of that hardware. And so the idea of virtualization virtualizing in that hardware or rather isolating things inside the OS, that was actually a mainframe concept back in the UNIX days. But once we got to PCs, we kind of undid all that. We said, no, no, no, the. Hardware is where we isolate now, like it's super cheap. It's not $1,000,000 like a mainframe is. So we can buy these super cheap pieces of hardware and then we'll just isolate it, the layers of hardware.

Bret Fisher: [00:12:55] And then we got smaller and smaller computers, cheaper and cheaper servers that were going faster all the time. But then we realized the OS as a boundary and the hardware is a boundary is actually a really poor way to do things from a speed standpoint. So we if you remember 20 years ago we were all struggling with, hey, in order to do this new idea, we had this been this idea, we got to do something. We need servers for that. Well, now you've got an eight week lead time, if you're lucky, because you got to buy the hardware, wait for it to show up, wait for someone to implement that, get the operating system, the backups, the monitoring, all the things planned out, and then you can finally put your app on it. Right. So it was that was fast then because mainframes took a year to buy. So, you know, PCs were really fast to acquire even though it was eight weeks. But we always are pushing the envelope, right? We want to implement ideas at the speed of human minds. Basically, if I if I think it I want it to happen. So the PC grew into this idea of we need to break that PC down and put more things on it and isolate them better. This is how we came up with virtualization. We virtualize the OS and then eventually you and I were in that world of, okay, I'm now one person and I'm managing 1000 OSs and I can't manage all the different operating systems.

Bret Fisher: [00:14:08] They're patching their updates. The replacement of those OS's OS is when they break or they, you know, they go out of warranty or whatever. And that became the huge nightmare that we were actually, you know, we implemented virtualization as a solution. Then we created our own new problem and then containers came where the cloud came really first. But it was there to say, Hey, we've got your problem, we've got your back. We already bought all the hardware. We're going to make it faster for you because that's ready to go. You can do it in minutes. And then we realized we're still underutilizing the hardware, we're still managing thousands of operating systems. How can we avoid all that? And I think that's really the the world of the container, which you were clearly already working on, a different idea of what that might look like, like let's isolate work, isolate workloads in a consistent manner. How do you do that in PHP was your original idea, but now you're just saying, how do we do that in containers? So that's really all the same stuff, right? We all kind of arrived around that same time frame, 2010 to 2015. That was kind of the idea of let's isolate inside the OS rather than the OS itself.

Jake Warner: [00:15:09] Yeah. And you know, like you're talking about isolation. Like obviously the speed that Containers has brought developers has been, has been hugely fundamental, but the consistency is well. And the problems that you solve there with the isolation from that component is also just so wildly valuable. And I think back to again around 2010, where so many people are encountering these problems back with shared hosting, where someone would throw a C panel on a server and next thing you know you'd have how many dozen unrelated accounts sitting on that same server. And then you'd have a customer reach out and say, Well, right now we're on PHP version 4.2. We need to go to a version 4.3, and you're like, Well, that means I have to update it for the entire server. And I know there were some workarounds where you could start kind of segment that, but it was one of those things where, where it was when you were when you were isolating everything from a hardware perspective, as you're mentioning, it's so hard to be able to run slightly different variations of software on the same machine, whether maybe one application needs a slightly older version of Python or whatever the underlying language is. It was so hard to do that in a responsible and easy to maintain way. And right around the time of the in Cycles case that we decided to pivot the platform fully into containers. One of the I remember it was a we so we had a customer that we were working with again back when we were a dev shop or not, not Cycle being a dev shot, but the company I was building at the time was a dev shop. And so there's one of our clients that we were scaling up their infrastructure and this client had they had chosen that they wanted to use Digital Ocean for, for deploying things. And I don't remember exactly which dependency the issue was with now, but we had said and I don't remember like what what version of Ubuntu was back in 2014. Someone knows. I have no idea what it is. We'll just say it was [00:17:06] Ubuntu. [00:17:07]

Bret Fisher: [00:17:07] 1404 I'm going to guess 1404 okay.

Jake Warner: [00:17:09] We'll go 1404 Right. So, so we I guess my point though is that we said, hey, we want to make sure that this is not all in just one geographic region. We want to deploy these across different Digital Ocean data centers on the chance that something goes wrong. We just want to have that reliability that that availability there being in multiple regions. Right. And so I remember we were deploying servers and we had chosen Ubuntu 1404 or whatever the right version was at the time. I don't remember. And we had we had we had deployed five or six servers at that point and then we realized we had encountered this really weird bug. And it took us like five days to figure it out. And it turned out that even though we had chosen the same base operating system, Digital Ocean had slightly different variations in their operating system images for each region, and we did not expect that. So there was a version of I believe it was like OpenSSL or something like that that was slightly different on one region versus another and maybe.

Bret Fisher: [00:18:09] Laws of encryption laws or something.

Jake Warner: [00:18:11] Maybe I don't remember what it was, but whatever it was that that variation in that version was causing an issue for us. And so it was one of those things that was right around that same time that we were discovering containers. And that was kind of one of the epiphanies that we had of, Wow, we can build applications and not have to rely. So, I mean, yes, the speed of being able to take that code and get it deployed, but also the the speed of debugging, where if we know that it works today, we're not going to have to debug it. If it's running at two different servers in two different regions or whatever that is. So the speed of debugging has been obviously a huge value add to containers as well.

Bret Fisher: [00:18:48] And a lot of times we're not. You know, back then it was also hard to track all this stuff, right? So what are you changing on the host that's affecting all those, you know, change route, you know, or whatever you're doing to isolate back before containers? Right. We were doing we were doing concepts of isolation, but they were all like one offs and they weren't really consistent and it certainly didn't have all the dependencies. Right? So you still did a lot of shared dependencies and whatnot, especially because you were trying to save space. You know, if you're a fancy, fancy hosting company. Right. You'd, you'd, you'd mount in system utilities like OpenSSL for the, for the user. But, but then they could run stuff on top of that. So it was funny how we had some of these ideas, but none of them were really truly complete in a full isolation pattern, including their own IP address. Right. That that their operating system file system is blank by default, which is essentially what most people don't realize is when you can actually start a container from something called scratch. And the scratch means truly empty, not a single file is in my container and you have an IP address, you have a file system, you have memory and CPU, but you have no files until you start putting them in the container images. And we do have we do see kind of kind of see that trend now, especially with all these newer languages, right? You talk about PHP and so if we can go down that language rabbit hole for a second, one of my soapbox moments, a lot of times when I'm talking to people that often say, Hey Bret, you're in this container stuff like what's the right language? And of course that's a horrible question because there is no right language, right? It depends on people's expertise, what you're trying to go for, what type of app you're building, all that stuff.

Bret Fisher: [00:20:18] And that's why I think what we're seeing a lot of times now is that at a fundamental level, in the nineties when Linux and FreeBSD or any of the BSDs or any of these concepts on Unix were all being invented. And of course we had Windows at the time. It was a lot about shared components, right? If you think about like we all had shared DLLs on Windows and we all had shared binary libraries and you know, we all had the same Python version on Linux and, and so these were all about shared components, and that was a time where the the bandwidth was limited. So we didn't want to have to install things. Of course, in the nineties and early 2000s, we were installing things in CD's and floppies, but it was all about like we didn't have enough space on the hard drives. Hard drives were really expensive and we didn't have a lot of space to get new things there in an easy way with floppy CD's, USB drives and slow internet. So we needed shared components. But one of the interesting trends I also see now is when people start deciding on languages is we see that all the leading edge tech companies are all moving to languages like Golang and Rust and not to get into like why those languages exist or which ones better or whatever.

Bret Fisher: [00:21:24] But the single thing that's unique about them is that they're going back to statically compiled binaries, which means when you ship your code, you're shipping a single binary with everything, including the shared libraries in it, which sounds wasteful, right? It sounds like shipping a 50 meg binary doesn't sound near as optimized as distributed little python by, you know, binaries and libraries everywhere. Right. That sounds better. But in fact, today we got to look at our problems, right? Today, our problems aren't disk space or memory space or internet bandwidth to get stuff downloaded to our server. Right? Those are no longer the problems. We've kind of solved those more or less. Disk space is one of the cheapest resources you can get on the internet. So we don't care so much about shared components anymore. So it's funny because if you go read about these things on the Internet, often you'll get 15 year old answers like, Should I build statically compiled binary code like C used to do, write C++ and stuff and you'll see people talk about, No, you don't want that model, you want this shared component model and we all want to use APT, get to go get our dependencies and have shared dependencies amongst everything. And but those were different motivations. Those were different times solving different problems. And so it's it's just like containers, right? We were all trying to learn containers.

Bret Fisher: [00:22:39] The first time I tried to learn it, I did not understand it. I actually failed in my first attempt to implement it and had to come back six months later and revisit even the concept of it, because it didn't really make sense to me when I first tried it. And we're doing the same thing now with languages where we're realizing that there's new problems, and those problems are the speed of business, the speed of ideas. I need to be able to ship my stuff faster and be able to test and deploy it in an isolated way even faster and multiple updates a day. And this whole trend of just shipping constantly all the time, and that actually works a lot better and is a lot easier when you can build a single file and ship a single file. So if you're doing it ten times a day, shipping 10,000 files ten times a day, turns out to not be super optimistic and shipping a couple of files or a single file in a statically compiled binary is actually the trend. So we're again, these are all really when you break it all down, why do we implement these new technologies? They will never catch on unless they're saving humans time. And at the end of the day, that's kind of what Cycle is trying to do, right? Saving humans time and trying to optimize the path from idea to it's. Now in front of customers. And let's go do that again ten times a day.

Jake Warner: [00:23:53] Yeah. And you know, I guess so. So yeah, the whole goal behind Cycle, as you mentioned, is focusing on that simplicity. Our idea being that most people don't need a lot of complexity to get things online. How many companies out there just building SAS offerings where they don't need to have their hands into a whole bunch of different things that maybe some of these bigger healthcare companies might need their involvement. There's there's there's somewhere in the middle of the market where there's these companies that just need a base foundation to go off of. But going back to the the point that you made about languages. So I think I mentioned on on your show, I don't remember, but I think I mentioned that Cycle was built entirely and Golang now after we switched from after we realized the value containers were like, All right, is gone, we're going to switch over to going. But it's really interesting. You're kind of bringing all these things together now. I know that we're getting a little into the weeds, but the developers watching this will appreciate that is that I don't know if if you write Googling yourself, but one of the really interesting things that has come out recently is the whole bed functionality where you can not only just have that static binary, but now you can at compile time reference like outside files that should need to be embedded in that binary at build time as well.

Jake Warner: [00:25:11] So then you have you can have a JPEG. I don't know why you would, but you could have a JPEG literally embedded in that same binary app at build time. And, and it's really neat as we start like, like one of it starts getting my mind wondering as time goes on. Like, obviously, you know, you and I both agree on the value of containers, but as time goes on, are we going to be in a position where and I know that they're trying to solve this with Webassembly, but where we end up in a position where instead of running, you know, like instead of using all this extra REACT and things to have like client side stuff running in your browser where we're shipping full blown binaries as web applications that you just hit from a browser and it just runs in your browser, but it's native Golang or whatever the underlying technology is.

Bret Fisher: [00:26:00] Yeah. And I think that that's the I think that, you know, the way that Webassembly is going to win is if everything I'm doing doesn't have to change. And this is kind of what happened, this is the magic of Docker, is that nothing had to change other than one step at the initial idea of Docker. It was don't rewrite your programs, don't design your applications differently. You just take them and you knew encapsulate them in our package format, right? And then it can go everywhere and the exact same way and repeat 1000 times over. And that's why, like for example, serverless is to me always going to be something that you eat. It's either all or nothing, and it's always going to have to be greenfield written like you're going to have to start from scratch with that stuff because it's a new paradigm of how you write the code. Whereas Docker said, No, we're bringing everything with us. You've got a 1985 app that still runs on Windows. We can do that for you, right? You've got a Linux app that ran in 1997. We can bring that forward for you. So that's kind of what I think really caught on with containers was it didn't exclude anyone. It was very open. And I think that Webassembly can do the same thing if I can keep writing and everything I'm doing, and then if I want it to be in a browser, I just run one one utility, one command and outputs the single binary or whatever it might end up being.

Bret Fisher: [00:27:13] Right, because I'm not a webassembly expert, but I think if that was, if that was what was really happening and what was going on, then I think it becomes a no brainer because at the end of the day, what makes more sense? We're constantly moving all these files around. You and I are, you know, when we're building our apps, we're never just building our own app. We're building all these dependencies that come from everyone else in the world, you know? And so if I go scan, let's say the typical Nodejs app and I go scan a container of everything that is needed for that app to work, it's probably 100,000 files just for that one web app. Does it make sense to do that or ship one file? Right? It just it doesn't. At the end of the day, a lot of the times, the performance and the problems and just collaborating together as engineers, a lot of these things all come down to the assumptions of the language and the assumptions of the frameworks on top of that language. And I think when we look at older languages, I love Node.

Bret Fisher: [00:28:04] I was a huge fan of Node and I was very early on adopting like 0.6 I think was the first release that I actually ran in production of my own apps. And so, you know, 2011 time frame NPM was a huge game changer because again, NPM being a package manager, it's a little low level, but it's a package manager and originally it was just for JavaScript, but now all kinds of things come in NPM and it's sort of almost almost as universal as Docker, but not quite. So the thing about NPM was it was all about using other people's code and it was a huge differentiator in 2010, 2011 when we didn't really have that. Everywhere we end it. It also, of course worked in web browser, not just in servers. So it changed the game and sort of upped the level of package management and distributed application code by us all being able to pull in everyone's else's stuff. I remember shifting from dot net. We took our app and dot net and did what you're never supposed to do in a startup and we completely redeveloped it in Nodejs and we lowered the lines of code by over 50%. And we did that because we were moving from dot net, which was traditionally buy it or build it. You have to, you know, buy shared components in that world.

Bret Fisher: [00:29:13] Back in the day, there wasn't really any open source, there wasn't much open source internet. I think it was all like Source foundry or something like that or built it. So you were usually building it and you had to implement date and time format, you know, things that would automate that, right? So you did all these low level things as a team, but NPM changed that. It wasn't the only package manager at the time that did this, but it was becoming the best very quickly because I could just put in a one line command or one line in my code and boom, I have all these things over and over again and there was thousands and then tens of thousands, then 100,000. And now there's millions of these packages and you and I are now seeing the future of, well, that idea for package management is actually now burdensome because we're now all carrying around hundreds of thousands of files and sometimes sometimes in some of these Node.js apps, there's gigs worth of dependencies just to make your app work. And I think we're realizing that that inhibits business speed, that slows down the speed at which a business can operate. And we're we're always trying to solve more of these problems in this speed loop.

Bret Fisher: [00:30:15] And one of them now is too many, too much complexity in the dependencies, too much too many layers of dependencies I need now apt and yum dependencies and then I need NPM and Ruby dependencies on top of that. And then I need front end dependencies and I have like five dependency package managers. Right. But if you can get to these newer languages that they work in front end and back end, they work everywhere as well as being able to isolate down into just very discrete packages. It tends to, you know, teams are larger teams are finding that that's a much easier way to operate. So it's funny because it's not really about the components of the language or the necessarily the features of the language. Like you said, it's not always about that, which I think engineers sometimes really get into the geekiness of that and they start talking about. Classes and just various functionality of the language. But I think at a larger, much larger level and of course there's always fashion in tech, so it's fashionable to be on these new languages. But when you move beyond that, if the if the team can do more faster with a new language or a different framework, that's always going to be the number one thing that pulls the team away from whatever they're doing.

Jake Warner: [00:31:25] Yeah, it's something that it's something that I mean, look, this is me coming back to being a going developer. But to your point, that's one of the things that I really appreciate about the golang language and how it was built by the early team is that it's a it's a very opinionated language. There's not there's not so much there's there's only a few ways that you can really accomplish certain things, right? Like there's a very defined normalized pattern of how you build going applications. And so as we talk about the speed of even whether that's pulling in dependencies, whether it's standardization, like it's nice to be able to go from as a Golang developer to, yes, I'm building Cycle, but then to be able to go and look at code that someone else has written for a different project and not have to start making a whole bunch of guesses of how they approach things. It's just it's just so opinionated. But the other thing with the dependencies is we're talking about that is, is always that I think receives less attention than it should is the vulnerability that can be found in dependencies. I think people just have this inherent trust of, Oh, that's a widely used dependency. I can pull that into my code and maybe my code is clean. I spent a lot of time focusing on the the integrity of my code, but the second they pull in these dependencies, there's not much thought into well, what vulnerabilities might exist in those dependencies. And if you're if you're pulling into dependency, that has another as a dependency of a dependency of a dependency where you're four layers deep, are you how confident are you or how sure are you that you have not just pulled some of these vulnerabilities into your code? And so it's one of my worries is the company continues not just Cycle, but in general as companies grow and the projects grow, what other things have they pulled in to their critical applications? That might have just been a single line to include, but now is going to lead to a large data breach or something like that down the road.

Bret Fisher: [00:33:23] Yeah. And that's a security is something that's near and dear to my heart, especially container security. Now that that's all I talk about. But we're as an industry, we're getting much better at scanning dependencies, especially like if you're on GitHub, for example. Now if you're storing your code on GitHub, they have dependency scanners called ones called Dependabot that's built in to your code storage essentially. So you don't even have to you know, it wasn't that long ago that if we wanted to do something like that, you needed someone to implement a project to scan your code on a regular basis. And that took work and time. But now these you know, I'm talking about I've been talking actually just this week, I talked about them on my show. That actually wasn't my show I talked about at DockerCon's All Hands Community event, which had thousands of people. It's like a mini Dockercon. And I talked about the pipeline of building your apps, like what happens when you build your app? How should what are the different steps you should build? Where should you, where should you test it, How should you test it? How do you ship it to the servers? All those steps in the middle between committing code and it actually going to the server and the eventual deployments? What happens in the middle and what should be happening and what order should it happen? And that's a very gray area, I think, for a lot of people that the Internet's really good at giving us basics, but not really digging into examples and then varying degrees of examples.

Bret Fisher: [00:34:38] And so I kind of talked a lot about that. And one of my focuses was on CVE scanning the vulnerabilities in your app and its dependencies. Mostly nowadays those scanners, they really don't scan your code. That's a separate tool. But these code scanners that sorry, these dependency scanners will say trivial and sneak and there's other ones white sauce. These all focus on your dependencies. Right? And the nice thing is we all are at least defining our dependencies now. Right? So even if you're in Python or Ruby or go or Node.js or whatever, you've got these dependency files that are it's pretty easy for these scanners to just go down that list, find the versions and then compare it to the CVE database table that we all know about that lives, I think in Miter or whatever. And they, they show the list of vulnerabilities. Well, the the, the older the older languages have this sort of inherent problem where they're standing on the shoulders of giants. And so that's a good thing.

Bret Fisher: [00:35:37] But the bad thing is, is that when you need all these nested layers of different languages in order to make your app work, there's inherent risk that you can't avoid. So a great example of this, just a quick example is I think it was last year, probably probably 2020, maybe 2021 to in order to install Node.js in a container image with the most popular way to do that with something called node source, it actually would install Python in order to install node, which meant every container you had node in technically had Python as well. And so we started to have we had started having issues where Python vulnerabilities would show up in our container scans, but we never ran Python in the container. So we're all looking going, why do we have Python in here? We're using this particular node image. And it turns out that the the people that were installing Node just needed Python. So they assumed that they just installed it. They thought it was fine. Well, that was flagged as a bug. Eventually it got fixed. You no longer need Python to install node, but this is one of those things where I think some of these newer languages again, Golang, Rust, they're probably webassembly, but I'm not going to I don't know enough about it to know that whether or not that's confidently true, they don't assume all those things anymore.

Bret Fisher: [00:36:51] They don't assume that you have all these other tools on your system, because we all now know about containers and we know that we're trying to do things in isolation. You know, we we've all even got these local package managers like brew that install packages for us without us having to depend on the operating system to do it for us. So I think that that that trend is going to continue as we make more and more abstractions away from the OS itself and away from those base package managers like APT and Yum! We're going to get to the point where your app is really just one language. That's the only language that you're probably going to see in your container. And even if it doesn't build it to a single binary or static binary or whatever. It's still only going to be that language. And there's nothing really else in there, right? Like no base utilities, no systems package manager, because your app builds, installs everything itself. That's not true today for so many languages. They still assume you have host packages installed, other dependency managers involved, and that level of complexity is just quite frankly unnecessary nowadays. So we're.

Jake Warner: [00:37:55] Yeah well and I guess to to as we begin to wrap up the conversation around specifically building apps and correlating it back to containers, I think one of the neatest things that came out with containers, it's a few years old now, but it's something that it's still amazing how. How I don't want to say a little adopted it is because I think that it's it's much more widely adopted now than it used to be, but it's just the multi stage builds where if you do need those other technologies during that build process, you can just drop them out before you get to your final build. So if you did need Python for building, I don't know, just something in your application, you have that freedom to, to pull that in. But once you have that final output, you can pull that out into the last stage of your container and just not care about it anymore. And and I think that that was such a valuable addition to containers that yeah, I think that people new to containers sometimes gloss over the why that is so important I think that but but yeah.

Bret Fisher: [00:38:59] I do that when I teach containers nowadays you know even even when I assume people don't know containers like people that are brand new to Docker itself. And I at some point I'm always teaching the multi stage because I think it's an automatic evolution of you're going to have different things at different times in your images and let's design for that out of the gate. So it's a I think that's sort of almost a default that should be considered a default feature of any team using containers today is we need to use multi stage in order to get our test and dev stuff out of our production images and just isolate our app. And it's true production dependencies because again today a lot of teams those those package JSON files and Node.js apps, I'm kind of picking on Node.js because I know it the best, but you do an NPM install and you're technically installing all the dev dependencies and production dependencies and teams don't really fret about this too much until some security team person comes knocking on the door and saying, You got all these vulnerabilities in your application and it's dependencies and then the team realizes, oh, a lot of these are tools we don't even need in production. Why are we shipping them? And, you know, you and I couldn't have that conversation ten years ago or 15 years ago that that conversation would have resulted in completely changing how we even installed the operating system that we maybe couldn't even stop those packages from being on the server, you know, So it it truly became those are the days really where we were depending all on Red Hat and Ubuntu heavily or Canonical the company behind it.

Bret Fisher: [00:40:31] We were we were depending on those companies to constantly update things because we were shipping them on every single server. We had to protect them and we were waiting on them. If you remember coming from the Windows world, we had Patch Tuesday. Patch Tuesday was this once a month when Microsoft would patch, bundle up all their security fixes into one big lump. Because we were all waiting on them to fix everything in windows all at the same time. And nowadays we're realizing that that model is almost like a flawed approach, that maybe I'm not using IIS or a web server in my app. Why do I need to patch that? Why do I even have it installed? You know? So I think that's what containers in multi-stage even more. Just give us that freedom to to break all those things and isolate them up, you know?

Jake Warner: [00:41:16] And and that last point is a great segue way into the next thing I kind of want to dive into, which is as an industry around containers, what have we done right and what have we done wrong?

Bret Fisher: [00:41:28] Well, of course, being a container fanboy. Containers are right.

Jake Warner: [00:41:31] Yeah.

Bret Fisher: [00:41:32] We did that, right?

Jake Warner: [00:41:33] Yeah.

Bret Fisher: [00:41:35] I think. I wouldn't say it's wrong. I don't honestly, I don't think we're doing anything wrong. I think if anything that's wrong in the industry, it's that we're not dedicating enough time in the teams for explicit learning and sort of workshopping with each other new ideas. I think that honestly, for me, the teams that I work with, that the culture is one of their largest challenges to container adoption, to orchestration adoption to automation adoption, which I quite frankly, I think that's sort of the next revolution in in our teams is automating everything from the point that if when I commit code, eventually everything if you're not fully automated from that point forward, everything all the way to production deploy is being fully automated, then that is your future as a as a software team or as this software company that makes any software. That pipeline eventually has to be fully automated or were never going to get to the next point of speed. Right? If we're going to try to go faster and faster without employing more humans, we need to automate. So containers end up themselves. And that Docker build is technically a level of automation, right? That Docker built command was a huge step forward and speeding up our ability to package all our apps, all of its dependencies, and then isolate it in a single unit that we can do a Docker push, pull and move it all around the the internet, right? So that was a level of automation.

Bret Fisher: [00:43:06] But the teams often stop when it comes to that fully automated testing. We, you know, so much testing in fact that we completely trust the robots. We don't need humans to validate our code anymore. We just need robots like that level of automation and testing. Not a whole lot of teams have done that yet, right? That is that is a level of risk we haven't really consumed or been willing to do yet. And I think for a lot of teams when you look at their their pacing, that is where I see if you want to call it wrong, that's where I see the wrong is. We're focused on tooling but not focused on the culture of continual learning and continual automation. I mean that DevOps background automation is one of the core pillars and continual learning and sharing your knowledge is also another core pillar of DevOps. And so I still think that DevOps itself, if you really focus on those core principles, they will eventually drive the team to these levels of change rather than, oh, we're, you know, we're more excited about this particular tool or we're going to change languages or frameworks like those give you a little, little bumps. I don't think that that that you will see the huge shift once you've adopted containers to the next huge evolution in your team automation and your speed of business until you, I think, really get the the automation down and the continual learning.

Jake Warner: [00:44:25] Yeah, I mean, I would agree with that. I think it makes a lot of sense. And at the same time, you kind of, I guess, in line with with that as the adoption of the maybe let's just call it the adoption of the wrong tools, whether that's due to hype or whether that's due to maybe as these companies are continuing to grow and and build their build their products, etc., getting into a position where, I mean, let's face it, and I guess we actually did talk about this on on your show, but at some point, if you automated enough DevOps and some of these things can get boring, right? Like if you've done a good enough job at what you are doing, you've kind of made your job boring.

Bret Fisher: [00:45:09] As you should, as you should. Every good engineer is trying to work themself out of a job every day. Yeah. Like, honestly, that's, that's where we're at.

Jake Warner: [00:45:17] But, but I think I think we almost have kind of a like a counterbalance where we have people that there's something that we talk about pretty often in our company is are you adopting the right technologies for your company or are you adopting the technology or are you adopting the technologies that gives your DevOps team a challenge? Right? Like like do you need the complexity of whatever that solution is or would have something whatever? Obviously I'm biased for Cycle, but would would the a much more foundational technology get you to where you want to be without all this extra technical debt? Because like, just like how we talk about with developers with dependencies when you have I mean, like you can look at so many of these tools in the container space or even in the DevOps space as I mean really they're still dependencies, they're just not code dependencies. I guess technically they still are code dependencies, But like, I guess the point I'm getting at is tool dependencies, right? Where you end up with this, this very large stack of different things that you need to piece together this puzzle of tools and it's more there to help with job security and to help keep those DevOps teams excited about what they're doing as opposed to reaching the point that I think we both talked about of, of ultimate automation. And so it's just one of those things that I spend a lot of time thinking about is how can we how can we make that easier to write for these companies to adopt?

Bret Fisher: [00:46:45] I think too. This speaks to a little bit of a larger point. But I think that you're you're on to something where you're saying. Tools can be like fashion. You know, that's why I'm saying we have tons of fashion in IT. We have, you know, the the if you want to look at the latest fashion, you go look at the StackOverflow surveys every year of developers. And that list is basically the top ten of fashion. Right. And in tech, because we all have a gluttony of choice now we. I think there's a there's some really good older engineers that talk about you know in the seventy's and eighty's there was only a few programming languages one person could actually learn them all all the major languages you could know because you could fit them on one hand. And now we have gluttony of choice everywhere at every level of your decision matrix from the bottom. What languages do we use and support to? How do we implement? How do we manage this project? What tools we use to manage the the human ideas and the workflow? Like there's there's so many levels of choice. It's and it's infinite. It's just really is and it's that is in and of itself almost a job role to navigate the choice of all the different tools that all the different levels and that choice is not something that humans are good at.

Bret Fisher: [00:47:55] We don't we're not used to having 1000 choices and then having to pick one. Yeah, it's just not innate in our our humans human nature. So I think that obviously it's hard to talk about right choices. I mean, again, I'm very biased to containers. I think it's probably the best idea we've had in ten years of how do I implement business ideas faster in technology. The answer to me is the container needs to be a fundamental part of that, which is why when you talk about the CNCF, these other bodies, the Linux Foundation and whatnot, these are sort of the sole the, the entire realm of computing minds are sort of collaborating on all these ideas. And the assumption nowadays on all of them is most of the time you're probably in a container like you should be. Almost all the CSF tools assume you're in containers at this point, which they don't really state that because they it's almost obvious when you start looking at all the tooling, you don't have to do containers, but you probably should be. And I think that that again, it comes down to how fast can your business operate. The the level of learning that we should not understate is the learning is one of the hardest parts here because if you have one person in your team that's the avid learner, right? The one that goes home at night and reads books and the one that, you know, stays over or works at home on the weekend and they have a closet in their home and they're they're tooling around with all this stuff.

Bret Fisher: [00:49:18] They're often the one showing up with new ideas. But if you don't have a culture where everyone is learning, and I'm not saying on their personal time, but implement it as a part of your business processes. I used to run a team where I would. I was mandating 4 hours of quiet time study every week, and then the next week you were doing a luncheon learn on that topic. So it was happening. So someone every week was getting picked to teach the others what they learned in 4 hours on a Friday. And Fridays are good because like everybody is just sort of already checking out anyway. So we would do 4 hours of learning, you know, pick your probably hopefully should already know what you're going to be learning. And then one of you on Monday and this is sort of a challenge I would do. I would I wouldn't tell them who I was going to pick.

Bret Fisher: [00:49:59] And so on Monday, I would say, okay, so and so you teach us what you learn Friday. On Wednesday, we're just going to do like a 30 minute lunch and learn thing. And if you don't have I find if you don't have that culture of learning and sort of institutionalizing the learning process and teaching others, not everybody's going to be great at it and that's totally fine. But if you don't do that, what you end up with often is that one person who's always like, Hey, I've heard about containers, hey, I've heard about serverless, hey, I've heard about this orchestrator, that orchestrator, this new tool. And the problem is, is since no one else isn't paying it, no one else is paying attention. No one else is learning from them. So their ideas are often the one that get gets implemented or, you know, there's not a lot of people to validate or invalidate whether their idea is good or not because no one else is learning new stuff. No one else is reading or going to conferences. And so I often try to look at teams and say, how are you know, if I look at their technology stack and I look at their automation and I look at how they're doing things and I say I can kind of see to myself, yeah, they're really not optimized for speed.

Bret Fisher: [00:50:58] They're kind of they're stuck in a lot of these different human gates, as we call them, where they're they have to wait for someone to do this part and wait for someone else to do this part. And only this person can deploy to the servers or whatever. And they really need to optimize their team in order to get better productivity. I start to find often, in fact, I probably should start tracking this because it's so common that I start to wonder if it's a key indicator that if your team, your technology teams aren't making the learning and sharing of knowledge a foundational part of the team's infrastructure, the team itself, then you're going to end up in these weird loops where you're adopting the faddish tool that doesn't actually make you better as a team. And by better I mean faster. So it it I do think that there's there is a problem there that we we need to solve as humans and figure out how do we conceptualize the fact that there are now, you know, three plus container runtimes when nine years ago we had Docker, now we've got three or more. So it's. It just that one decision becomes problematic if nobody's learning.

Jake Warner: [00:51:59] So, yeah, and as we as we talk about learning, obviously you have your courses and things like that. And so for people who might be watching this podcast or listening to it and maybe maybe they have an entry level knowledge of containers or some of these DevOps tools are, but they want to deepen their knowledge into those areas. If those people are interested in taking your courses and stuff, how can they get involved with that? And yeah.

Bret Fisher: [00:52:33] Yeah. You can find me at BretfFisher.com. That's one t in Bret. No c Fisher. So it's like the opposite of what some people type. But you should be able to just Google that and Docker and hopefully it'll send you to me pretty quickly. Bretfisher.com is where you can find most things. I have a podcast, I have a newsletter, I have a weekly live show on YouTube. We have courses all on Udemy. So if you're in a company that's lucky enough to buy Udemy business, which is sort of the all you can, all you can learn plan for every company employee like big companies like IBM and eBay, do it, then you already have access to all my courses actually. And any new courses I have because those all go into that program. So but yeah, you can get coupons, links, discount codes, all that stuff at Bretfisher.com.

Jake Warner: [00:53:18] Excellent. Well Bret, it's been phenomenal to have you on the podcast. Really enjoyed our conversation. I think we've talked about a number of things that sometimes go undiscussed. People I think, go into the weeds. As we started talking about with the beginning of this podcast, some people will get so into the weeds with the underlying tech where it's I want, I want A versus B, and I can make a pros and cons list for each of those. The the that the, the underlying reasons why we're making a lot of these decisions and this advancement that we've seen in our companies has is not gotten the attention that it deserves. So it's been great having you on the podcast have really enjoyed this conversation and really looking forward to the next one as we as we wrap up, is there any other links that you'd like to share with our audience?

Bret Fisher: [00:54:12] You can. You can join me. You can join me live every week at on YouTube on Thursdays at Bret.Live. And yeah, I hope to see you around. We also have a Discord server with 10,000 DevOps professionals. You can do that at DevOps.Fan. Yeah, I could send you links all day, but just show up on our live show or show up in the Discord. We'll help you out. We'll find you what you need and see what we can do. So thanks, Jake.

Jake Warner: [00:54:35] Perfect. Sounds good. All right. Thanks so much, Bret.