Jake Warner + Darren Shepherd

Jake Warner: [00:00:12] Hello and welcome to the 15th episode of Cycles podcast. I'm Jake Wagner Cycles founder. Today we have a good friend, Darren Shepherd. For those who I mean, let's be real. Most of you already know who Darren is on the chance that you don't. Darren has spent his career writing orchestration systems first in the infrastructure as a service space and then Docker in Kubernetes. He is best known for co-founding Rancher Labs and creating such projects as Rancher, Longhorn, k3s and many others. Darren, great to have you on the show.

Darren Shepherd: [00:00:46] Yeah. Yeah, I'm excited to be here. Yeah.

Jake Warner: [00:00:49] Darren Let's let's dive straight in. So as I just kind of alluded to, most people know you from from k3s and, and obviously Rancher. Right. So I think let's start with that. In terms of in terms of rancher. I think most people in the ecosystem know that it was acquired a few years ago, but given that we have a lot of technical founders and developers that watch our podcast, I think it'd be really interesting to to hear from the co founder of Rancher himself of what was that journey like? What was your favorite part and what were the things that if you had to do them again, you do them differently. I know that was a lot.

Darren Shepherd: [00:01:33] Sorry. Yeah. Well, I wanted to say I like that you stumbled on k3s because like the k3s is the stupidest name. Like when we launched that project, like our marketing people are like, can we come up with a better name? And they kept talking about like pitching all these other ones. And I'm like, No, I really like K3s is like, That's stupid. And so it's like, it's a dumb name. Nobody knows how to say it. What's the official pronunciation? So anyways, but.

Jake Warner: [00:01:56] Is it actually K3s? Yes.

Darren Shepherd: [00:01:57] No, that's how I usually say it is. K3s But like the official statement is, there is no official way to say it. So some people say like K3ss or keys or. Yeah, different things but I always just call it K3s. S So we can go into the whole history of K3s. We'll talk about that later. Whatever. But, but. Rancher Yeah, just kind of like my story of Rancher. So Rancher was the first startup that I've done. I, I guess the only startup I've done, you know, before I was working, I don't know, just regular programmer doing whatever in the orchestration space. But when I saw Docker, I really got excited about Docker because I worked in the, the, the VM orchestration space and there was a lot of like promise of like hybrid cloud and all this nonsense. You know, OpenStack like people will see me on Twitter, I always like crap on OpenStack and I have good reason for it.

Jake Warner: [00:02:51] No wonder we get along.

Darren Shepherd: [00:02:53] Yeah. So it was kind of like my frustration with the ecosystem and what was going on with OpenStack and all the like empty promises of OpenStack. When I saw Docker I was like, Holy crap, I can actually accomplish what I really want to do with this. And so I immediately jumped on that and then luckily got like the, the other co founders of Rancher, they had founded Cloud.com. There was three other guys. Luckily I was good friends with them and we all decided to create a startup kind of in the Docker space. And so initially when we started Rancher, this technology was so early that it was just like help people run containers, just like because Docker was really cool on the laptop, but there was really no great way to run it. Like in a cluster, you could do a single server, but then across multiple servers and, and so we got started building Rancher, the first like rancher 1.0, which was just like, try to help people build or just run containers on servers. And and it was kind of interesting. We were very closely associated with Docker. We wanted to stay true to the Docker way of doing things. And so Docker was building Swarm, but it wasn't that good. So we kind of assumed where they would be going and we built an orchestrator with the idea that we would throw it away and eventually just adopt like Swarm.

Darren Shepherd: [00:04:18] And so that orchestrator was called cattle. And so our our rancher one was built on this technology called Cattle, which was our own orchestrator that we kind of built by accident. We didn't really want to be in the orchestration space. And so it kind of got wrapped up a little bit in the orchestration wars, but we never very heavily pushed it because there was like we figured someone else would solve this problem. So the rancher one product was like all Docker oriented, and it was largely we got most of our users off of our own orchestrator called Cattle, and we built like a good following, a lot of good following of people who liked Rancher one. Then Kubernetes came around and it was quite clear that Kubernetes was going to take over the market. So we did a massive pivot, dumped everything, rewrote rancher from scratch, and that was Rancher 2.0, and it was 100% Kubernetes focused. And so that's what we ended up building, like the. Biggest kind of business, and that's kind of what we sold to Souza. And everything was the Kubernetes business. But there's a really interesting history there because, like the rancher one was all about, it was very focused on the user helping users run containers.

Darren Shepherd: [00:05:27] Whereas the Rancher 2.0 product was really about helping people run Kubernetes clusters. So is this big pivot from like kind of helping the end user run applications to helping organizations basically run Kubernetes clusters? So kind of like the whole game change. And it's like if you follow me on Twitter and kind of like my I have a kind of a negative approach, like, I might say some negative things about Kubernetes. And it's it's kind of this of like where the ecosystem shifted from the user to the IT. So it's like we went from like containers or awesome, let's run them in production to Oh crap, we don't have a good orchestrator. And then the whole ecosystem was just standing up clusters. It's like it shifted down a layer to this really boring, annoying thing of like, I need two clusters and it just completely shifted away from users. So it's like, that's kind of the state I think the ecosystem is in right now is that we focus so much on like enterprise IT of running Kubernetes and just completely abandoned the end user of running containers. But anyways, that was probably not what you asked, but that was where I went with that question.

Jake Warner: [00:06:39] Well, no, I mean, I appreciate that, though, because, I mean, I think that, you know, as you as you were were telling that story, like, it's something that resonated with me. Right. And I think that from from our conversations, you know, just personally, I think that we've always been on a very similar page about that, except where when Rancher started to go all in on Kubernetes Cycle kind of stood like steadfast and like, hey, like, like so Cycles entire approach from the early days of, of building all these things was we we to the same point that you made about OpenStack was I had experience with OpenStack before building Cycle, right? I was, I was working for a company called Single Hop and we were deploying some open socks up and it was, it was, it was a whole mess. And so then when we started hearing about Docker and Kubernetes, I was like, Hmm, my long term bet is that Kubernetes is going to go through that hype. At some point it's going to be king, king of the Hill. But then at some point, just like we've seen over and over again and everyone's always like, no, this time it's different. It's never different. It's always the same. Sometimes it just takes longer to happen. Right? But the whole idea with Cycle was was how do we focus on that end user, you know, to the point that you're talking about, but at the same time be able to throughout that process have a opinionated approach.

Jake Warner: [00:07:58] So that way, instead of developers having to we always call the 1000 decisions, right? And in terms of deploying something, usually there's 1000. I mean, it's kind of a metaphor, right? But there's 1000 decisions that might have to be made to get something online and maybe a few hundred of those are decisions that you just make kind of subconsciously, because you've just all you've had that experience, right? You're not you're not actively making the decision, but you've just done it that way in the past or you're just kind of doing it. But in terms of automating it, obviously you're very familiar with all those subconscious decisions are still things that you need to automate and plan for, right? And so our goal is like was like, how can we empower the end user but be in a position where we still leave a high enough ceiling for people to build stuff, right? Because there's so many of these these kind of wrappers and platforms out there today where it's like, yeah, you can get started really easily, but the ceiling is so low you can't build anything of meaningful value, right? And so I think that's one of the things that I think has always resonated between the two of us. And so as you as you were sharing that story very much, it's like I it's just like, you know, we were kind of both going through time with with that same approach, I think.

Darren Shepherd: [00:09:02] Yeah, yeah. I mean, I might disagree on some of the like the, like I'm very negative towards Kubernetes, but I also see a huge, huge amount of value in it just because, like I since I've written these orchestration systems, I have a high degree of appreciation for what they've accomplished with Kubernetes, like reading through the code and stuff like that. It's like, like it's it's very well done. Like, like we're the opposite was OpenStack. Like OpenStack, I would read through OpenStack and I'm like, this is just pilot. Like, like, you know, these days, you know, whatever. It's like ten years later, it's probably much, much better. But I'm talking about the very early days and what people are writing. I mean, it was just garbage. Like there was not a there was clear when OpenStack started that there was not a kind of a heritage of how to write these systems. So they were making a lot of novice mistakes in how they built these systems. Whereas Kubernetes, it's very clear that it's like, yeah, they, the, the people who've contributed the initial code and stuff like that have a history. They know they know the past stuff. But so I think there's a chance. That's why I haven't completely given up on Kubernetes, because I. I think there's a chance that, like if somebody can build can succeed in building the layer above it, that there's an amazing opportunity because Kubernetes is basically ubiquitous now. It's like it's everywhere. But, you know, when I look out at the ecosystem, what people are doing, I don't have a lot of faith because I think everyone I like the majority of people are just like, it's just madness.

Darren Shepherd: [00:10:36] Like everyone just makes everything more and more complex. But I haven't personally given up on this base, so it's like I am like after I've last left, I left Suse in the beginning of this year, I am working on kind of trying to do the next thing or whatever to somehow make this a better, a better situation. But if I was, it's like, like looking at like Cycle, if I was to tell someone today, like, you know, a small company, like they're starting up, they have a small engineering team, I will never tell them to go to Kubernetes. Like my my number one, my number one thing that I typically tell people like your small company, you're like, you're getting going do ECS. ECS is typically just Docker with containers, and it works for most people. I mean, I think Cycle is a great platform platform too. So like that that it's like so it's like Cycle I think is more on like that the Cycle significantly better than I think, than ECS and, and Docker. But I'm saying it's kind of on that level of like people can grasp Docker containers. And so if you have a solution that's oriented towards putting a Docker container into production, go towards that. Just don't mess with Kubernetes. Kubernetes is like that's if you have an IT budget, you know, like.

Jake Warner: [00:11:50] And that's kind of where we found our success. Success so far is we've we've always kind of targeted companies before they begin hiring DevOps engineers, right? Yeah. And then but the idea is that once that happens, these companies, once they see the value of Cycle, that's where I talk about like having that high ceiling because there's a lot of companies out there that are like, Oh, we make Docker containers really simple, but like you really can't do much with them, right? Where with Cycle, you could choose your infrastructure. You can have hybrid infrastructure, you can have whether you want to be multi cloud like, there's so many different things that it begins with. But to your point, we have focused a lot on that simplicity of like, Hey, you probably don't need 90% of what is in Kubernetes. You really only need that 10%, but you need a 10% to be easily digestible and understandable. Yeah, yeah. But then throughout that process, like we've, we've, we've since brought companies on where they were in those early stages, but now those companies have grown like a huge amount. I mean, for example, we have a company on top of Cycle that is building, I don't want to say exactly an AWS Lambda competitor, but very similar. I mean, I mean, Cycles now deployed north of a million containers across all of our customer infrastructure.

Jake Warner: [00:12:55] So like, I guess what I'm trying to say is like it makes sense for people to like as we target new customers and things like it makes so much sense for us to target those startups when they're in their early days but then grow with them. And instead of being a platform where there's like there's so many platforms out there today where there's a very easy on ramp, but the exit ramp is just as defined, right? And you know, because I mean, there's I mean, a number of them out of the words like it's like, you know, the exit ramp. There's a point where a lot of these companies, if they're using some of these other platforms, where they're like, oh, I need to use Kubernetes now because I need more of that power, Right? Yeah. And so our goal is to be like, hey, like what Cycle? You shouldn't need to leave the platform. You have all these we've made it simple, but you still have all these different things that can do really neat things, like we have companies that are running north of a terabyte worth of RAM on the platform today. So things like that I guess is what I'm saying.

Darren Shepherd: [00:13:50] So yeah, yeah. And that's what I like, you know, because it's, it's like Kubernetes was such a good solution for the, the ecosystem of vendors. You know, it's like OpenStack was terrible for vendors. Kubernetes was, was really good for them. So it's like. Um. You know, I. I'd like to see, I don't know, like, you know, kind of be able to move forward and and have a better user experience of something like Cycle but being built on top of Kubernetes. And because it's like your product where it sits, it's kind of where I think most people should be operating. You know, it's like I like the user experience and what you're doing and everything. So it's like, but I'd like to see that kind of on top of Kubernetes, because there's a lot of advantage of like the just the ubiquitous nature of that just being everywhere, even for like edge and stuff. Because like the stuff we did with k3s was, was I just see a huge, huge potential in running containers at the edge. And so like K3s as we saw a lot of opportunities there.

Jake Warner: [00:15:02] Yeah. And so and that is one thing that Cycle has is purposely kind of shied away from is going after is I don't want to I mean I don't want to say this definitively because there's a couple of Fortune 500 companies that we've we've been talking with. But in general, it's not an industry that we've been we've been chasing because there's so many of these companies to the point that you already made, already have these IT. Budgets, they're already using Kubernetes, things like that, where, you know, they're so down that path that a solution like what you're talking about just makes so much more sense for them because like, they've already deployed these clusters, they already have them up and running. They just need other things on top to help me make it easier, simplify that experience. And and that's where like so like Cycles goes to go like from startups to like for the most part, let's say mid market with with the exception of a few Fortune 500 companies, that probably could make sense. But it sounds like the solution that you're talking about is more focused on like that, that extreme upward side of that.

Darren Shepherd: [00:15:58] Yeah, because because a lot of enterprise. So it's like, you know they've spent the last couple of years basically setting up and managing Kubernetes. I mean that's that's all the stuff we did at Rancher was just helping people run Kubernetes clusters. But like where they really struggle is onboarding teams and getting teams like setting up those pipelines. So it's like it's it's like the every single company builds some type of mini baths, even though all that is is like a pipeline. But they like they pretty much nobody really lets like the developer touch Kubernetes. It's kind of like you can check in and change a couple of these parameters and then we'll automatically deploy them. But so it's like we're companies are struggling right now. They've got Kubernetes and now they need to onboard teams and then they've in- house built something, a solution for that. But that solution doesn't necessarily scale or just the amount of effort it took to build that own solution. So it's like, you know, somehow accelerating, getting the developers on to Kubernetes or just basically being able to take the asset from the development team and put it on onto Kubernetes. That's where people are struggling right now. And that's where like, you know, you see these different solutions popping up like Octteto or LOFT, you know, Docker just acquired Tilt, all these kind of like developer oriented for Kubernetes, which I think I think developers and Kubernetes is like an oxymoron. Like, it's like Kubernetes is just not the system for developers. But yeah.

Jake Warner: [00:17:34] Well, and as you say that as we start talking about, you know, obviously you were foundational in, in building k3s and maybe maybe it's, you know, maybe it'd be good for our our our listeners and people who are watching this to hear some of the history of K3s like what was the decision that went through that you're like, hey, this is a great idea, we should build this, etc..

Darren Shepherd: [00:17:57] Yeah, yeah. So like there was no forethought into K3s is a complete accident. So I was working on a different project called Rio. So what happened is like, let me see, do I give you the honest story? Or the more so.

Jake Warner: [00:18:12] The honest story is the fun one

Darren Shepherd: [00:18:13] That's the more fun. That's more fun, basically, you know. So when we released Rancher 2.0. That was a major pivot where we went to Kubernetes. And it was clear that from when we went from rancher 1.2 to like a lot of it upset a lot of users, we knew it was like it was like it was just the thing we had to do for the business. And it made sense that like for us to survive as a company, we had to embrace Kubernetes. And and then if you looked at what are the needs of the Kubernetes market was really being a cluster administration. So like, so we moved away from, as I said before, like helping the user run containers. So like Rancher 2.0 was like, if you wanted to set up a Kubernetes cluster, it's great experience if you want to run an application on Kubernetes, good luck. You know, it's like, you know, it's Kubernetes. Who the hell knows? You know, it's like one day you'll figure it out. So so so what happened is so after we launched Rancher 2.0, honestly, I got kind of frustrated because I'm like, you know, this sucks. I hate you. Like, I don't like using Kubernetes. And like, it just it's so tedious to deploy anything. So I started working on a platform called Rio Rio. So that project is dead now. It largely was not a success, but when I started Rio, it was kind of like the scope was everything.

Darren Shepherd: [00:19:36] It was like, just how the hell do I embrace Kubernetes? Like use Kubernetes technology, but like not make it suck to run containers? So we built Rio, and largely that solution failed. But as a side effect of it was like, in order for development, I needed to be able to run Kubernetes really easily and like, I didn't want to use like Kubernetes and deploy a cluster. It's like as a developer, I just want to be able to turn on Kubernetes. So I built this like the very first hash of Kubernetes. Like, I think if you K3s guys, if you go all the way back into like, the commits or whatever it was like, it was basically it was like Kubernetes without the crap I don't like. It was something like that. So I just took Kubernetes and stripped out all the crap I didn't like in it, like the stuff I didn't want to use and just made a small, simple executable that when you turn it on, it ran Kubernetes. Somehow that got on to like Hacker News. Somebody noticed it or whatever and posted on Hacker News. And there was a lot of interest in like just the simplicity of running the how I packaged and made it. But people wanted all the features of Kubernetes because I ripped out all these these these features. So when I saw the response on Hacker News and then I started seeing more activity on Twitter, like people asking about it, and it seemed like, hey, people are looking for some like, simple solution to just.

Darren Shepherd: [00:20:58] Like just getting Kubernetes running was hard. Like, you know, you could run mini cube on your laptop, but it was just like, I don't know, like it was just hard enough for people. So. So k3s was really just built for myself of like, if I as a developer, how would I want to run it? And so, so I ended up adding all the features back into it. I didn't remove anything except for some drivers and crap. But for the most part, it's completely fully compliant Kubernetes distribution. And then we put k3s out there. We marketed it as an edge solution. But that was just from a business perspective. That's where we saw what message was, what would resonate. But by no means was the success because of EDGE. The success came from just people were able to easily run Kubernetes. So I got tons of responses to my first launching of like, Hey, this is the first time I've been able to actually, like, run Kubernetes and like, run a deployment by myself. Like, I would never have been able to get Kubernetes up and running before this. So, so K3s It was just, you know. Just packaging it up in a really simple way. So that's really all k3s is, is like a very simple packaging of Kubernetes that like when you turn it on, it just works and everything's there by default.

Darren Shepherd: [00:22:19] It has like a storage class and a service load balancer and ingress and, you know, it's like all the features and stuff are all kind of all turned on and ready to go. And there was really at the time, there was just nothing like it. So now definitely the k3s has been copied. There's like K zero and other things. I mean, I guess that's make micro caids didn't exist at the time, but, but I guess one of the fundamental difference between K3s and micro micro Kate's was k3s from the beginning when we did, it was like I didn't want to create a toy for development. I actually want to create something as capable to run production workloads because I'm like, you know, if I can spin up three VMs put k3s on three VMs, you know, then that's that should be production grade. Why not? And so that that was that was what we did. But that was a really long, long story, kind of like K3s. S but K3s was just a complete accident. It was, it was just something I built for myself and then seemed like other people liked it. And so we put it out there and then it's I think by far by far the most popular thing I've ever done.

Jake Warner: [00:23:22] And correct me if I'm wrong, I read a headline, I believe it was yesterday or the day before that Home Depot was starting to adopt. K3s.

Darren Shepherd: [00:23:30] S Oh, you can like I mean, I can't say all the companies that run k3s. But so it's like. Yeah. I mean, almost every major retailer fast, like at least in the US, like all these chains and stuff like that. Yeah. They're all looking or, or running K3s because so it's funny, it's like because we had Rancher, we talked to all these companies and a lot of them are our customers. And what we found is we might not have sold them on Rancher and then buying Rancher but they definitely went with K3s. S because because the thing is, is like when you start looking at, you know, a lot of times like when you have an engineering team, they kind of like the new shiny. So they want to they want to spin up a solution themselves. So they're not they don't immediately want to just go in and buy a product like Rancher or something like that. So. So k3s is like the most reasonable thing for anyone to, to deploy on the edge. And so yeah, so it's, it's running, I don't know, practically everywhere. I mean it's, and it's on like every type of moving vehicle to like, I don't know, like it's gone to space I think like in satellites and it's on boats and trains.

Jake Warner: [00:24:56] And, and how does that make you feel.

Darren Shepherd: [00:24:59] I don't know. You know, so it's like I don't know, like I don't take like a lot of like credit for k3s or anything because like, at the end of the day, like, I know what it is. It's like it was just a simple packaging of Kubernetes. So it's still just Kubernetes. So it's like so it was kind of one of those things where it's like, it's a very odd that that project like, I couldn't say like I had some great insight and created that. Like it was like it just kind of happened. And it's cool that I was a part of it. And because like if you were to put on paper like before K3s has existed, if I was to put on paper of like, here's what I'm going to build, it's a lightweight packaging of Kubernetes. It didn't like didn't make any sense. Like, why would that be popular and really take off or whatever. But, but it did. So it's kind of one of the Yeah, yeah.

Jake Warner: [00:25:47] I mean, it shows that the implementation is so important to.

Darren Shepherd: [00:25:51] Yeah. And that's what I like I kind of you because this is like I look at the success of Docker, you know, it's like the technologies of Docker existed for a long time. You know, there was LCS before it, but there is a subtle there's a subtlety to how they packaged and presented that technology that really clicked with users. And so I think that's kind of what we do with K3s. S is like there's something, there's some subtle nuance to how we did it that clicked, and I can't say exactly what that is, but it's there's like evidence that people used it. So, so so that's where I kind of get frustrated, like, let's say like on Twitter or something where people like there's a rant or there's an argument I had recently where people the the the Pod Man crew or whatever, renamed a Docker file to container file. And I'm just like, Yeah, I'm like, That's so stupid. Why can't you just name it Docker file? And they're like, No, no, no, this is a container. Like, it's not a Docker. And because the whole Pod Man crew is so adamant about, we're not calling it Docker anymore, these are not Docker containers, they're containers. And it's, and it's stupid in my mind because it's like, no, what you're building, what Kubernetes is built on is Docker containers. It is not a container. Container has existed forever, but Docker put a very specific spin on it that just worked and it clicked with everyone. So it's like at least give those guys some credit that like they created something. So it's like it, it annoys me to kind of like to just say, No, this is container and it is because like containers, now this like what a Docker container is, is now become our interpretation of a container. But, but it's like no, like that was a unique invention. Like that is like something different.

Jake Warner: [00:27:33] And all the specs that came along with it, right? Yeah. Like so many of those specs are like what became the OCI spec? I mean, I know that there's been changes to it over time, but so many of those things like to your point, like, yeah, containers have been around for a while with C groups and namespaces and all those things, but that implementation of it. In those specs is really what brought it all together.

Darren Shepherd: [00:27:54] And the specs aren't complete either, because the people that make the argument is like, No, no, no, there are OCI containers, right? And it's like, No, no, no, because trust me, go write a CRI runtime and you'll figure out when you do a cri runtime the reference for how all the nuances of what the container does is you have to look at what the Docker Damon does. If, however, there's a lot of nuances of just little things of like, you know, you launch your launch a container and it has a volume defined, therefore you must copy up the content from the, you create a new like ephemeral volume and copy up content. And those aren't in specifications anywhere like it has not. So it's like what we still so we're very far off from like an OCI container. It's like it's not the behavior is not fully defined.

Jake Warner: [00:28:41] But and that was, you know, that specific point was one of the challenges of building Cycle was like because we wanted like, we don't run Kubernetes and we don't run Docker, right? But we wanted to feel like we were running Docker for you, right? Like, we want we wanted it to feel like we were running a multi cloud, full infrastructure because like Cycles whole, like, I don't know if we, I don't think we talked about it yet, but I think you are from previous conversations like we like Cycle. So I'm jumping all over the place Cycle is more than just container orchestration like after that's one of the things that we've learned recently. We were we were talking with with Nick and I'm going to pronounce his name wrong. Nick Stein. [00:29:21] Nick Steinmates [00:29:21]

Darren Shepherd: [00:29:22] Okay. Yeah, yeah.

Jake Warner: [00:29:23] We were talking with Nick and and he's like, Why do you keep calling yourself like a teen orchestration platform? I was like, Well, because people use Cycle for containers. He's like, Yeah, but like, and this was mid demo and he's like, he's like, Yeah, but there's so many other things that the Cycle does here. Like you have an operating system that standardizes all these things. You, you ship with a standardized kernel. Like at one point he's like, he's like if you really wanted to, you could throw Kubernetes into this mix because you like he's like, he's like only half a Cycle is containers. The other half is all this infrastructure orchestration stuff and all these other things that go along with it that you could you could like he's like ended up being you know, ended up being stopped calling yourself a container orchestration platform. And so now we've started changing some of our language to be just the all in one platform because he's right. There is a lot of the problems that we solve for companies that are have nothing to do with containers at all. And but to your point of where like that Docker definition, since we don't run Kubernetes or Docker, like, you know, it just was built, I mean the only thing that we we, the only thing we run is Run C

Darren Shepherd: [00:30:25] Holy crap. Wow. You went all you went. Yeah, you went real raw.

Jake Warner: [00:30:28] We we went all the way. I mean, so, so we run Cycle, which is an alpine build with a specific kernel and then run C with nothing else. Right. But I had to so, so like when, when Michael Crosby I think you know. Yes, of course so Michael and I are good.

Darren Shepherd: [00:30:44] For legendary was Michael Crosby.

Jake Warner: [00:30:46] Yes so so so when when he when he was sitting here working on Container RD, I was building a shim for Cycle at the same time. And so we're sitting here like sharing our ideas about like how to implement this. And so it ended up being that they kind of were developed in along the same way. So we don't even run container D but if I were just having those conversations at the same time, so it ended up being this really like this, really implemented this really neat implementation where because we just run run C and that's the only thing that we run. I mean, who says we couldn't swap out run C and. I mean, people could. We could. It wouldn't be. A big stretch to be like, Hey, let's run micro VMs, let's run Kubernetes clusters on top, because there are so many, like by by taking control of that vertical stack and the way we have, we've gotten a whole bunch of functionality that I think that during, during a previous conversation you and I had, this was like two or three years ago, I was mentioning how a Cycle, everything is Golang, 100% of the platform is Golang, but we use Gob. Yeah, we use Gob encoding for everything, right? So we have, there's no such thing as legacy code for Cycle. Right. And I think that that's kind of going back and kind of going full circle here. That's where as I was talking with Nick and like it started to, like as it started to click for him to he's like, he's like, wow, if you don't have legacy code that you need to support, you have strictly design types everywhere, like why container orchestration? Like there's so many other like you've built almost a full ecosystem approach and to handle some of these things.

Jake Warner: [00:32:14] So remember how we got on this topic anymore? Oh, it was Docker. We had to, as I hit my microphone, we had to. Mirror that docker behavior. Right. And to your point of those intricacies, like this is something I'm embarrassed to admit, but we didn't have seccom support until like seven months ago. It was like one of the only things that I just kept kind of delaying, delaying, delaying until there was one of those big exploits that came out. And I was like, We should probably added Seccom, but even just adding seccom support. There are so many specific like again, if the goal is that Cycle should be mirroring Docker, we had to go through a Docker and make sure, okay, what are the what are the seccom rules that they automatically implement So that way if someone runs a container that was running perfectly fine on their local machine on Cycle, it should not be complaining. Right? So I guess to your exact point of that definition of what a Docker container is, again, I fully agree with you there. I think it's just people if it works in Docker, the rule is that it should work anywhere else. I think is is I guess what it.

Darren Shepherd: [00:33:21] That you do. Yeah. Yeah. Yeah.

Jake Warner: [00:33:28] But no, I mean, it's just it's just really interesting. So. So with K3s, so with K3s, with k3s it's, it's the implementation of it is we were talking about before, I think the implementation of it is why it ended up winning so much because I think that developers have this tendency to always overcomplicate everything and every once in a while it takes someone one kind of stepping back and saying, This is dumb, like why are we doing this? And it kind of gets everyone else thinking like, Hmm, you're probably right. Like, we don't need to be doing all that.

Darren Shepherd: [00:34:05] Yeah, I mean, and that's that's kind of what being at Rancher is like, had the kind of ability of seeing people doing this stuff over and over and over and over again where it's like, yeah, there's a definite pattern of like, this is kind of what you want because the problem with Kubernetes is like, you know, it's the it's the it's the platform for everyone. It can do everything. And so it has a million switches and all these random things and stuff like that. So it's like it's hard because I've gotten like comments of like, why don't you upstream k3s And why don't we just make Kubernetes itself easier? And I've always pushed back on that of like, you know, that kind of goes against the idea of what Kubernetes is. And what made Kubernetes so successful is that, like, everybody can contribute to it because it's everything for everyone and it doesn't have a specific opinion. K3s is extremely opinionated of like we picked certain technologies and bundled in certain things and set certain settings, and we're like, this is what we really think is the majority use case. And so has a very strong opinion, and that's something I don't think should exist upstream. So it's like I push back of like, no, this should be its own project. And I can't really make Kubernetes itself more easy. Like it needs to say kind of as the complicated beast it is. The thing is, people just need to build layers above it or and that's where I think the ecosystem has just completely failed. And it's it's partly because of users, too.

Darren Shepherd: [00:35:32] It's like, is that like when I built Rio, like I was saying Rio, that was a failed project. Whatever the intention of Rio was, I wanted to simplify the developer experience on on Kubernetes. So it first started off of like heavily extracted, abstracting Kubernetes and was like its own kind of syntax, DSL, whatever. But as I talked to people, it was like, No, we want it to be more Kubernetes. Like, we want it to, I don't know, fit into the Kubernetes ecosystem more. So I started moving that project closer and closer to like, be more Kubernetes native or whatever. And then it ended up just kind of falling apart because it was like. Like kind of the insight coming out of that project was like, No, you just can't have people touching you. If you want a better experience. You can't have people touching raw Kubernetes. You need to put a layer above it. So the problem with the ecosystem was nobody would accept a layer above it. Like everyone's like, I just want raw Kubernetes and I don't really blame them. But like, we're at least now at the point where people have Kubernetes and they realize the complexity and now are like, Hey, this isn't shiny, fun and cool anymore. I just want to get my job done. So people, I think, are willing to accept the layer above it that abstracts it to a certain degree, but I don't think anyone is anywhere near come close to a proper solution for that. So that's where we're struggling. So like the success of Kubernetes long term is really going to be if somebody can come up with that next layer, because Kubernetes itself is not a productive enough platform like the promise of the agility and stuff of containers largely gets hampered by Kubernetes.

Darren Shepherd: [00:37:06] It's like containers are great. But then, like, you move across and as soon as you hit Kubernetes, it's like this wall and it really slows it down. It's still better than what we had before, but it's nowhere near what we could have. So, like, in order for Kubernetes to continue to exist, if somebody has to build a layer above it, if somebody can't build the layer above it, then somebody will build the adjacent one like that. The layer above it will not be built on Kubernetes. And that's what we'll succeed in. Companies will go away. So I personally am trying to do that one because I think there's enough value in Kubernetes. And it's Kubernetes has given us like one of the problems with the cloud was like, everyone has a different API, and it's very hard to build a solution that goes everywhere. Now, with Kubernetes, I at least have an ubiquitous API for kind of infrastructure, so I want that to work, but I don't know if it will. That's what I'm. Myself personally working on right now is like, what's that layer above it? What's the the kind of the application abstraction so that developers can build an application and deploy it on Kubernetes without having to know Kubernetes significantly and and just make that whole, that whole thing work, which is so complicated.

Jake Warner: [00:38:21] It'll be interesting, though, like, like as you as you work on that and it's like, you know, what's like, oh, we can work on the adjacent. But that's the that's, that's the neat thing with how big the space is, is that there's always like, like I mean, you could have 20 companies in the space. And I mean, that's the thing about developers. Everyone is so opinionated and you'll have developers that have been burned by Kubernetes that are like, Hey, I don't ever want to touch Kubernetes ever again. Et cetera, etc.. Even if there's an abstraction. But then you have other people that are like die hard Kubernetes where they're like, like, okay, fine. I can have an abstraction on it that can make it easy as long as the underlying foundation is Kubernetes. But it's just so neat that the ecosystem is so large that it will support. I mean, I think the ecosystem is so large that there will never be a single company that is I mean, yes, right now Kubernetes is the king of the hill, right? Yeah. But I mean, it's kind of like the book that I have sitting behind me of Crossing the chasm. Right. And what you already to the point that you already had made previously of know it's not the shiny thing anymore, right? I just need to get my job done.

Jake Warner: [00:39:26] And so many technologies start out with that of so many people are just like, oh, like, look how shiny it is. Look how newest. I could customize everything. But as time goes on, people want something that just works, right? Like when, when, when we've been in sales meetings, we always refer to it. And, you know, I get slack every time I every time I say this, but I always refer to it as the Android versus iPhone model, right? Most people start out with an Android phone because they want to customize everything, right? Yeah. And of course I'm generalizing the shit out of this right now, but as time goes on, a lot more people start switching to an iPhone where they're like, okay, yeah, I can't customize everything anymore, but that's okay. I just wanted to work, right? And again, and maybe that has changed, especially recently. I haven't used Android like. Eight or nine years now. So that might be very different. But back when I mean, like back when I was using Android, it was Eclair or yeah, something like that.

Darren Shepherd: [00:40:20] I actually just switched from Android in this last year to iPhone. Yeah, yeah.

Jake Warner: [00:40:25] Yeah, yeah. And so, so that's, that's the analogy that we've always been saying is, is like so we had Kelsey Hightower who saw a demo a Cycle a few months ago and at the end of the demo he's like, he's like. He said something along the lines of like, this is really neat. He's like, he's like he's like, I appreciate how Cycle's taking an apple like approach to containers. It's highly opinionated. It's not for everyone, but there's a certain subset of the ecosystem out there that are like, I don't give a shit about all of these things. I just want people to run containers. And so it's just kind of neat there. The one question I do have about your long term still optimistic about Kubernetes is what is your thoughts about supporting legacy code and the update process as new versions of Kubernetes are coming out? Because that's like that was one of the biggest issues or Sorry, yeah, I'm excited.

Darren Shepherd: [00:41:15] Are you talking about just like supporting older versions of Kubernetes or newer versions like or you just mean like people have like an old, you know, dot net application or something like that.

Jake Warner: [00:41:24] So I guess let me set the stage. So with Cycle, one of the biggest problems we solve is that all servers connected to Cycle run the exact same OS, the exact same kernel in the exact same right. So, so that way whenever we push out updates, which is ever on average, like every two weeks, everyone gets the latest version of Cycle, right? There's no maintaining existing code. But I've also read how and to put it bluntly, I've never taken a Kubernetes cluster and upgraded it to a new one, so I don't have much experience there. But from what I've heard from other people and things like that, I've learned or I heard that the experience was very was very difficult to the point of, I believe Forrester did a report about it where they said that for most companies it's better off to deploy a new cluster and then migrate to it, as opposed to doing an upgrade or an update to a in place cluster. And so I'm just curious what your thoughts are in that long term.

Darren Shepherd: [00:42:22] Yeah. Well, I mean, so upgrading clusters, I don't know, like every year we did well, I mean it's an effort. Yeah, it's an effort, but we wouldn't go as far as they like deploy a new one. You could always upgrade it and Rancher did a decent job. But just. But the point is, it's like it is a major shift. Like, that's that's kind of like one of the difficulties of Kubernetes right now is like they have new releases coming out. I don't know what it is like 3 to 6 months. I can't remember. The schedule is now, but it's way too fast. So they have new releases coming out and then it's faster that than most enterprises can actually do the upgrades of the clusters. So they'll they, they fall back like a couple of versions and whatever. So it's one of the advantages of like EKS right now is like EKS will maintain a version of Kubernetes for like 18 months, which is like, which is really funny because like for, for enterprise, 18 months is like, not that long, but like for Kubernetes, that's a lifetime. So it's like, holy crap, you'll maintain it for 18 months. But anyways, so this is, this is my point about this is like why I'm kind of optimistic. So the problem is that people focus too much on Kubernetes providing the solution, the core Kubernetes primitives of pod, service, you know, a config map ,secret, like everything that's in basically like the core of V1 API and maybe like the apps APS V1 and batch B1.

Darren Shepherd: [00:43:44] If you take those three APIs, whatever, like the very core ones, they're good and they're very stable. They don't really change that much. Like so they've all, everything's moved to like a stable version. Now if you just focus on those APIs, you've got a good enough solution to build any layer above. You don't need much more from Kubernetes. So the core Kubernetes is pretty stable and boring and dumb right now. And so if you just were using like just that, then upgrading cluster should be significantly easier. But when you start throwing in like operators and CRDs and all this other stuff, it just is madness. I am so not a fan of operators and really CRDs to a certain degree of like Kubernetes is not the solution. Like, like Kubernetes is a really good system to run pods and do container orchestration, but don't use it as your core orchestration platform for everything because like, you're just going to create this really big, like it's just way to advance everything. So what I believe is, is basically, if you can standardize the layer above Kubernetes, then it means the requirements of the of the of the lower level becomes so much easier to the point that running those clusters is a no brainer.

Darren Shepherd: [00:45:09] And pretty much you can still run them yourself if you're doing like on premises or you can get them from a cloud provider. So what I think is that like the only way for the insanity like is like right now, like what you're saying the current state of upgrading clusters is kind of a disaster, but it's because there's too much crap going on in the cluster today and everyone's just doing so. The success again, is like the success of Kubernetes is like if we can start moving up a layer a little bit to get a more standardized application layer of like what's the package and and how things are being deployed on top of Kubernetes. Then it clarifies the requirements of the actual cluster, it simplifies it, and then the operations becomes drastically easier because I do think if you can do that, then it will just get to the point that pretty much everyone will just buy, like they'll either be for like in a cloud situation, you'll just get like EKS or GKE or whatever. You just get the like you won't be running Kubernetes yourself just the same as nobody runs VMs themselves with the exception of VMware. So if you're if you're on premises, then you're going to buy an on premises product, which will then give you the clusters. So that's, that's kind of where I where I see things going.

Darren Shepherd: [00:46:19] But this is all but you can see like the whole thing is like it's very I'm only skeptical. I mean, I'm sorry, I'm only optimistic because I'm personally working on it. And if I personally like not to say like I'm great or something, but the point is, is like. The current state sucks. It's terrible. And so but I see the opportunity to fix it. And so I, I'm still trying to fix it, you know, it's just like because there's to me, you know, I think there's a big opportunity. It's like, you know, it's like you're saying the space is large enough. There won't be like one winner for sure. And if you look at the cloud, it's like, well, we have like the major cloud providers, but like it's like AWS does not negate the need for Digital Ocean, right? Like those to still exist. So it's like there's still going to be all these players but like. But Kubernetes could kind of become the AWS right? There is there is a possibility of it becoming significantly larger. But like because when we started Rancher, the entire purpose of like the initial vision of Rancher was that we kind of wanted to build like the overlay cloud. We looked at it and we said, you know, the way I feel is that like AWS is the modern equivalent of the IBM mainframe.

Darren Shepherd: [00:47:42] They've built this this massive like kind of vertical stack of like you run on us and you run all the AWS clones of software. You get like the AWS, like kind of like store brand of all the technology, you know. So and it's all kind of locked into AWS. Like, I want to see a solution where it's like I can have cloud providers which provide infrastructure and I can just run wherever I want. And so Kubernetes gives me that chance of like everyone's exposing Kubernetes APIs. And so if I can capitalize on that, then you can get your compute storage, networking from anywhere you want. And then then you can and then you just run the workload. So it's like if we can somehow make it so that like we have a package for an application which is more like a like an iPhone app or whatever, where it's just like install and that just runs on any Kubernetes cluster easily, then I think that would be amazing. And that's honestly what I'm working on. But like the but the problem today is like, that's not the case. To get something onto Kubernetes, it's like you take your container and then 5000 lines of YAML, you know, and then some customize or maybe throw in a helm chart and then do some git ops and then you can get it on there. Like that's, that's a, that's madness. Yeah.

Jake Warner: [00:48:58] Yeah. I mean, and I mean again one of those places that we aligned because like with, with Cycle that was, I mean that was kind of the foundation behind this idea too is we want, we wanted someone to be able to run AWS and maybe at, back, back when we started it was like AWS and Packet right now, now Equinix metal . But the idea is that that's why we went down and built Cycle OS because the idea is that like even today, if someone can, you know, someone creates an account and begins deploying infrastructure, if they say, Hey, I want to use AWS, I want to use Vultr, and I want to use Equinix metal, which are the three providers we natively support today, they can deploy an infrastructure at each. The only thing they needed is an API key, but when those servers boot up, they have the same kernel, they have the same OS, and that way it's fully standardized. So that literally the interface you can say, Hey, I have this one container image deployed everywhere and then Cycle automatically figures out how to build. Like Cycle will build a flat L2 network everywhere automatically. That's that way. It's just that way. Everything like when you are inside a container, you don't know what host you're at, you don't care what you host you're at.

Jake Warner: [00:50:01] The only thing like if you if you need to care, there's an environment variable tells you what provider you're at, right? But otherwise it doesn't matter. You're just a container and everything else is so standardized as well. And so, so there's, there's, there's that side of it. And then but one of the things that we've learned recently is, I mean, like I said, we only have three native providers today, but and let's see, this podcast is going to go live in two or three weeks or so. I don't know, I maybe I might be jumping the gun a little early here, but we are getting ready to release on our side what we were calling our Infrastructure Abstraction Layer. And what happens there is that you have 13 API calls that Cycle will callout, you give Cycle a base URL Cycle, we'll call out to that base URL and there's 13 endpoints on that. And as long as you can satisfy all of those with JSON return, you can add any infrastructure provider back to Cycle. And so it's one of the things I'm particularly excited for because that way, instead of being like the three providers that we have today, because like I guess for some context of the last 200 to 300 Organizations that have signed up for Cycle.

Jake Warner: [00:51:13] Somewhere around 65% of them came back to us and said, You don't support the infrastructure provider that we would like to use. Yeah, that's obviously a big problem. But at the same time, we don't want to be the company that is building every single integration for every single provider, right? And so the idea is that with the Infrastructure Abstraction Layer, we would open source that. So that way suddenly these developers all over the place could build those integrations and then share them. So that's kind of one of the things that we're working on. And like again, just it's amazing how it's amazing how you and I align on so many different beliefs, even though the tech stack that we've been working with is just so different. But I mean, it just shows that. It just shows that there's I mean, obviously there's many ways to accomplish things, but it's. It shows that the. The fact I guess let me try and say it this way. The fact that the fact that you and I align in so many of these different kind of Foundational concepts and ideals shows that how many of the developers out there are looking for these exact same things that you and I. The same problems that you and I are seeing. And that's what gets me excited.

Darren Shepherd: [00:52:28] Yeah. And I would encourage, like anyone, like there's so much opportunity still out there because it's like there's, there's still like there's room for so many different companies, so many solutions because there's still tons of people struggling. There's like, there's no way. When Docker started, there was kind of like this funny opinion that like. Ok Yeah, we solve VMs in the open stack. Like they just were like, Oh yeah, that's solved and moved on to Docker when it was never solved. And we're kind of getting to that state right now in the ecosystem when people think that like, Oh yeah, Containers is all figured out and it's just like, no, it's like a total tire fire. Like it's like it's just like, no, no, Like there's still so much to just go and talk to users. Go start. If you go and talk to companies like, what are you doing? What are you struggling with or whatever, there's still. So I, you know, I, you know, it's kind of like a a parting note or whatever here. It was like, it's like, I would love to see more innovation. I want more people trying stuff. Like people assume that this stuff is already figured out and it's really not like the space is kind of become boring. And I'm not seeing enough people trying dumb things like just do something and, you know, make users happy.

Jake Warner: [00:53:41] Yeah, and I would definitely agree with that. And so as I know that usually we aim for 45 minutes here. We've had a great conversation. We're 10 minutes over. I know that you're working on a a a new project right now, and so it'd be great to have you on the podcast again when that goes live so we can dive into what that looks like. What, what, what problems have you encountered along the way? How does the solution look today? I think that'd be a really neat to dive into. Yeah, but yeah, definitely.

Darren Shepherd: [00:54:12] I was just saying just, just plug it sometime in July. Expect something from from me. I've put a lot of effort into this and I really hope people like it. I'm kind of afraid to launch it because it's really like put a lot of effort into this. I'm trying to make it a good thing for people, but, well.

Jake Warner: [00:54:32] I'm definitely interested in taking a look at it and learning more. But it was great to have you on the podcast today and looking forward to the next time, if anyone wants to, to follow you on social media where they can, where can they find you?

Darren Shepherd: [00:54:47] Twitter That's the only place I am. It's Ibuildthecloud is my handle. Yeah, Please follow me on Twitter. Basically, I just I mostly just rant about how terrible everything is. That's that's pretty much it's Twitter is where I go when I'm frustrated. So I as I'm working on technologies, I get frustrated and I rant on Twitter on how terrible everything is.

Jake Warner: [00:55:08] So excellent, same, same for anyone listening and watching this. If you're listening, this podcast is also available on YouTube. If you're curious and you want to watch, you want to watch videos of us talk. Otherwise, if you want to listen, we're on Spotify, Apple Podcasts, and Google podcasts and until next time. So it's been great.