John Lindquist has a conversation with the Mark Shust, an expert with Git and Docker. They talk about the 12-factor style of building an application and why devs should have a standard method.
John Lindquist has a conversation with the Mark Shust, an expert with Git and Docker. They talk about the 12-factor style of building an application and why devs should have a standard method.
Often developers don’t have a standard process with git. Mark talks about the gitflow workflow, a way of working with features and managing how that feature gets merged into the code base.
Though due to working with so many branches gitflow has its complexities. So, Mark trimmed it down and created a new workflow he calls git ship, which is gitflow without the development and hotfix branches.
Before Docker, Mark was running through a dependency hell. Though with Docker Mark was just able to deploy an image and not have to worry about anything. Docker is like a VM but without all the memory overhead! You can even deploy as many images as you want at a time. You can run Postgres, Node servers, and also use entirely different languages in each image!
Check out Mark’s course which covers all mentioned topics, Build a Twelve-Factor Node.js App with Docker
"12-factor Javascript Applications using Docker with Mark Shust" Transcript
John Lindquist: All right, I'm here with Mark Shust. Mark, why don't you go ahead and introduce yourself?
Mark Shust: Sure. I am a full stack developer. Mainly have a background in PHP and e-commerce. Yeah, I work a lot with Node, and Git, and Docker.
John Lindquist: Sweet. When was the switch from PHP to Node?
Mark Shust: I still work primarily with PHP, actually.
John Lindquist: Okay.
Mark Shust: But I work on, actually, a side project that runs with Meteor, React, and Node, so some experience there.
John Lindquist: That's an interesting stack. I just wanna say first off that I absolutely love your 12 Factor course that you're releasing, or that was released I guess. It's just like the perfect, for me, for someone who's a front end JavaScript developer, and who's not very confident in the whole Docker. I use Git, just the basic Git push stuff. I don't follow really a lot of the guidelines, but just to have something really laid out for me like, "here are the best principles". Then you go and show it, like that just ... Anyway, thank you for making this.
Mark Shust: Sure, yeah. I've always been a big thumper of best practices. I think they really help streamline and guide you in how to do things.
John Lindquist: Yeah, for sure. Let's go ahead and talk about Git and Docker and kinda the 12 Factor stuff. It starts with a lot of Git information, like best practices around Git. What sort of experiences do you have with Git around using the tools and shortcuts? Do you use just terminal-based stuff? Are you advocating Git flow to everyone you meet?
Mark Shust: Yeah. I primarily use command line and terminal for everything around Git. There's a GUI out there called Sourcetree, but I kind of always had problems with it. I use Git flow, it's a branching model for Git. It's basically a way of working on features and managing how that feature gets into the code base. I actually came up with a simplified version of Git flow, because there are a lot of complexities with managing the branches. It seems like you're always merging branches into another branch constantly.
John Lindquist: Okay, I can see that, yeah.
Mark Shust: Yeah, so I actually like, call it Git ship, and it's basically Git flow without the develop branch, and without hot fix branches.
John Lindquist: So features are just based on master?
Mark Shust: Yep, features are based on master. Then you create release branches at the start of every development period. And then you're just really doing merges into those release branches. It greatly simplifies things.
John Lindquist: Have you seen devs struggle with the Git model, or have you seen Git flow kind of fix that for people?
Mark Shust: A lot of the problems I see with developers working with Git is they don't have a specific way to manage their Git process. You need a way to manage your code base and kind of guide your code through the whole process. Git flow is typically used by everyone. It's kind of like the standard for a normal deployment process, so that does fix a lot of things. There's some complexity of merging branches, and all of that, but other than that it works quite well.
John Lindquist: I think if just you open the Git docs and you see the crazy amount of commands you have available you're like, "Well, I don't even know where to start". Git flow gives you that nice, "Well, first you're gonna work on a feature, so let's make a feature branch. Then commit to that, then push, and then merge it back in". It's like I think that's what people need to understand about Git is "Start here, do this, move it back over here, release it, test i". I think you explained that really well.
Mark Shust: And there are really so many commands with Git. I typically use about four or five of those during normal working process. There are some very helpful commands like Git bisect, and Git blame, but typically you're using just commit, merge, push, you know all the normal, basic things. So I wouldn't get overwhelmed with all of the commands with Git.
John Lindquist: Compared to what SVN and CDS and the stuff I've used in the past, and the catastrophic merge failures we used to have, Git seemed pretty stable and reliable.
Mark Shust: Yeah. I got started with my repository tracking on SVN, pretty much. And that was just a nightmare with a lot of folders being created with that SVN directory. The size of those repositories was huge. Always merge conflicts, things like that. But Git seems to solve all those.
John Lindquist: Yeah I think where people think, or JavaScript devs think now, "well just rm-rf the node_modules directory". You used to do that with your SVN directories. You're like, "well, I'll just blow it all away and check it out again, because that's the only solution that's gonna solve it". I don't think I've done that with Git.
Mark Shust: I've done Git push to SF quite a few times in the past. But I guess there's always a situation where you're gonna have to do that eventually, but for the most part not.
John Lindquist: Can you think of any good war stories around merges and releases with Git?
Mark Shust: Not really. I think the biggest faux pas I remember is trying to delete all of the folders in a certain directory. I do rm-rf.*, thinking that would delete all of the files and folders in that local directory, but what that actually does is wipe out your whole computer. Luckily I caught it before it erased everything, and it had time to process all the files.
John Lindquist: That's kind of a rite of passage, though. I mean we've all been there.
Mark Shust: Oh yeah. Especially on a production server.
John Lindquist: Yeah. If you haven't done it yet, just go do it tomorrow. Like if you're listening, and you haven't deleted everything off your computer or production, just go do it tomorrow, and get it out of the way, so you can experience it. Just get through it. And thank me later.
Mark Shust: Get it through when you're still a junior or middle developer. If you do it as a senior, you're done for.
John Lindquist: How about so 12 Factor also talks about dependencies, configuration, environment variables, all those sorts of things. Any good stories or advice around managing dependencies and environment variables and all that? That sort of seems to me like the, "I know front-end development, now the next step for me is I'm gonna have to manage my app as it updates, and getting it to a remote environment", and all these things.
Mark Shust: You know what, Docker actually makes all that very easy. So I remember working in the past, I went into a big dependency hell with Python and HiP. It's known for having dependency problems. I was building a POS system at the time, that used a Django back end with a package called Tastypie. The front end was JavaScript and VC. This is back in the day, I think it's called Dun-js now. It was kind of about jQuery and JNVC. I was trying to, I remember working and I was trying to assimilate instant reactivity with the RESP API. I was using stuff like, this was before redux and everything. I was using local storage and the firebase and web sockets and all sorts of things. Yeah I had a lot of dependency problems with that. This was before Docker was really released, but now if I had Docker, I'd just deploy an image and not have to worry about anything. Really never ran into any issues with Docker or using it with MPM or Yarn.
John Lindquist: Alright so, on the topic of Docker, How would you explain Docker to a junior JS developer who's never heard of it. Like, what's the- that you really need to start, like this is important to your career.
Mark Shust: Sure, I'd see it like a virtual machine, but without the memory and overhead, and the large footprint of a VM. You have your Docker demon that runs locally on the host, but that just takes whatever memory you set it at. And then you can run as many Docker images as you want at a time. As opposed to a VM where you can only run really one or two VM's at a time because they run, usually you know, two gigs or higher of RAM.
John Lindquist: You're buying more RAM. Literally purchasing more RAM so you can setup VM's like this.
Mark Shust: Yeah. I mean, the running unit of Docker is an image, so I'd see an image kind of like a VM in that aspect. And then images are built from containers. So a container is really a skeleton blueprint of the operating system and a list of instructions built from what's called a Docker file. And those instructions build that image. It's like your baking an image from those instructions. The Docker file's kind of the equivalent of a package JSON file, you're just defining everything you want to install and how to run it just like MPM run. It's pretty amazing that you can run Docker run, Dash IT, Ubuntu, Bash, and then you're in the Bash prompt in less than a second. As opposed to a VM where you have to actually boot it up and shut it down, and everything just works very quickly.
John Lindquist: Yeah, I think the most magical experiences I've had with that were working on Windows and doing that, and then putting whatever Postgres or anything into Docker, and just accesing it through ports on Windows. And like, wait a second, it almost felt like having a home brew or whatever on Windows, where I could just create these images, launch all these services, and just magically having them behind these ports.
Mark Shust: Very similar, yeah.
John Lindquist: So, how did you get started with Docker? What turned you on to it?
Mark Shust: Actually, again my background is like PHP and E-commerce, so, I work naturally with a platform called Magento, it's kind of the standard in that field. A few years ago, in I believe it was the summer of 2015, Magenta 2 was under heavy development, and it was very hard to install. So, I was working with Magenta 1 at the time, which ran on PHP 5, and Magenta 2 was built really for PHP 7. So I had that problem where I have a version mismatch, and I can't easily switch versions on my machine and run both of them at the same time. So I was passively following Docker for a while, and thought I'd give it a shot. I kind of went underground for like a month, and learned Docker. It was a bad month for me, because I have one year old twin girls at home, and I work from home doing my own kind of consulting agency, and yeah, it was kind of hectic and I really needed Magenta 2 to start running so I can kinda get a head start from everyone else. So yeah I came out of it a month later and I had a working Docker image set up, and yeah it's now the number one Docker set up for Magenta 2 still, a couple years later, and I still maintain it.
Mark Shust: Without Docker, I would have had a really rough time there. There was people delayed a good six months because they couldn't get it up and running.
John Lindquist: I love those brute force learning, or whatever you want to call it, where you're like "well, this is not working, I have no other option".
Mark Shust: It's learn by fire, as we're programmers and I think the more experience you have programming, you kind of have to learn sometimes you just have to learn by fire, and just pile through it.
John Lindquist: There's something about those, and as much as I hate and don't recommend all nighters, or writing code all night or anything, but every now and again there's times you just have to shove that information into your brain and have it all in there, no distractions, and just let your brain tear it all apart until it makes sense.
Mark Shust: Yeah, it seems like that happens to me every two years. It's not a common thing for me at all, I'm a big proponent of work life balance and everything, but yeah there are times you just have to plow through and do it.
John Lindquist: The whole, I'll throw out the paradigme shift, even though that's one of those buzz words, but you have to change your way of thinking to a new way, and that somehow requires a baptism of fire or whatever you want to call it, through those late nights. So how has, since your initial experience with Docker, have you seen it evolve and change, simplify, or anything?
Mark Shust: Yeah, so mainly it's just matured. A lot around networking capabilities and volumes, I'd say the volume improvements will affect a local developer more than anything else. But not too much has changed, it's really just gotten better. The final system has gotten quicker, more needed speed on Lenox for Mac and Windows. Yeah, and a lot of additional support for Mac and Windows. It runs kind of natively on Lenox, but you need a damon on Mac and Windows to connect it all.
John Lindquist: Yeah, I remember my initial struggles with it on Windows, but I think Microsoft kind of put some backing behind it and we worked Windows a bit to support it better.
Mark Shust: Oh, it's been great since the betta real, Docker for Mac I believe. The betta release, it's been fantastic, versus what it was before.
John Lindquist: Yeah, there's so much support behind it, so much momentum, and it's lovely seeing people work together that way. So, deploying an app with Docker, how do you go about that, compared to just having a Node index.js file?
Mark Shust: Sure, so there's another software package that Docker creates called Compose, and it allows you to run multiple containers at once very easily, so you basically set your config and enamel file, and then you do Docker Compose commands like Docker [00:16:16], and up and things like that. And your environment is set from a dotenv file typically, so you can use Compose to deploy containers. The more popular option, especially lately, has been Kubernetes. So Kubernetes is a platform that was developed inside of Google, it was originally named Borg. It ran all of their internal infrastructure, so that's kind of where Kubernetes, and really, Docker, was born from. There's a lot of complexities around Kubernetes and Clusters, and Nodes. It is complex to run, and there's a lot of additional terminology you have to learn on top of Docker.
Mark Shust: So I'd say a more basic version of deploying your software is Docker Swarm. It's greatly matured in the last year, and it's really just like Docker Compose, with just a few changes. So if you can run Docker Compose, you can most likely run Swarm, and deploy your whole fleet of containers out there.
John Lindquist: I think there, on the Docker website, Swarm and Kubernetes are side by side, and if you want to learn one or the other they'll kind of guide you through it.
Mark Shust: Yeah, Docker actually just built in the official support for Kubernetes. I don't think they really promoted it, they were promoting their swarm package instead. Yeah, Kubernetes has been just, very, very popular, so, yeah it's very cool.
John Lindquist: That's from Google right?
Mark Shust: Mm-hmm (affirmative).
John Lindquist: Because they're, I think on Google Hosting they're always pushing the Kubernetes options.
Mark Shust: Yep, because you have to run at least three [00:18:01] senses, to clear run it so. Yeah, Google cloud has a fantastic cloud service to let you set up and manage your whole Kubernetes Cluster, if you're looking into where to host it.
John Lindquist: Have you used Google Cloud much? Or other services?
Mark Shust: Yeah, I typically run everything on Google Cloud lately. When I started using it, it was very basic and it didn't do a lot of things that ajobs did, but yeah that's completely changed, it's a great platform to run on. It's very cheap.
John Lindquist: They have a CLI now, right?
Mark Shust: Oh yeah, they have a G cloud coming in that you could deploy your whole Kubernetes fleet with command line from your local environment, so it's awesome.
John Lindquist: My history with Google Cloud hosting and services and everything, have been just terrible UI's through their websites, and as soon as they released, I think Firebase did a similar thing, where they released CLI, and I'm like "finally, just give me a simple command and a couple flags", compared to some of the UI's they try and build around these commands and flags just boggle my mind.
Mark Shust: Yeah, Google Cloud got tremendously better once they re did everything on material design. It happened a couple years ago, but it's matured and I think it's actually easier that a UBS right now.
John Lindquist: Alright, so any other recommended tools for a Docker?
Mark Shust: Not too much, I typically run everything from command line. If you wanted a GUI aspect, there's something called Portainer, so you can run that locally or in production, or both. It pretty much overlays a GUI for your, like a whole overview of your app, what containers are running, what images are persisting, and everything. I'd say probably a more basic version is their package called Kitematic.
John Lindquist: Isn't that Docker's in house solution?
Mark Shust: Yeah, I believe so. We work with SVN so if you remember tortoise SvN, where they put little check marks on the files and folders. So I kind of see Kinematic kind of like that, where it's very basic and very simple. It doesn't allow you to do a whole thing, or a whole lot. But for a started, if you're just getting started and looking for something really basic, check out Kinematic, it's Mac and Windows.
John Lindquist: My experience with Kinematic has been, after installing it you see all these images, and all these wonderful options, and then you pick something to install, and it looks like its working, and then you go and try it and you can't figure it out past that. Like it's just full of potential and excitement, and then it shows that it's running and you're like, "well, I can't get to that port".
Mark Shust: Yeah, you saw the overhead, you should still, it doesn't prevent you from learning Docker. It's kind of like, just a very lightweight gate weigh into Docker, but yeah, once you're doing anything customizable at all, it's not useful.
John Lindquist: It seems like the idea is there, the execution is off, and I'm sure they're improve.
Mark Shust: Yeah, if they make it better that'd be great. Yeah, there is some over head with Docker, I don't think it's too bad, but you set a peradoxime shift of how you think.
John Lindquist: But do you see people making big mistakes with Docker? Common mistakes?
Mark Shust: Yeah, I mean, the number one thing I see is people trying to squash everything in one image, in one container. So, you take your Monolid thick app. It might be okay to do that, just getting started. If you wanted to kind of start getting into Docker and didn't want a full set up, or if you had a really hard time managing all your dependencies and installing everything, it might be an option. Typically, you want to run things like a micro services, architecture where you have one service for one container. And that's kind of utilizing the true nature of Docker, so you can scale serves independently of each other.
John Lindquist: I think if you don't know Docker Compose, you wouldn't think that way, right? Like you wouldn't think, I should launch all of these, because you're just thinking "I just need to shove everything into one, and then I'll deploy that one.
Mark Shust: Yeah, and you can, but when you're doing that, you don't have too much more benefit over running something like a VM on production. So if you build everything service oriented, you can deploy and scale those services independently one another.
John Lindquist: So let's just, just to clarify for people listening, can you give an example of services, like an example of what they do in Docker Compose, what they'd separate out.
Mark Shust: Lets say, you have a Nginx load balancer, so that could potentially be a service. And then you have, lets say you have one note app under it, so that's another service. So, typically front ends, front end load balancers like Nginx they don't require a whole lot of computing power. You know, one load balancer might be able to manage 100 times of the request of that one load balancer, on the up side. But, let's say your note app is utilizing a lot of memory or CPU, you can actually deploy many, many, note apps, behind that load balancer. And configure engine x to look at those note apps. So you could scale your note app independently of Nginx. So that's probably a pretty basic example.
John Lindquist: No, it sounds fairly complicated when you say it like that. I highly recommend watching Mark's course because he shows just how simple that can be, with a couple flags and a number, like of how many you want.
Mark Shust: That's a problem I have, you know sometimes write ups you find online look so confusing, there's so many steps. But, to set up an engine X Proxy, the proxy requests a note, it's actually like two or three lines, not too complex so. A lot of it's probably just copy and paste from [00:24:48] Can't blame them but.
John Lindquist: I've personally never set up anything in Docker without having some sort of a reference or other example project open, because I just don't want to get it wrong.
Mark Shust: And the same thing with Nginx configs, I don't know how many configs I see with all these values changed, and I come across another blog post with those same values and actually could look into the Nginx documentation and find out, "what do these do, like why are these here?" So, I'm always questioning why things are dawn, and rather than just blindly copying things, but sometimes you have to just to get started, but it's a good idea to really digest and dig everything, especially with Docker. You might find another image out there that does what you need it to do, but it might be completely over blown, you might be able to do the same thing with one - two lines yourself.
John Lindquist: I have a friend who, if he answers something on Stack Overflow, or puts it on GitHub, and there's configuration in there, he'll add in some non nonsensical configuration option in there, that won't affect anything, it has nothing to with it, just to see if people copy and paste, and leave that in. And you can google that, and see how many people are using it, as a way of tracking it.
Mark Shust: Oh that's great. Number one thing I see also is people copying commands, but they're setting the default value so, they have no reason to set it, because it's set by the default, so that's like, I see that 90% of places.
John Lindquist: No judgment here!
Mark Shust: Aha, I do that too sometimes. Lets get her done.
John Lindquist: So, the 12 factor absent general, have you seen a large adoption from the community once this got written up as kind of guidelines?
Mark Shust: Yeah, initially it had great feedback everywhere, but it's around the same time that micro services were getting a lot of great press, and you had two sides of the field here where some people were against micro services and some for. So, I think Docker naturally aligns with the micro service side, because it's a service oriented architecture and it just plays right into that. So I think, if you like micro service you're probably going to like Docker and the 12 factor concepts. So yeah, there's kind of a split way of thinking, I'm more on the service oriented architecture side, where you just have that ability to scale everything independently, so if you had a large app I don't see why you wouldn't move to it.
John Lindquist: Right, even we use some of those services for a few things.
Mark Shust: Yeah, and what's great about Dockers, you can use even different programing languages for different sections of your app, so. If you think of, I can run something and go, and expose a APIN point, and deploy with Docker, and my note app doesn't have anything to do with go. So it's really great if you have teams that are multi versed in different programing languages.
John Lindquist: Or go is the right solution for [the 00:28:19] we've had.
Mark Shust: Yeah, you have different problems and you have different ways of solving them, maybe better than using no for everything.
John Lindquist: Right on. Do you think anything is missing from the 12 factor principles?
Mark Shust: I don't think so, I don't think I'm that much of [00:28:37] to realize that there's something missing there. Yeah, I think the guidelines are pretty solid.
John Lindquist: I would never have the guts to write something like the 10 commandments of development or anything like.
Mark Shust: Yeah, whoever wrote 12 factors knew what they were doing.
John Lindquist: I missed their name, I just read their name when I looked at it this morning but, thank you to the person who wrote 12 factor, it's an incredible guide for people of my level.
Mark Shust: Absolutely, and it's something too, you can pull in three or four of these concepts and start applying them, rather than switching everything at once. You can kind of start digesting the micro service, and service oriented might have thought around that.
John Lindquist: Alright, any closing words of wisdom you'd like to share with everyone?
Mark Shust: No, just go forth and Dockerize. It's really great for a lot of applications so, I highly recommend it.
John Lindquist: Thanks a lot, Mark.
Mark Shust: Alright, thanks John.