Joel Hooks interviews Andre Staltz, an open-source hacker, and creator of Cycle.js. Andre quit his job to become an open-source hacker and now spends 30% of his time on open-source development and 40% on the Scuttlebutt project. Today they discuss the current web's stagnation, the vision of the peer to peer web, and what André is doing to reach that goal. They'll also discuss things that are more in Javascript land, such as Cycle.js and the callbag spec.
Joel Hooks interviews Andre Staltz, an open-source hacker, and creator of Cycle.js. Andre quit his job to become an open-source hacker and now spends 30% of his time on open-source development and 40% on the Scuttlebutt project.
Today they discuss the current web's stagnation, the vision of the peer to peer web, and what André is doing to reach that goal. They'll also discuss things that are more in Javascript land, such as Cycle.js and the callbag spec.
Scuttlebutt is a web protocol, like HTTP. It's like a vast array of JSON objects that sync between two computers whenever they are both on the same network; this enables data to never reach an outside server, a true peer to peer network! Andre goes into his work on the project and why he believes it is necessary for the future of the web.
But what is the peer to peer web and why is it better/different than the internet as we know it? Andre says that we are reaching a point where innovation is beginning to stagnate, where it is just enough to have Google, Amazon, and Facebook. We have reached a sort of peak, and things aren't evolving further. Andre goes on to say that one of the fundamental things that the internet missed early on was that it didn't guarantee a p2p connection.
Andre gives some examples of how you begin to use the p2p web today. The Beaker browser, for example, can still access HTTP and HTTPS connections. However, it can also use the DAT protocol. What is DAT? Well, it allows you to directly "seed" your website out, and others can "leech" it. Like torrents, the more peers there are accessing your website, the better! He also talks about Fritter, a twitter clone that only runs on DAT. You download the front-end and JSON files of what people are saying. You are even able to fork the front end and customize it for yourself!
Back in Javascript land, Andre talks about how he plans to properly support the Pull data source in Cycle.js, as well as having web-workers in the middle. He also talks about why he's removing the last library dependency from Cycle.js, xstream, in favor of just using a set of callbag utilities.
"Reactive Programming and the P2P Web with André Staltz" Transcript
Joel Hooks: Hi, Andre.
Andre Staltz: Hey, Joel.
Joel Hooks: Yeah, so I wanted to catch up with you. You've had an interesting ... 2017 seems like it was really interesting to you.
Andre Staltz: Indeed, yeah.
Joel Hooks: Yeah, you quit your job, so you could work on open source and be an open source hacker as a career. How's that going?
Andre Staltz: So, basically, it's not that simple that I quit my job because prior to quitting, I actually was part-time. So, the other time, I was actually doing open source maintenance and doing Egghead instructions, like courses, and also doing workshops here and there. So, I was transitioning quite gradually and not sudden. But yeah, I basically wanted to do even more open source. One of the ways I'm getting income is through Open Collective. The other way is workshops here and there and Open Collective ... I mean, Egghead. So, all these different types of incomes is what keeps me alive.
Andre Staltz: Yeah, and then I'm doing a bunch of open source stuff that doesn't make money as well, not that it doesn't make money, but in the sense that it doesn't have funds like Open Collective.
Joel Hooks: Right, yeah. So, what are you working on? What are your favorite projects?
Andre Staltz: I tried to put something like 30% or 35% of my time to open source Cycle.js with funds, and then something like 45 or 40% of my time goes to a project called Scuttlebutt, and I'm building a mobile app for that. Scuttlebutt is like a distributed or decentralized social network. It's pretty much a social network in the style of Facebook or Twitter, but it's like it's fully local. You have all of its database on your computer. So everything that you say, everything that your friends said are stored on your computer and that's pretty cool. It actually works really well already. A lot of people are using it.
Andre Staltz: I think a lot is actually hard to measure because there may be people who are using it, who are not attached to my friends, my friends of friends. But last time I checked, I have stored on my computer about 2,400 people. That's just like what I want to store on my computer. So, yeah, and it's pretty cool.
Joel Hooks: They're fully anonymous though if they want to be, right? Is that part of the point of decentralize? Is it part of it or-
Andre Staltz: Not necessarily. I mean, some people want to be anonymous; some people want to be just tell who they are. The point is not so much about being anonymous. The points is about just having power over the medium really when you think about it. It's disappointing that one person could be banned forever from a platform. I mean, these kind of things happen, and that's not our main concern, like avoiding people being banned by decentralizing. Our main concern is mostly like, "Hey, I just actually want to own my data, and I don't want to have any intermediates."
Andre Staltz: Then on the other hand, Scuttlebutt also enables a lot of stuff because it's not centralized, and that's related to different patterns that we could build with, like the social protocol. So, for instance, you could build pretty much anything, but just as an example, people have built already a chess app on top of that. They've built a small blogging platform like Medium. One subproject is building a democratic voting system on top of it and a bunch of things. But everything's like ... Imagine if there would be no restrictions on what kind of social features you can build because now, we're just dependent on what Twitter corporation decides to roll out as a feature and whether the favor is going to be a heart or a star. That kind of stuff.
Andre Staltz: But imagine really if you could just do anything, and we're exploring that kind of possibilities as well. It's pretty cool.
Joel Hooks: It's kind of like protocol, I guess, XMPP or something like that. Is that fair to say but for a social network, like a social network protocol?
Andre Staltz: Sure, yeah. So, it's a protocol. When I say Scuttlebutt, it's basically just a protocol and then there are apps that use that protocol. Sort of like HTTP is a protocol, and then the browser is one of those things that use HTTP. But the protocol is actually a pretty simple idea. It's basically everything that you say in a news feed or basically your timeline, you could say. That's just like a huge array of JSON objects and one object is one message that you said and that's it. Then your friends will replicate that, copy-paste that whenever they can sync with you through the internet or through LAN.
Andre Staltz: So, basically, it's a way of syncing all of these arrays so that you can copy-paste whatever your friend said and keep it locally. This local thing is pretty awesome as well, because then you can read it on the plane, or you can read it when you're offline. There may be thousands of stuff, and I mean a lot of content, and you can read that all offline. Yeah, the protocol just dictates how to do that syncing and how to avoid problems with, let's say, tampering of the content. So there's also some cryptographic hashing and signatures going on there in order to keep it safe even if you don't trust the medium, let's say.
Joel Hooks: Yeah, this all gets into the idea, to me, anyway, and that you use this term. It's the first time I think I've ever heard it. But the peer-to-peer web versus the internet that we all know and love. I was just wondering your take on the differences and why we need a peer-to-peer web at this point in our technology.
Andre Staltz: Yeah, so I think we're reaching a point with the internet where innovation is starting to die, I would say. There's a lot of services that are now just enough, like it's basically becoming enough to have just Google, Facebook, and Dropbox, and that's it. There's little innovation going on. I think we reached the peak, and it's not really evolving beyond that when it comes to the web. One of the things that was missed with the web from very early on is that it didn't guarantee peer-to-peer connections pretty well.
Andre Staltz: I mean, we only started having that with WebRTC, which is still not great. So, one of the good examples of the peer-to-peer web is the Beaker browser, and it's pretty awesome and also a pretty simple idea. It's just that your browser will communicate to other browsers directly, and there's no servers in between. They're just like finding each other through a so-called distributed hash table. It's the same idea as, let's say, BitTorrent. So, if you've ever seeded a large file, then imagine your browser seeding a website and then people can leech that website from you, and that's a very amazing idea because we never had that before.
Andre Staltz: Now, it also enables a bunch of innovation, and I think that's something quite awesome to explore given that ... There's a problem that the web is not innovating because of the tech giants. But there's also the problem that the tech giants are behaving like police nowadays in the sense that you can't watch something on YouTube if you're in a specific country, but if you're in another country, you can, and all kinds of things going on, which remove that freedom and that chaos that existed in the early internet and the early web.
Andre Staltz: That chaos created a lot of innovation, so I think it's becoming necessary to go back to that chaos period, that good type of chaos where we're just really trying to rebuild or come up with new ideas on top of this cool foundation.
Joel Hooks: Yeah, because it feels like the Facebooks and Google and these guys, they figured it out, right? They know what they're doing and how they're going to do it and how they're going to capitalize on the internet in its current state. It's understood and limited, and everybody understands the limits, but can we push that? And how do we push that? It's a really interesting question. I think in the next 10 years, it's going to go from question to reality to how it has to be.
Andre Staltz: Yeah, actually, one of the interesting things is that for the non-technical person, they are already living the peer-to-peer internet because they don't really ... The point is that they can do something from person to person, and they're already doing that just on a level that is not on the infrastructure. So, basically, something like Facebook basically hides as well as it can the idea that it's actually centralized. So it gives you this idea that you're connecting directly with your friends, and it tries to make the layer below it hidden or irrelevant.
Andre Staltz: So then people who are not technical already think that they're living the P2P web, but it's on a different level. So then there's always this hidden middleman in between, and we're starting to see the problems with that middleman. I think one of the nice ways of putting that problem is just basically you have a single point of failure, and that failure is usually described in distributed systems as just, "What if a server crashes?" But in this case, we're starting to see different types of failures, and one of them is basically a single point of abuse. Let's say some kind of election hacking.
Andre Staltz: So, it's like a single point of hacking elections basically or a single point of trusting the directors of that company who manage that thing. You see, it's a single point of whatever, and since we're living in a such huge planet with a lot of people, it's quite risky to have a single point of anything in that sense. So it's more safe in general also for the very long term. Let's say your children or children of your children live in a world where there's not a single point of whatever, you know?
Joel Hooks: Yeah, I mean, you notice that a lot, and I've seen more presses like the world's biggest online retailer. You can no longer trust their search results because they're pushing garbage, because people have figured out their system, and what happens when you can no longer ... You rely on this so much to run your household or whatever or your business. Then all of a sudden, it's garbage now just like overnight it feels, or our last election and Facebook and some of the press that's gone around that. If we put all our eggs in this one basket, then the whole thing is likely to just become problematic.
Andre Staltz: Yeah, and I mean, we could trust the directors of the company now. Maybe there are angels. Maybe there are great people that just want to do great stuff for us. But that's usually not the case in the very, very long term when they die, and who comes after that. I mean, we've seen this happen many times in history, let's say with dictators or kings. Maybe there was one king that was good, but then there's 10 kings that are not good after that, and that's basically what we have on the internet right now is basically have kings or emperors. So, it's not a good thing to keep.
Joel Hooks: Now, they're like little mini-governments in ways. It's not the exact [crosstalk 00:12:02]-
Andre Staltz: Yeah, yeah.
Joel Hooks: ... and they're not elected. They're more like dictatorships or some of them might be philosopher kings. I don't doubt that some of these folks are super smart and have made the world better in what they do, and they're not be trying evil, like they're not trying to do evil. But intentions versus what actually occurs is often different too, so it's slippery slopes. So the Beaker browser I think is interesting. I actually installed it and played around with it. What it looked like to me was a normal browser that I could just use, but it has some additional features. Is that true to say?
Andre Staltz: Mm-hmm (affirmative). Yeah, so Beaker browser, it basically works with HTTP and HTTPS, but it adds one more protocol, which is called Dat. So just like you have https: lala, you could also put dat: and et cetera. So Dat is basically like a magnet link of torrents. It's similar, but it's not torrent. It's like another thing, which is called Dat. Then you just start leeching that file, and it shows on your browser. So, basically, you could also just say, "Hey, I have a website here, and I want to start seeding it." So then your browser will start hosting it to the rest of the world.
Andre Staltz: It may seem that, okay, you may get a lot of people using your computer, like a lot of requests coming in. But due to the nature of peer to peer, the more people you have downloading your website, the more people you also have uploading your website. So, actually, it's the reverse effect of a distributed denial-of-service, because then it's a better case when there are many downloaders. So that's actually really nice, and Beaker is great for prototyping things, and they also announced that the Brave browser is about to support for that as well. It's not there yet, but Firefox whitelisted Dat, so that extensions could support caching that Dat link whenever you insert it and then loading that site.
Andre Staltz: So there's a lot of innovation going on in that space, and I'd recommend people to give it a try. One of the examples is actually an app called Fritter that only works on Dat, and it's basically like a Twitter clone. But the idea is that you download the front end just like you do with any other website. You also download JSON files that represent content, people, what people said, and news feeds, and you could fork the front end just in the browser. You could literally press the fork button, and then you have that in your computer. You can change how the front end looks. So then it's like a super easy way of customizing your own social network site.
Joel Hooks: Sounds like it might just be fun too just to hack on and explore these new technologies, which I think is a great thing to do too.
Andre Staltz: Yeah, so it's a really good space of innovation as well, and I think it's basically like front end on steroids, because it basically takes back end and says like, "Hey, nice to meet you," but now go alone, you know?
Joel Hooks: Yeah.
Andre Staltz: It's just front end talking to front end literally. There's nothing else.
Joel Hooks: That's cool.
Andre Staltz: It's awesome.
Joel Hooks: You said you spent 30% of your time on a library called Cycle.js, which is a stream-based reactive and functional programming framework.
Andre Staltz: Framework.
Joel Hooks: Yeah, so how's Cycle going and why should people check it out?
Andre Staltz: So Cycle is going good. We're three people in the core team. Me, Jan, and Nick, and we're constantly improving the framework. It's like bug fixes and that type of stuff. But recently ... Well, not that recently, but in the last CycleConf, we sat down, and we talked about how to support push and pull. It's like a new big thing that's coming. It was like really long discussions on how to support different types of, let's say, data sources, and we're getting there.
Joel Hooks: What does push and pull mean just in terms of ... What is push and pull?
Andre Staltz: Push is like a type of data source that comes by itself. Let's say like click events and that kind of stuff. Everything that RxJS covers is basically push. Then pull is basically like passive data sources that just sit there, waiting for you to do stuff, and JavaScript already has a lot of pull data sources, and those are the iterables. So let's say whenever you create a generator, it's basically just sitting there, waiting for you to get more data out of it, and then it resumes that stuff, and it gives you data.
Andre Staltz: But it's just there sitting. You can't actually write a generator that will give you click events, let's say.
Joel Hooks: Right.
Andre Staltz: So then there's these two different worlds, two different types of data sources, and then we're thinking how to support properly the pull type of data source. It's more like a corner case that we want to support, but it's something big, and it's making us think how to support this properly. Then we also thought of how to support WebWorkers in the middle because we're doing a re-architecture of the framework, and then we basically found like, "Hey, this is actually a good case for just making things work with WebWork really easily."
Andre Staltz: So you can say you run that part in WebWork or run that other part in WebWork because everything is message passing. So, WebWorkers were a good fit for that. So yeah, there's a lot of progress going on, and we're just doing work as we can. All of us three in the core team have jobs, and we basically also work part-time on Cycle. That's kind of the deal.
Joel Hooks: Yeah, I've always said that there's two types of developer on the planet, and there's a lot more, but those that will write a framework and those that will never write a framework. I'm definitely in the never camp. It wouldn't occur to me, but it's really great, and I love that folks like you aren't satisfied. You probably could be satisfied if you really wanted to be but aren't satisfied with the status quo, like it isn't ... I don't get the impression like you think react is really horrible. But it's just not how you want to develop your application, so you give us this amazing tool, and Cycle is really, really great.
Andre Staltz: Yeah, sure. So, I believe I'm an idealist in the sense that I'm really not satisfied with how things are. But it's not also, like I'm not angry at the things how they are. I'm just thinking like, "Okay, this is good, but how can we make things better?" So I'm basically, like I don't think I see things how they are. I think I see things how, like where they're going. So if some tool is going somewhere, then I judge it basically on where it's going and not like where it is. I think that's a different way of thinking.
Andre Staltz: I thought it was normal, but then you encounter people, and you see that, "Okay, people think in a different way." But yeah, I think I ended up building libraries and frameworks just because I was always trying to find that ideal thing. I was imagining this ideal thing and trying to bring it to today. That's what I want.
Joel Hooks: Yeah, and it's true even not just for other people's frameworks, but it's your own tool kits that you build, because you released xstream, which is a reactive programming stream management library. Then last week, you announced Callbags, which is another ... Feels similar, but it's different, and I was wondering Callbags actually looks really cool, and I looked at it and it was probably ... In terms of readability and the ability to understand what I was looking at, it really jumped out to me to be simpler, I guess maybe, than RxJS or other kind of libraries. So why does Callbags exist? I guess is my question.
Andre Staltz: So, Callbags was like an accidental discovery. As I told, like we're planning some things for the Cycle.js framework, and we wanted ... So, Cycle.js uses as an engine the xstream library. So, engine, as in everything, is stream based, but we also want to support RxJS and stuff. So we used to have RxJS inside the framework, and RxJS is big, which is okay. But as an engine for a framework, we want to keep it very small, and then we made xstream has helped us quite a lot. But it's still not small enough, and it has these particularities.
Andre Staltz: So then I started thinking like, "Okay, let's just use callbacks just to send stuff back and forth," just message passing, and that's what I did. But I also wanted to support just like RxJS has next, error, and complete. I want to support those. I just put a number in front of it. Then I started making this abstraction just basically Callbags, but it's basically streams. Then I just said, "You know what? I'll just make a small library and see what happens." Then I was actually quite happy with it, and yeah. So, I think it's really good as a foundational thing.
Andre Staltz: So for instance for frameworks or libraries, I think it's better to use Callbags because it's not even like a core library. Because when you look at it, it's just a bunch of utilities, and there's no core around those. I mean, there's no core where the utilities are surrounding. So it's just like a spec, you could say. So, then I just decided to know open source, the spec, or basically just write it down and put it as public creative commons public stuff, and that's it. It's a spec. So, if people want to follow ... Like if you want to play by these rules, then you're going to be in the game called Callbags. That's the idea.
Joel Hooks: Thought it looked like a great teaching tool as well, like just because of at its core, right? Like in it. Sometimes, these concepts are difficult for people that if we can simplify it, that's always a great thing. John and I have talked a lot of times that Cycle in itself seems like a good way to teach the principles of front end and JavaScript in terms of ... Just like giving people this core functional experience and put them in a container. Always thought that was a solid thing to do.
Andre Staltz: The interesting thing is that I've been also teaching RxJS, and also Ben has been teaching, Ben Lesh. When we're teaching, we're usually going through this process where we start with almost nothing, and then we add a function, and then this function is called subscribe. That takes an observer. At some point, we just say, "Okay, but we're going to put all of this inside an object." Then you get to RxJS API. So, I've done this quite a lot in conferences and workshops. The other day I just thought, "Why am I putting it in an object?"
Andre Staltz: I'm doing that just in order to get to the RxJS API. But what if I would just not put it in an object and just keep it as a function? Then I came up with the Callbags specific, and it turns out that it can do everything that RxJS can do. So I was like, "Huh, this is interesting." So, for now, I'm actually not using Callbags in anything big, but we will use it soon in Cycle.js as the new engine basically. So it's just callbacks. It's very light, very small, and there's basically no dependency that that's one of the things.
Andre Staltz: So, for now, xstream has been the only dependency that is in every Cycle app, and it would be nice if we wouldn't have that. So, it's just like-
Joel Hooks: Just some utility functions, right? Like at the end of the day.
Andre Staltz: Yeah.
Joel Hooks: Well, Andre, it was great to catch up with you. I'm looking forward to see what you come up with next, and we'll talk soon.
Andre Staltz: Yeah, cool. This was great, and thank you.