By Ev Bogue - January 4th 2017
Four months ago I got it into my head that I was going to live with
npm for month. I lasted for 4 days, and managed to piss off
npm's lawyer. Although I'm sure he'll say I didn't get to him. It's cool, we down now ke.
I wrote a piece to sbot that said 'imagine if
npm went down tomorrow' and that started the conversation. The thread is available on ssb-viewer if you want to read it.
The reason I decided to try to live without
npm was simple. I'm working on a distributed social network, built with Node. We've got a distributed version of FB. We've got a distributed version of Twitter. We've got a distributed version of Github. But we're relying on
npm to install our packages. What if
npm goes away? Or doesn't like me anymore? What will we or I do?
We will all be fucked, that's what. Because after 4 days of trying to live without
npm I was completely unable to install any Node software. I had to cheat more times than I'm willing to admit to get even the most basic of upgrades going.
When you're working with distributed technologies, the most important thing to keep in mind is not relying on a centralized authority. Instead, the goal is to be distributed. We want everyone to be able to install our software whether or not
npm is around in a year or five. Or whether or not
npm loves us.
Now, I know
npm is doing their best. They've put in a lot of hours. They've been a pivotal part of the Node world for years. However, the pivotal part is the problem when we're trying to create distributed technologies.
Here's what I mean when I say distributed:
Distributed technologies are not reliant on one single server. Instead, they share data between many nodes making it available to everyone. In the specific case of ssb we are replicating the data of our friends, and our friends friends on our computers so that it is available when our crappy wifi is down, and in the event of a nuclear war -- even though that'll (probably) never happen.
Ok, so how are we going to live without
This is the problem I faced 4 months ago when I decided to try to live without
npm. My first thought was "I'll just clone the individual git repos down from wherever and build them" but it turns out that every Node module has dependencies of dependencies of dependencies, and thus it devolves into this recursive problem of needing a tool to handle your install for you. It's not dependency hell, it's only dependency purgatory because you'll be installing your dependencies until you're old and frail.
Being a fairly terrible coder, I had no idea how to write this tool myself. Now, everything would be alright if
npm would cache your dependencies in your project folder and install from there. But it turns out that
npm always wants to phone home before installing. This makes you reliant on a centralized authority.
Now many people said 'oh, just use an
npm mirror.' But that's also remote (so it won't work when you're offline), and I also discovered that all of the
npm mirrors are down! Which makes it difficult to use them. While
npm is nice enough to let you mirror all of the data they host, the trouble is that you have to mirror all of their data -- which according to their website is over 350,000 modules. I don't want host all that data. I just want to host the data to install a few distributed applications.
So I failed. I gave up. I went back to using
npm on day 5, because how the hell was I going to solve this problem?
But it turns out that another organization has this same problem too. And it's not the kind of organization that you'd expect to want to encourage everyone to distributed everything... Facebook.
I know, I hate it. It's horrible. But you know how tech companies, before they go under, have guys at them with actual skills who want to get jobs after the bust? Well, a few of those guys are working on a cool project. It's called Yarn. It's all sweet and shit. There's a cat silo. The goal of their project is not to kill
npm, it's just to solve their problem -- which is they are not allowed to use
npm because they are using packages that they don't want other people to see. You know, all of those packages with human manipulation algorithms built into them. Which is to say the interface for all of those old ladies sitting in a back room hitting up/down on your Facemash posts.
So it turns out that Facebook and distributed social have the same problem. We don't want to depend on centralized technologies.
However our usecase is very different. Facebook wants to keep everything secret, and we want to keep everything available to everyone.
Yarn is pretty cool. It's a tool very similar to
npm. But instead of relying on a centralized authority, you can instead tell it to install your node dependencies from another source. Yarn also de-dupes packages faster than
npm and uses checksums to guarantee file integrity -- so no one can slip malicious code into your dependencies.
In our case, we want to install our packages from a distributed social network. The easiest way to do this right now is commit our dependencies using
git-ssb, clone them down, and install them using yarn.
To do this, reading this article on creating an offline mirror with yarn is very interesting.
To create an offline mirror of your dependencies, all you gotta do is create a
.yarnrc file that says you want to save tarballs of your dependencies to your project folder.
Here's the one for minbay.
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY. # yarn lockfile v1 yarn-offline-mirror "./ssb-pkgs"
I generated this by running the command
yarn config set yarn-offline-mirror ./ssb-pkgs, and then moved the generated
.yarnrc file into my project folder.
yarn, which will download all of your dependencies from
npm or the
yarn mirror, save them to your
node_modules folder in a familiar fashion, and generate a
yarn.lock file which tells yarn that you've done so.
yarn install is over, you'll see a folder called
ssb-pkgs which contains all of the
node_modules in tarball form. You can check this into a git repo, push them to
git-ssb and they will be available over the distributed social network.
Instead of the install flow looking this way:
git clone git://project cd project npm install npm start
Your project will look more similar to this
git clone ssb://project project cd project git clone ssb://project-dependencies ssb-pkgs yarn yarn start
They key difference is the second workflow works completely offline and across a distributed social network. So you don't have to rely on anyone but your friends to get amazing software installed. The added complexity can be easily automated in your
package.json install script.
Instead of installing
npm, you need to install
git-ssb -- and make sure that you're peered into the network.
The other cool thing about Yarn is if you're not on ssb, you can fall back to a traditional mirror which will install your dependencies from a remote.
Now that we're no longer dependent on
npm, we can live free on the Internet knowing that no centralized authority can ever run out of startup funding and leave us hanging. Hurray!
Feel free to comment below using sbot on the original 'Yarn works now' thread: