Evbogue.com


How to live without npm (using yarn for distributed package management)

By Ev Bogue - January 4th 2017



Four months ago I got it into my head that I was going to live with npm for month. I lasted for 4 days, and managed to piss off npm's lawyer. Although I'm sure he'll say I didn't get to him. It's cool, we down now ke.

I wrote a piece to sbot that said 'imagine if npm went down tomorrow' and that started the conversation. The thread is available on ssb-viewer if you want to read it.

The reason I decided to try to live without npm was simple. I'm working on a distributed social network, built with Node. We've got a distributed version of FB. We've got a distributed version of Twitter. We've got a distributed version of Github. But we're relying on npm to install our packages. What if npm goes away? Or doesn't like me anymore? What will we or I do?

We will all be fucked, that's what. Because after 4 days of trying to live without npm I was completely unable to install any Node software. I had to cheat more times than I'm willing to admit to get even the most basic of upgrades going.

When you're working with distributed technologies, the most important thing to keep in mind is not relying on a centralized authority. Instead, the goal is to be distributed. We want everyone to be able to install our software whether or not npm is around in a year or five. Or whether or not npm loves us.

Now, I know npm is doing their best. They've put in a lot of hours. They've been a pivotal part of the Node world for years. However, the pivotal part is the problem when we're trying to create distributed technologies.

Here's what I mean when I say distributed:

Distributed technologies are not reliant on one single server. Instead, they share data between many nodes making it available to everyone. In the specific case of ssb we are replicating the data of our friends, and our friends friends on our computers so that it is available when our crappy wifi is down, and in the event of a nuclear war -- even though that'll (probably) never happen.

Ok, so how are we going to live without npm?

This is the problem I faced 4 months ago when I decided to try to live without npm. My first thought was "I'll just clone the individual git repos down from wherever and build them" but it turns out that every Node module has dependencies of dependencies of dependencies, and thus it devolves into this recursive problem of needing a tool to handle your install for you. It's not dependency hell, it's only dependency purgatory because you'll be installing your dependencies until you're old and frail.

Being a fairly terrible coder, I had no idea how to write this tool myself. Now, everything would be alright if npm would cache your dependencies in your project folder and install from there. But it turns out that npm always wants to phone home before installing. This makes you reliant on a centralized authority.

Now many people said 'oh, just use an npm mirror.' But that's also remote (so it won't work when you're offline), and I also discovered that all of the npm mirrors are down! Which makes it difficult to use them. While npm is nice enough to let you mirror all of the data they host, the trouble is that you have to mirror all of their data -- which according to their website is over 350,000 modules. I don't want host all that data. I just want to host the data to install a few distributed applications.

So I failed. I gave up. I went back to using npm on day 5, because how the hell was I going to solve this problem?

But it turns out that another organization has this same problem too. And it's not the kind of organization that you'd expect to want to encourage everyone to distributed everything... Facebook.

I know, I hate it. It's horrible. But you know how tech companies, before they go under, have guys at them with actual skills who want to get jobs after the bust? Well, a few of those guys are working on a cool project. It's called Yarn. It's all sweet and shit. There's a cat silo. The goal of their project is not to kill npm, it's just to solve their problem -- which is they are not allowed to use npm because they are using packages that they don't want other people to see. You know, all of those packages with human manipulation algorithms built into them. Which is to say the interface for all of those old ladies sitting in a back room hitting up/down on your Facemash posts.

So it turns out that Facebook and distributed social have the same problem. We don't want to depend on centralized technologies.

However our usecase is very different. Facebook wants to keep everything secret, and we want to keep everything available to everyone.

Yarn is pretty cool. It's a tool very similar to npm. But instead of relying on a centralized authority, you can instead tell it to install your node dependencies from another source. Yarn also de-dupes packages faster than npm and uses checksums to guarantee file integrity -- so no one can slip malicious code into your dependencies.

In our case, we want to install our packages from a distributed social network. The easiest way to do this right now is commit our dependencies using git-ssb, clone them down, and install them using yarn.

To do this, reading this article on creating an offline mirror with yarn is very interesting.

To create an offline mirror of your dependencies, all you gotta do is create a .yarnrc file that says you want to save tarballs of your dependencies to your project folder.

Here's the one for minbay.

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


yarn-offline-mirror "./ssb-pkgs"

I generated this by running the command yarn config set yarn-offline-mirror ./ssb-pkgs, and then moved the generated .yarnrc file into my project folder.

Next, type yarn, which will download all of your dependencies from npm or the yarn mirror, save them to your node_modules folder in a familiar fashion, and generate a yarn.lock file which tells yarn that you've done so.

Once the yarn install is over, you'll see a folder called ssb-pkgs which contains all of the node_modules in tarball form. You can check this into a git repo, push them to git-ssb and they will be available over the distributed social network.

Instead of the install flow looking this way:

git clone git://project
cd project
npm install
npm start

Your project will look more similar to this

git clone ssb://project project
cd project
git clone ssb://project-dependencies ssb-pkgs
yarn
yarn start

They key difference is the second workflow works completely offline and across a distributed social network. So you don't have to rely on anyone but your friends to get amazing software installed. The added complexity can be easily automated in your package.json install script.

Instead of installing npm, you need to install yarn and git-ssb -- and make sure that you're peered into the network.

The other cool thing about Yarn is if you're not on ssb, you can fall back to a traditional mirror which will install your dependencies from a remote.

What now?

Now that we're no longer dependent on npm, we can live free on the Internet knowing that no centralized authority can ever run out of startup funding and leave us hanging. Hurray!

Feel free to comment below using sbot on the original 'Yarn works now' thread:


Eight years (is a long time) →

← Eliminating email in 2017?


about | blog | ssb | decent | ev@evbogue.com | gitmx