Evbogue.com


IPFS for package management (ipm.js the ghetto way)

By Ev Bogue - June 22nd 2016

Experiments in distributed Node.js package management

If you've ever programmed with Node.js with an inconsistent Internet connection, you know the pain of typing npm install and watching your system hang until you can grab the bandwidth to download all of the packages.

Not to mention the latest npm scandal that revolved around someone deleting all of their packages because of a tiff, and thus making it difficult for upstream package maintainers to download the dependencies for their projects.

Npm is centralized, so we must consider all of the concerns that come from using centralized services. If the ecosystem of Node.js modules is to be resiliant a distributed solution for Node.js packages is needed.

Because of the latest npm drama, many developers were calling for a distributed immutable package manager for Node.js. A few projects got started, but no one coded on them.

Why I'm interested in distributed immutable package mangers for Node.js: I want to store Node.js packages on my local, so when I don't have Internet or I have fritzy second world Internet, and I can still install packages on the fly when I'm coding on Node.js projects.

IPFS The Interplanetary File System may be one way to solve this problem. The IPFS team has already developed a package manager that is used to build IPFS from source, it's called gx.

Why not use ipfs for Node packages?

The most obvious reason why not is that gx isn't quite ready to use with Node.js. No one has built the required driver to use gx to create and install Node packages.

But then I got to thinking, isn't there an easier way of doing this?

So, last week at some point I did a weird thing. I typed

% ipfs add -r site

which produced this hash

Qmd5P6KYHWA5prWJ2q7EsEePgLiFdPwz6cxuXhbXaWHzGj

I added my entire project to IPFS, node_modules folder and all.

Adding my entire node_modules folder took a little time, but only on the first try. When a file is added once to IPFS, it's cached until you remove it from the local IPFS datastore.

Once I did this, I went a little crazy and went around adding all of the node_modules folders for all of the Node.js projects on my computer.

After the first add, everything went a lot faster, because there is a great deal of overlap in the number of Node modules that I use on projects. For example, I always find myself using metalsmith koa and jade on projects, with all of their associated dependencies.

Now, this is a very ghetto way to bootstrap a package manager. I realize that it might not work at all for Node.js packages that contain programs written in anything other than JavaScript. Out of the box, just added files to IPFS does you no good if you want to download all of the associated dependencies of dependencies (and dependencies of dependencies) that are required by most Node.js packages.

I think by Merkle DAGing my projects I'll get a better idea of what kind of tooling is needed to create a immutable package manager for JavaScript.

In addition, it was trivial to replace the frontend two dependencies that I was bringing in using Bower.

Once I added the dependencies to IPFS on my local, and then ipfs pin added them, all I had to do was add

ipfs get --output=build/open-sans 'QmfQyxtcLcsKoDNMFrdc4QWpMe3iZCX2tDr5E9HnUWNa7K'
ipfs get --output=build/reserva.css 'QmcqpfFijmC6SNq5WzyimU2xs5yB9E3NenTQryU9svk5yD'

to my build instructions.

Anyway, all of this is an experiment. npm fanboys everywhere, don't hate. There is a lot of work to do before the Node.js community has a working distributed immutable package manager. npm won't be replaced overnight.

But in the meantime, consider Merkle DAGing your whole project. You may find the experience slightly exhilerating.

If you have ideas about how to improve this workflow, or projects I should look at, email me

And if you're using IPFS, feel free to pin my website using this IPNS hash: QmPGW3d31TKTyktoVGGeQRdyh61LRi2uKokFqrKyzqMG4T

At the event horizon of the Immutable Web →

← A simple guide to creating Jade layouts

git

This is Decent: ?


about | blog | ssb | decent | ev@evbogue.com | gitmx