#+TITLE: floco

=floco= is a JavaScript package management and build tool powered by
[[https://nixos.org][Nix]].

=floco= is a bold departure from conventional JavaScript package management
tooling focusing on reproducibility, distributed caching, and sandboxing.

Every package is built in a strictly declared sandbox isolated from the runtime
system, much like an /OCI/ container.
This approach allows packages to be built once and reused on any system that
shares the same architecture and platform.
Artifacts are cached locally and may be distributed among a cluster of systems
allowing development environments to be created at more than twice the speed of
=npm= or =yarn=.

Underlying installers are implemented in =bash= and limit themselves to using
=coreutils=, =findutils=, and =jq= improving reproducibility and readability
compared to tooling implemented using a sprawling maze of JavaScript.

Despite this repository's seemingly small form, it is the result of nearly
a year of exploration, trial, and refinement.
A great deal of effort was expended to make this piece of software
/suck less/ than the competition, but rest assured that these routines were
not implemented naively.
Writing a JavaScript package management framework in =bash= and =nix= was
done /not because it is easy/ - on the contrary it was fucking hard.


* Getting Started

There's a dedicated
[[https://github.com/aakropotkin/floco/blob/main/doc/guides/basics.org][Getting Started]]
guide that is the best place to dive in.

For projects that depend only on registry packages without =install= scripts that require
native dependencies or cycle breaking,
the following process will have you up and running:
#+BEGIN_SRC shell
set -euo pipefail;
cd ./my-project;
nix flake init -t github:aakropotkin/floco;
nix run github:aakropotkin/floco -- translate;
[[ -r ./package-lock.json~ ]] && mv ./package-lock.json{~,};
nix build -f ./. global;
ls -R ./result;
#+END_SRC


** Documentation

A collection of documentation is hosted on GitHub on the project's
[[https://github.com/aakropotkin/floco/wiki][wiki]] tab, and is also
available under
[[https://github.com/aakropotkin/floco/blob/main/doc][<floco>/doc]]
alongside many of the workspace directories used in examples.

Additionally all CLI tooling and scripts support the =--help= option.
The =floco help CMD= sub-command may also be used to view "help" messages
for a given =CMD=.


** Templates

A simple template with the boilerplate needed for use with our /updaters/
is available through ~nix flake init -t github:aakropotkin/floco~.

More templates are on the way.


* CLI

The =floco= CLI interface is currently under active development and is
expected to change rapidly in the near future.


** Common Behaviors

Across the =floco= CLI a few behaviors are consistent. 

The following config files are included/applied if they exist:
- ~/etc/floco/floco-cfg.{nix,json}~
- =${XDG_CONFIG_HOME:-$HOME/.config}/floco/floco-config.{nix,json}=
- "Local" =floco-config.{nix,json}= searched for between =PWD= and git
  project root, or =/= if =PWD= is not a =git= repository checkout.

References to =floco= will be pulled from the nearest =flake.lock=, or
=nix registry list=, using =github:aakropotkin/floco/main= as a fallback.


** =floco translate=

Generate a =pdefs.nix= file from a =package[-lock].json= or
registry package.

This routine utilized =npm= internally to resolve packages and form
=node_modules= trees.


*** Local Project Example

#+BEGIN_SRC shell
mkdir -p /tmp/foo;
pushd /tmp/foo;
echo '{
  "name": "@floco/phony",
  "version": "4.2.0",
  "dependencies": {
    "lodash": "^4.17.21"
  },
  "scripts": {
    "build": "touch ./built"
  }
}' > ./package.json;
nix shell github:aakropotkin/floco;
floco -- translate -pt;
nix flake init github:aakropotkin/floco -t;
floco build;  # or `nix build -f ./. global;
ls ./result/lib/node_modules/@floco/phony/;
#+END_SRC


*** Remote Project Example

#+BEGIN_SRC shell
mkdir -p /tmp/foo;
pushd /tmp/foo;
nix shell github:aakropotkin/floco;
floco -- translate -pt lodash@4.17.21;
nix flake init github:aakropotkin/floco -t registry;
echo '{ ident = "lodash"; version = "4.17.21"; }' > ./info.nix;
floco build;  # or `nix build -f ./.;'
ls ./result/lib/node_modules/lodash/;
#+END_SRC


** =floco list=

List all declared projects by "key", being =<IDENT>/<VERSION>=, such as
=@foo/bar/4.2.0= or =baz/4.2.0=.

#+BEGIN_SRC shell :exports both :results output
nix run github:aakropotkin/floco -- list;
#+END_SRC

#+RESULTS:
#+begin_example
@webassemblyjs/wast-printer/1.9.0
@xtuc/ieee754/1.2.0
@xtuc/long/4.2.2
abab/2.0.6
abbrev/1.1.1
abbrev/2.0.0
abort-controller/3.0.0
accepts/1.3.8
acorn/6.4.2
acorn/7.4.1
acorn/8.8.2
acorn-globals/4.3.4
acorn-jsx/5.3.2
#+end_example


** =floco show=

Print the =pdef= record for a given package.

#+BEGIN_SRC shell :exports both :results output
nix run github:aakropotkin/floco -- show lodash@4.17.21 --json;
#+END_SRC

#+RESULTS:
#+begin_example json
{
  "ident": "lodash",
  "version": "4.17.21",
  "ltype": "file",
  "fetchInfo": {
    "narHash": "sha256-amyN064Yh6psvOfLgcpktd5dRNQStUYHHoIqiI6DMek=",
    "type": "tarball",
    "url": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz"
  },
  "treeInfo": {}
}
#+end_example


* Core Scripts

=floco= uses a small collection of =bash= scripts to perform install tasks
and drive builds.

These scripts do not depend on Nix, and are suitable for standalone use
as replacements for ~pacote extract~ ( =install-module.sh= ) and
~(npm|yarn) run~ ( =run-script.sh= ).

You can install these as standalone executables using the =floco-utils=
installable in the top-level flake.

For example usage and more details please see
[[https://github.com/aakropotkin/floco/blob/main/doc/scripts/README.org#core-scripts][Core Scripts]].


* Updaters/Generators

The top level flake provides an installable =floco-updaters= as well as
=app= targets ( =fromPlock= and =fromRegistry= ) that can be used to generate
=pdefs.nix= and =pdefs.json= files to be loaded by as configs.

These scripts will allow you to convert existing JavaScript projects to be
used with =floco=, and update/regenerate configs as projects'
dependencies and build requirements change.


** =fromPlock=

This generator is intended for use with local projects.
It is essentially a wrapper around =npm i --package-lock-only=.

For example usage please see the
[[https://github.com/aakropotkin/floco/blob/main/doc/guides/basics.org][Getting Started]]
guide.


** =fromRegistry=

This generator is intended for use with published registry packages that
you'd like to make accessible to =floco= and =nix=.

This script behaves almost identically to =fromPlock=, except that it
ignores =devDependencies= entirely, and accepts package descriptors as an
argument ( as =npm= or =yarn= would ).

For example usage please see the
[[https://github.com/aakropotkin/floco/blob/main/doc/guides/native-deps.org#preparing-a-workspace][Native Dependencies]]
guide.

This script most useful for packaging executables and generating =treeInfo=
information for packages that have =install= scripts ( such as =node-gyp=
compilation ).


* Modules

Package metadata collection, also called /translation/, and project
composition is managed using
[[https://github.com/NixOS/nixpkgs/blob/master/lib/modules.nix][Nixpkgs Modules]]
similar to those used by
[[https://nixos.org/manual/nixos/stable/#sec-writing-modules][NixOS]],
[[https://github.com/nix-community/dream2nix][dream2nix]], or
[[https://github.com/nix-community/home-manager][home-manager]].

These modules are organized as sets of =interface.nix= and
=implementation.nix= files and are designed to be extensible.


** Organization

The core of the module system revolves around a record called =pdef=, short
for "package definition", which organizes translated metadata, and
=package= records which organize the build pipelines.

This separation simplifies the organization of the /translation/ and
/builder/ APIs, but the rationale runs further.
The split allows us to flatly state: build routines must never perform
impure operations, and translation routines must only produce fields that
can be serialized to JSON.

Serialization of translated metadata allows Nix's =flake= features to
drastically improve performance by leveraging
[[https://www.tweag.io/blog/2020-06-25-eval-cache/][eval caching]] to avoid
re-evaluation of recipe generation on successive runs.


** =pdef= Package Definitions

The =pdef= record closely mirrors the pseudo-standard schema used by most
=package.json= files; but is much stricter about how declarations
are written.

If desired, users could ditch =package.json= files altogether and simply
write =pdef= records for their projects.


** Translators

At time of writing only a few translators have been migrated from the alpha
iteration, [[https://github.com/aameen-tulip/at-node-nix][at-node-nix]], but
in the near future these will be finalized for production use.


*** =package.json=

This is our bread and butter, and serves as the default implementation for
creating a =pdef= record.

On its own this translator would require users to explicitly declare the
structure of their =node_modules/= tree using the =treeInfo= submodule.
For this reason we strongly recommend using the =package-lock.json=
translator for projects with large dependency graphs.


**** Progress on /Ideal Tree/

The term /ideal tree/ refers to the mapping of a =node_modules/= tree
from a dependency graph.
This process is by far the most complex and challenging aspect of
Node.js package management.

While =floco= currently relies on =npm= to generate /ideal trees/, this
is expected to end soon.

The alpha repository
[[https://github.com/aameen-tulip/at-node-nix][at-node-nix]] contains a
large body of routines to perform /best effort/ =treeInfo=
mapping, specifically handling projects which only require a single
version of any package ( this property is called /The Golden Rule/ in
package management contexts ).

Additionally the
[[https://github.com/aameen-tulip/at-node-nix/blob/main/lib/sat.nix#L372][semver resolution]]
routines used to fetch closures of /packument/ records effectively solve
half of the /ideal tree/ process, leaving only scope and /follows/
management to be completed.


*** =package-lock.json= v2/v3

This is by far the most developed translator, and is the recommended
option for large projects.

This translator will automatically fill =treeInfo= submodules, and
triggers minimal network fetching.


*** =yarn.lock= v5

A rudimentary translator exists to collect information from =yarn.lock=
v5 ( produced by =yarn= v3 ), but because these lockfiles lack
/ideal tree/ information users will need to provide =treeInfo= themselves.

In the future we intend to produce =treeInfo= from these locks using
the pinned version information they contain; but this routine still needs
to be authored.


* Experimental Features

** =treeFor=

A CLI frontend for the =npm= /ideal tree/ routine,
[[https://github.com/npm/cli/blob/main/workspaces/arborist/README.md][arborist]],
modified such that =package-lock.json= files can be emitted to =STDOUT=
without modifying the project.

This is expected to be used in later iterations of the /updaters/ allowing
them to be run on ~/nix/store/~ paths.
The =builtins.npmLock= example in the section takes advantage of this.

This executable is exposed as an installable and =app= in the
top-level flake.


** Nix Plugin

A =nix= plugin for use with ~nix --plugin-files ...~ is available in the
top level flake, along with a wrapper executable, =floco-nix=, which
automatically loads this plugin.

In the future this plugin is expected to grow into a full executable that
provides a suite of CLI commands; but for now it accepts =nix= arguments
and sub-commands.

This plugin was developed for Nix v2.12.0, but is likely compatible with
a wider range of versions.


*** New Builtins

Our plugin adds a few new =builtins= to the =nix= evaluator which are
useful for dynamically generating package definitions.

**** =builtins.npmShow=

Wraps ~npm show~ allowing Nix to query package registries using a users
existing =npm= config and any environment =NPM_CONFIG_*= variables.

While =floco= is already able to fetch package registry information
without any external tools; this builtin is useful for accessing private
package registries and inheriting authorization settings with
minimal setup.

#+BEGIN_SRC shell :results output :exports both
nix run github:aakropotkin/floco#floco-nix -- eval --json --expr '
builtins.attrNames ( builtins.npmShow "lodash" )
'|jq;
#+END_SRC

#+RESULTS:
#+begin_example json
[
  "_cached",
  "_contentLength",
  "_hasShrinkwrap",
  "_id",
  "_nodeVersion",
  "_npmOperationalInternal",
  "_npmUser",
  "_npmVersion",
  "_rev",
  "author",
  "bugs",
  "contributors",
  "description",
  "directories",
  "dist",
  "dist-tags",
  "gitHead",
  "homepage",
  "icon",
  "keywords",
  "license",
  "main",
  "maintainers",
  "name",
  "readmeFilename",
  "repository",
  "scripts",
  "time",
  "users",
  "version",
  "versions"
]
#+end_example


**** =builtins.npmResolve=

Resolves package descriptors such as =foo@^1.0.0= or =bar@latest=
using =npm=, returning a resolved URI.

This has the same environment and configuration properties as =npmShow=.

NOTE: if you use ranges such as =lodash@2.x= you will want to use
=builtins.split= to parse the output.

#+BEGIN_SRC shell :results output :exports both
nix run github:aakropotkin/floco#floco-nix -- eval --expr '
builtins.npmResolve "lodash@latest"
';
#+END_SRC

#+RESULTS:
: "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz"


**** =builtins.npmLock=

Produces a virtual =package-lock.json= for a given project path
without modifying the project or making any writes to the filesystem.

This is an ideal alternative to the =fromRegistry= /updater/ when
used in combination with =builtins.fetchTree= and =builtins.npmResolve=.

In practice you can dynamically generate full dependency closures'
=treeInfo= records using this routine.
I currently use it for this purpose out in the field; but have avoided
using it in the default modules so that they are usable without plugins.

#+BEGIN_SRC shell :results output :exports both
nix run github:aakropotkin/floco#floco-nix -- eval --impure  \
  --expr 'let
    url   = builtins.npmResolve "pacote@latest";
    src   = builtins.fetchTree { type = "tarball"; inherit url; };
    plock = builtins.npmLock src;
  in builtins.attrNames plock
';
#+END_SRC

#+RESULTS:
: [ "lockfileVersion" "name" "packages" "requires" "version" ]


**** =builtins.semverSat=

Runs =node-semver= to test whether a semantic version satisfies
a constraint.
In the future =node-semver= will be replaced using a native C++ port
[[https://github.com/aakropotkin/semi.git][semi]].

This largely exists as a stop-gap until the pure =nix= implementation
from the alpha repository is polished and/or =semi= is completed.

#+BEGIN_SRC shell :results output :exports both
nix run github:aakropotkin/floco#floco-nix -- eval --expr '[
  ( builtins.semverSat "^4.2.0" "4.0.0" )
  ( builtins.semverSat "^4.2.0" "4.2.0" )
  ( builtins.semverSat "^4.2.0" "4.2.1" )
  ( builtins.semverSat "^4.2.0" "4.3.0" )
]
';
#+END_SRC

#+RESULTS:
: [ false true true true ]


* Future Extensions

Many of the following extensions have function drafts or well tested
prototypes in the alpha release of =floco=; but are not developed enough for
use in production code-bases as pieces of reliable infrastructure.

- Improved support for package.json workspaces.
  + Currently reliance on =npm= and special configuration based on in depth
    knowledge of =floco= is necessary to accomplish workspace support.
  + Practically a template or example using workspaces is likely sufficient
    for the immediate future; but the NixOS Module system is expected to
    resolve issues that previously made workspaces complex to manage.
- Expanded CLI tooling.
  + Currently users are asked to interact with nix to drive builds, tests,
    update metadata, etc.
    Ideally a simple bash script would provide familiar commands such as
    ~floco add <PKG>~, ~floco publish~, ~floco update~, ~floco build~,
    etc that =npm= and =yarn= users are already familiar with.
- Nix plugin to read/write caches globally and into =flake.lock=.
  + This is the real end goal for =floco=.
    It should be possible to read/write =floco= metadata to =flake.lock= and
    existing =nix= caches.
  + There is currently a draft plugin which allows nix to adopt =npm= URIs to
    refer to packages as =lodash@4.17.21= which could be expanded upon.
  + Project templates and propagation of build recipes could allow =nix= to
    abstract away the generation of =flake.nix= for the vast majority of
    projects which would be a significant UX breakthrough.
- Semantic version parsing, and /ideal tree/ formation.
  + Currently =floco= really relies on =npm= and its =package-lock.json= to
    construct non-trivial node_module/ metadata declarations.
    This reliance is a major pain point for handling projects which currently
    use yarn since interoperability between =yarn= and =npm= across their
    associated lockfiles is implemented incredibly poorly, to such a degree
    that you cannot trust them to behave predictably in the same source tree.
  + Semver parsing and solving SAT is implemented in the alpha repository, and
    has been testing on large non-trivial inputs quite successfully.
    Still this effort requires a few weeks of polishing to really approve for
    use in production.
    - For now we have provided
      [[https://github.com/npm/node-semver.git][node-semver]] as an
      installable in the top-level flake for use in scripts and our
      [[https://github.com/aakropotkin/floco/blob/main/pkgs/nix-plugin][floco-nix]]
      through =builtins.semverSat=.
  + Construction of ideal tree from semver SAT is a project in and of itself
    in order to support things like =optionDependencies=, =peerDependencies=,
    =bundledDependencies=, and other oddballs which are a prerequisite for use
    in the general case.


* Community

** Matrix
Sadly IRC is dead. IRC remains dead.
So like most folks these days we use Matrix Chat.

Space: [[https://matrix.to/#/#floco:matrix.org][#floco:matrix.org]]

General Room:
[[https://matrix.to/#/!wMSeevIIjIbAOVbqHh:matrix.org?via=matrix.org]]
( Recommended )

Support Room:
[[https://matrix.to/#/!tBPFHeGmZfhbuYgvcw:matrix.org?via=matrix.org]]

Development Room:
[[https://matrix.to/#/!qDFpEnHkbpkhLSenko:matrix.org?via=matrix.org]]


* Supporters

** [[https://tulip.co/][Tulip Interfaces]]
=floco= was originally developed for use by Tulip Interfaces.
Without their support this project never would have been possible.