The Complete Computer

After I tried setting up deploy-rs and it (and flakes) is kind of not very good for what I am doing with my computers yesterday, I found that I still want to simplify my deploy tooling to make it easier to ship updates of my Nix systems to their hosts.

I landed on a solution inspired by roam:Xe Iaso 's hosts.toml setup. See Deploying from my =hosts.toml= for how this file is created, structured, and used in the morph commands themselves.

This page outlines a very simple script which ingests that hosts.toml file and provides a handful of options to make it easy for me to just specify hostnames and have the system figure out which manifest they should apply to and what to do with them.

We use toml and click and some builtins.

python source: :mkdirp yes :tangle ~/org/morph-wrapper/morph_wrapper/wrapper.py
import toml
import click

import os
import socket

import subprocess

Short help options are good imo, customize that.

python source: :mkdirp yes :tangle ~/org/morph-wrapper/morph_wrapper/wrapper.py
CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help'])

deploy-targets will print out the hostnames of all the hosts in the hosts.toml file suitable for using in arroyo-flood or so to interactively pick hostnames to deploy to.

 source: 
Usage: deploy-targets [OPTIONS]

  print a list of all the hosts in the hosts.toml

Options:
  -f FILENAME  hosts.toml file path
  -h, --help   Show this message and exit.

python source: :mkdirp yes :tangle ~/org/morph-wrapper/morph_wrapper/wrapper.py
import os.path

@click.command(context_settings=CONTEXT_SETTINGS)
@click.option('-f', 'hosts_file',
              envvar="HOSTS_TOML", type=click.File('r'),
              help="hosts.toml file path",
              default=os.path.expanduser("~/arroyo-nix/networks/hosts.toml"))
def list_hosts(hosts_file):
    """
    print a list of all the hosts in the hosts.toml
    """
    network = toml.load(hosts_file)
    for netname, host in get_all_hosts(network):
        print(host)

deploy does the thing. If you try to deploy to a host which cannot be pinged, it will be skipped. This doesn't read targetHost from the Morph network file, but those are lined up for me. -f will override this behavior, this is useful to me when I am bootstrapping a node.

 source: 
Usage: deploy [OPTIONS] [HOSTS]...

  build or deploy one or more hosts

Options:
  -f FILENAME                     hosts.toml file path
  --all                           deploy to all hosts in the manifest
  --deploy / -b, --build          choose whether to deploy or just build.
  -a, --action TEXT               choose the deploy action
  -c, --confirm / -C, --no-confirm
                                  ask before running morph commands
  -f, --force / --no-force        Don't skip unavailable hosts
  -l, --local / --no-local        Disable remote builders
  -h, --help                      Show this message and exit.

python source: :mkdirp yes :tangle ~/org/morph-wrapper/morph_wrapper/wrapper.py
@click.command(context_settings=CONTEXT_SETTINGS)
@click.option('-f', 'hosts_file',
              envvar="HOSTS_TOML",
              help="hosts.toml file path",
              type=click.File('r'),
              default="/home/rrix/arroyo-nix/networks/hosts.toml")
@click.option('--all', 'deploy_all', is_flag=True, default=False,
              help="deploy to all hosts in the manifest")
@click.option(' /-b', '--deploy/--build', 'do_deploy',
              help="choose whether to deploy or just build.", is_flag=True, default=True)
@click.option('-a', '--action', 'deploy_action',
              help="choose the deploy action", default="switch")
@click.option('-c/-C', '--confirm/--no-confirm', 'confirm',
              help="ask before running morph commands", is_flag=True, default=False)
@click.option('-f', '--force/--no-force', 'force',
              help="Don't skip unavailable hosts", is_flag=True, default=False)
@click.option('-l', '--local/--no-local', 'local',
              help="Disable remote builders", is_flag=True, default=False)
@click.argument('hosts', nargs=-1)
def wrap(hosts_file, deploy_all, hosts, do_deploy, deploy_action, confirm, force, local):
    """
    build or deploy one or more hosts
    """
    network = toml.load(hosts_file)

    if deploy_all:
        hosts = [h for net,h in get_all_hosts(network)]
    elif len(hosts) == 0:
        hosts = (socket.gethostname(),)

    subnets_to_deploy = get_pairs(network, hosts)

    for network, host in subnets_to_deploy:

        if do_deploy:
            if force or host_available(host):
                cmd = f"morph deploy --on={host} --passwd ~/arroyo-nix/networks/{network}.nix {deploy_action} --keep-result"
            else:
                click.echo(f"{host} is unavailable, skipping...")
                continue 
        else:
            cmd = f"morph build --on={host} ~/arroyo-nix/networks/{network}.nix --keep-result"

        click.echo(f"Prepared to run '{cmd}'")
        if confirm:
            input("or hit ctrl-c... ")
        if local:
            os.environ["NIX_CONFIG"] = "builders = "
        exit_code = os.WEXITSTATUS(os.system(cmd))

        if exit_code != 0:
            click.echo(f"the cmd \"{cmd}\" exited non-zero.")
            return

def host_available(hostname: str) -> bool:
    proc = subprocess.run(f"ping -w 1 -c 1 {hostname}", shell=True, capture_output=True)
    return proc.returncode == 0 

And these ugly list-comprehensions help to munge the TOML file in to a form the python commands here would like to use.

python source: :mkdirp yes :tangle ~/org/morph-wrapper/morph_wrapper/wrapper.py
def get_all_hosts(network):
    return [
        (name, host)
        for name, net in network.items()
        for host in net['hosts'].keys()
    ]

def get_pairs(network, hosts):
    return [
        (netname, hostname)
        for netname, subnet in network.items()
        for hostname in hosts
        if hostname in subnet['hosts'].keys()
    ]

This is made available to my system builds in my rixpkgs overlay, and added like this:

nix source: :tangle ~/arroyo-nix/nixos/morph-wrapper.nix
{ pkgs, ... }:

{
  environment.systemPackages = [ pkgs.morph-wrapper ];
}

So now when I make changes I can type deploy -b in to my nearest terminal to see my local system's build come together, then deploy to deploy it to this machine, then deploy $hostnames to deploy it to any number of my hostnames or deploy --all to deploy it everywhere.

This uses poetry and poetry2nix to generate a python application and nix derivation for use in my systems, a shell.nix is provided as well:

toml source: :mkdirp yes :tangle ~/org/morph-wrapper/pyproject.toml
[tool.poetry]
name = "morph-wrapper"
version = "0.1.0"
description = ""
authors = ["Ryan Rix <code@whatthefuck.computer>"]

include = []

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = "^3.10"
toml = "^0.10.2"
click = "^8.1.3"

[tool.poetry.scripts]
deploy = 'morph_wrapper.wrapper:wrap'
deploy-targets = 'morph_wrapper.wrapper:list_hosts'

nix source: :tangle ~/org/morph-wrapper/shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
  python-with-my-packages = pkgs.python3.withPackages (p: with p; [
    toml
    click
  ]);
in
pkgs.mkShell {
  packages = [
    python-with-my-packages
    pkgs.poetry
  ];
}

nix source: :tangle ~/org/morph-wrapper/default.nix
{ pkgs ? import <nixpkgs> {},
  poetry2nix ? import((import <arroyo/versions.nix> {}).poetry2nix null) {},
  stdenv ? pkgs.stdenv,
  python ? pkgs.python3 }:

poetry2nix.mkPoetryApplication {
  inherit python;
  projectDir = ./.;
  propagatedBuildInputs = [];
}

nix source: :tangle ~/arroyo-nix/hm/morph.nix
{ pkgs, ... }:
{
  home.packages = [ pkgs.morph ];
}