After I tried setting up deploy-rs and it (and flakes) is kind of not very good for what I am doing with my computers yesterday, I found that I still want to simplify my deploy tooling to make it easier to ship updates of my Nix systems to their hosts.
//files.fontkeming.fail/s/6NfrMZ66MPtJpit/download/terrible-meme-morph-wrapper.png
I landed on a solution inspired by Xe Iaso's hosts.toml setup. See Deploying from my hosts.toml for how this file is created, structured, and used in the morph commands themselves.
This page outlines a very simple script which ingests that hosts.toml file and provides a handful of options to make it easy for me to just specify hostnames and have the system figure out which manifest they should apply to and what to do with them.
We use toml and click and some builtins.
import toml
import click
import os
import socket
import subprocessShort help options are good imo, customize that.
CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help'])deploy-targets will print out the hostnames of all the hosts in the hosts.toml file suitable for using in arroyo-flood or so to interactively pick hostnames to deploy to.
Usage: deploy-targets [OPTIONS]
print a list of all the hosts in the hosts.toml
Options:
-f FILENAME hosts.toml file path
-h, --help Show this message and exit.import os.path
@click.command(context_settings=CONTEXT_SETTINGS)
@click.option('-f', 'hosts_file',
envvar="HOSTS_TOML", type=click.File('r'),
help="hosts.toml file path",
default=os.path.expanduser("~/arroyo-nix/networks/hosts.toml"))
def list_hosts(hosts_file):
"""
print a list of all the hosts in the hosts.toml
"""
network = toml.load(hosts_file)
for netname, host in get_all_hosts(network):
print(host)deploy does the thing. If you try to deploy to a host which cannot be pinged, it will be skipped. This doesn't read targetHost from the Morph network file, but those are lined up for me. -f will override this behavior, this is useful to me when I am bootstrapping a node.
Usage: deploy [OPTIONS] [HOSTS]...
build or deploy one or more hosts
Options:
-f FILENAME hosts.toml file path
--all deploy to all hosts in the manifest
--deploy / -b, --build choose whether to deploy or just build.
-a, --action TEXT choose the deploy action
-c, --confirm / -C, --no-confirm
ask before running morph commands
-f, --force / --no-force Don't skip unavailable hosts
-l, --local / --no-local Disable remote builders
-h, --help Show this message and exit.@click.command(context_settings=CONTEXT_SETTINGS)
@click.option('-f', 'hosts_file',
envvar="HOSTS_TOML",
help="hosts.toml file path",
type=click.File('r'),
default="/home/rrix/arroyo-nix/networks/hosts.toml")
@click.option('--all', 'deploy_all', is_flag=True, default=False,
help="deploy to all hosts in the manifest")
@click.option(' /-b', '--deploy/--build', 'do_deploy',
help="choose whether to deploy or just build.", is_flag=True, default=True)
@click.option('-a', '--action', 'deploy_action',
help="choose the deploy action", default="switch")
@click.option('-c/-C', '--confirm/--no-confirm', 'confirm',
help="ask before running morph commands", is_flag=True, default=False)
@click.option('-f', '--force/--no-force', 'force',
help="Don't skip unavailable hosts", is_flag=True, default=False)
@click.option('-l', '--local/--no-local', 'local',
help="Disable remote builders", is_flag=True, default=False)
@click.argument('hosts', nargs=-1)
def wrap(hosts_file, deploy_all, hosts, do_deploy, deploy_action, confirm, force, local):
"""
build or deploy one or more hosts
"""
network = toml.load(hosts_file)
if deploy_all:
hosts = [h for net,h in get_all_hosts(network)]
elif len(hosts) == 0:
hosts = (socket.gethostname(),)
subnets_to_deploy = get_pairs(network, hosts)
for network, host in subnets_to_deploy:
if do_deploy:
if force or host_available(host):
cmd = f"morph deploy --on={host} --passwd ~/arroyo-nix/networks/{network}.nix {deploy_action} --keep-result"
else:
click.echo(f"{host} is unavailable, skipping...")
continue
else:
cmd = f"morph build --on={host} ~/arroyo-nix/networks/{network}.nix --keep-result"
click.echo(f"Prepared to run '{cmd}'")
if confirm:
input("or hit ctrl-c... ")
if local:
os.environ["NIX_CONFIG"] = "builders = "
exit_code = os.WEXITSTATUS(os.system(cmd))
if exit_code != 0:
click.echo(f"the cmd \"{cmd}\" exited non-zero.")
return
def host_available(hostname: str) -> bool:
proc = subprocess.run(f"ping -w 1 -c 1 {hostname}", shell=True, capture_output=True)
return proc.returncode == 0 And these ugly list-comprehensions help to munge the TOML file in to a form the python commands here would like to use.
def get_all_hosts(network):
return [
(name, host)
for name, net in network.items()
for host in net['hosts'].keys()
]
def get_pairs(network, hosts):
return [
(netname, hostname)
for netname, subnet in network.items()
for hostname in hosts
if hostname in subnet['hosts'].keys()
]This is made available to my system builds in my rixpkgs overlay, and added like this:
, #+ARROYO_NIXOS_MODULE: nixos/morph-wrapper.nix
{ pkgs, ... }:
{
environment.systemPackages = [ pkgs.morph-wrapper ];
}So now when I make changes I can type deploy -b in to my nearest terminal to see my local system's build come together, then deploy to deploy it to this machine, then deploy $hostnames to deploy it to any number of my hostnames or deploy --all to deploy it everywhere.
Shell Environment and Pyproject manifest
This uses poetry and poetry2nix to generate a python application and nix derivation for use in my systems, a shell.nix is provided as well:
[tool.poetry]
name = "morph-wrapper"
version = "0.1.0"
description = ""
authors = ["Ryan Rix <code@whatthefuck.computer>"]
include = []
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry.dependencies]
python = "^3.10"
toml = "^0.10.2"
click = "^8.1.3"
[tool.poetry.scripts]
deploy = 'morph_wrapper.wrapper:wrap'
deploy-targets = 'morph_wrapper.wrapper:list_hosts'{ pkgs ? import <nixpkgs> {} }:
let
python-with-my-packages = pkgs.python3.withPackages (p: with p; [
toml
click
]);
in
pkgs.mkShell {
packages = [
python-with-my-packages
pkgs.poetry
];
}{ pkgs ? import <nixpkgs> {},
stdenv ? pkgs.stdenv,
python ? pkgs.python3 }:
let
inputs = (builtins.getFlake (toString ~/nix/.)).inputs;
poetry2nix = inputs.poetry2nix.lib;
in
poetry2nix.mkPoetryApplication {
inherit python;
projectDir = ./.;
propagatedBuildInputs = [];
}Add it to my systems in home-manager
{ pkgs, ... }:
{
home.packages = [ pkgs.morph ];
}