CCE

Rebuild of The Complete Computer 0x02: This Is (Not) Advanced

Contents

A walk-through of the layers, this is basically a long boring presentation, a chain of connected things which lead to a rat's nest or a social operating system

Components

  • org-roam, a free-range document organization and journaling system

  • arroyo, the file,key,value store and document relation database that drives The Arcology Project.

  • bootstrapping the development environment

    • start with an fedora install in a VM, install nix, enable flakes

    • actually running nixos comes at a later stage

  • arcology (start by just getting it running on localhost, and going over the code and the functionality, start to see how literate programming works in this design)

  • cce repositories

  • arroyo generators

  • arroyo nixos and the wobserver

  • the tailnet and the wobserver (sub-text: all of human progress is built standing on the shoulders of giants and an insane level of abstraction exists in the world -- what does that mean for us and what should we do about it, or: what's one more layer?)

Org-roam

org-roam is a knowledge management system built on top of org-mode bringing powerful navigation and discovery features to the humble plain-text document.

  • Completely offline, completely legible.

  • Networked with backlinks and graph visualizations

  • Extensible and powerful thanks to Emacs and the org-mode ecosystem

  • Free, Open Source, Communal

The Complete Computer centers org-roam. Rather than navigating to individual files, you're as likely to open up an org-mode file based on the title of a heading or portion of the document you wish to edit or view.

Org-roam changes your Org-mode behavior to be more oriented around individual Headings rather than an entire file. You may be viewing this entire file, but I can link directly to this Heading within it.

Arroyo

Org-mode supports some really nice faculties for embedding data and metadata. For metadata we have a few options: [See also Org-mode, File Variables, Properties, and Keywords]

  • org-mode documents support a Key/Value metadata called KEYWORDS. These look like #+THE_KEY: the_value

  • Headings can also store key/value metadata in "drawers" called PROPERTIES.

    • These drawers contain at minimum an ID that is unique to the org-roam database (usually just as a timestamp or UUID) but can contain any other stringly-typed

These can be extracted, normalized, and queried. This part is what enables Arroyo Systems Management.

Pages can be annotated with metadata to publish them to the web, to add Nix features or Emacs init.el snippets to the system, or to be generically queried by the end-user to provide a sort of limited vision of Org Mode as Hypermedia Documents. Other Arroyo Generators can be developed using Python to add new meta-cognitive powers to yourself and your community

Bootstrap Phase 0: Standing up a new environment from scratch

Install VirtualBox or QEMU

On NixOS we can do this to easily turn on virt-manager:

nix:tangle ~/nix/nixos/virt-mangler.nix
{ ... }:

{
  virtualisation.libvirtd.enable = true;
  programs.virt-manager.enable = true;
}

Install Fedora in a VM

  • https://fedoraproject.org/workstation/ or https://fedoraproject.org/spins/kde/ I really would rather not use VirtualBox personally preferring virt-manager/kvm/qemu but having a single example interface for installation and usage will make this easier for Windows or Mac users to experience the Complete Computer.

Install Nix

in the VM:

  • sudo setenforce 0 and =sudo grubby --update-kernel ALL --args selinux=0=

  • then the sh pipe curl 😩 command from https://nixos.org/download#download-nix

NEXT What do I think about https://github.com/DeterminateSystems/nix-installer

NEXT What do i think about just using distro package or copr?

Configure and enable Flakes

From the NixOS Wiki:

Add the following to ~/.config/nix/nix.conf or /etc/nix/nix.conf:

: experimental-features = nix-command flakes

In a system with a new Nix install we can safely run mkdir ~/.config/nix && echo 'experimental-features nix-command flakes' >> ~/.config/nix/nix.conf=.

Running The Arcology

In theory you could just nix run the Arcology straight from https://code.rix.si/rrix/arcology, but you're going to want to make changes to the configuration so clone it instead to your org-mode directory:

shell
sudo dnf install git
mkdir ~/org/
pushd ~/org/

[ -d $PWD/arcology ] || git clone https://code.rix.si/rrix/arcology

Take a look at the Arcology Project Configuration once you've done that; eventually you'll want to change the sites, the service configuration etc, but get yourself familiar with what is there in the meantime.

At this point you can set some configuration variables and run nix run path:$HOME/org/arcology# -- migrate to set up the SQLite database with the schema but with no data.

Further usage commands will come along later, this just makes sure we're ready to roll.

you'll want to run some Arcology commands in the background whenever you are working on your documentation system:

for now we can run these in Emacs whenever we are working on the system but eventually they'll be run as persistent services in a systemd session.

With the arcology cloned and running, you can extract metadata like this:

shell
pushd ~/org/arcology
nix run . -- migrate
nix run . -- migrate --database writable
nix run . -- seed
nix run . -- ingestfiles ~/org

Bootstrap Phase 1: local arcology and home-manager

Write something like this to ~/arroyo-nix/home-manager.nix

nix:noweb yes
{ config, pkgs, ... }:

{
  # Home Manager needs a bit of information about you and the
  # paths it should manage.
  home.username = "jdoe";
  home.homeDirectory = "/home/jdoe";

  # This value determines the Home Manager release that your
  # configuration is compatible with. This helps avoid breakage
  # when a new Home Manager release introduces backwards
  # incompatible changes.
  #
  # You can update Home Manager without changing this value. See
  # the Home Manager release notes for a list of state version
  # changes in each release.
  home.stateVersion = "24.05";

  # Let Home Manager install and manage itself.
  programs.home-manager.enable = true;

  # Emacs, too, for now...
  programs.emacs.enable = true;

  # he he he...
  # <<bootstrap-steps>>
}

run =nix run nixpkgs#home-manager -- switch= and open a new shell where you can now invoke the home-manager command in your PATH as well as having a completely unconfigured emacs. Now to configure it...

Now that there is metadata ingested, if you ran for example nix run . -- generate -m emacs it will show you which files would be included in an Emacs init. There is probably not much right now, but what is included in the Arcology repository should help your system get started. The Arcology itself includes a very minimal Emacs configuration that should be suitable for starting to do literate programming and assemble your first Emacs init file.

You'll have to do that yourself for now to bootstrap this thing. Run Emacs in the Arcology directory ~/org/arcology and evaluate this in a scratch buffer:

emacs-lisp
(use-package s)
(let ((snippets (s-split "\n" (shell-command "nix run . -- generate -m emacs 2>/dev/null")))
    (with-current-buffer (find-file "~/arroyo-nix/files/init.el")
      (erase-buffer)
      (dolist (path snippets)
        (insert-file path)
        (goto-char (point-max)))
      (save-buffer))))

Hit C-x b and open =scratch= by pressing Return, then paste that text using the button on the window or hitting C-y, then type M-x eval-buffer. With any luck, the Emacs will open an init.el file and stick code in it. Assuming that file is valid, you should enable it:

In the configuration above, you'll see a comment with the text <<bootstrap-steps>>. If you're following along the VOD by hand, all you need to do for now is put this line directly below it:

nix#+name: bootstrap-steps:noweb-ref bootstrap-steps
home.file.".emacs.d/init.el".target = "arroyo-nix/files/init.el";

Now you can run home-manager -f ~/arroyo-nix/home-manager.nix switch in a shell, re-start Emacs, watch it download packages for a bit and then you'll have an environment that behaves more like vim and that supports org-mode and org-roam so that we can further develop this system within org-mode.

And when you rebuild your home-manager environment and run emacs you will see it download a bunch of packages and start up with support for Org Roam.

Congrats, in some small way you've constructed your first Arroyo system!

NEXT include instructions on how to validate the bootstrap

what does a valid init look like

NEXT include the emacs init bootstrap somewhere

INPROGRESS Bootstrap Phase 2: Moving in to documents, building your own CCE modules

Included in the arcology are a handful of really basic Emacs configuration snippets suitable for allowing you to now bootstrap your way in to a working home-manager installation. Let's start a new document like my Dynamic home-manager.nix, starting with the home-manager.nix file you've been working on, and the stub-home-manager.org in ~/org/arcology/bootstrap/stub-hm.org.

Let's start by including the modules by hand by creating an imports section below the home.file line you added:

nix#+name: bootstrap-steps:noweb-ref bootstrap-steps:noweb yes
imports = [
  # stick the output of the <<hm-imports>> command below here
  # soon these will come from a local HTTP API!
];
shell#+name: hm-imports:noweb-ref hm-imports
nix run . -- generate -m home-manager 2&>/dev/null

Aside: noweb-ref? =<<bootstrap-steps>>? what?

If you're following along to bootstrap a system, this stuff above just looks like line noise, that's okay. Nix is a weird programming language, the semantics of the declarative system is strange, and even then there is crap that can't be explained. what is the <<bootstrap-steps>> line about?

When you fold this configuration in to a new document, which you'll do now if you're following along with the VOD, you'll include that text, and when you go to tangle the file, you'll find that on disk the home.file line will be there twice, once commented.

What gives? if you're working with this document, you're literate programming, now.

this code segment is labeled with a noweb-ref reference and the double bracket syntax you see in the document is a noweb expansion, it embeds that text directly. and the wild thing about this is that if you were to type <<some-other-code()>> and some-other-code referred to an executable code block, whether in lisp, or shell, or python, or R, it will insert the output of that code's evaluation instead, so you can generate code dynamically. For now, go up to that <<manual-transclusions>> line, remove the comment from it and then hit the keys C-c C-v t to "tangle" the files

WORK

NEXT expand cce-core deps and vendor them

NEXT need to have a script that clones repos and NIX_PATH for first home-manager runs

NEXT need stub-home-manager.org and cce-core folded in to arcology repo

INPROGRESS crappy little lisp snippet to tangle missing files

for the bootstrap:

emacs-lisp
(thread-last
  "home-manager"
  (arroyo-generator-data)
  (seq-map (lambda (data)
             (let* ((org (alist-get 'path data))
                    (loc (alist-get 'destination data))
                    (path (expand-file-name loc "~/arroyo-nix")))
               (org-babel-tangle-file org)
               )))
  (seq-filter #'identity))

NEXT deploy the system with the arcology configuration module

shell
cd ~/org/arcology
nix run . -- ingestfiles
ARCOLOGY_LOCALAPI_BEARER_TOKEN=test nix run . -- runserver --port 29543

(setq arcology-localapi-bearer-token "test") and M-x arroyo-emacs-generate-init

at this point we start to pull in the modules we want to build the systems we need.

NEXT begin configuring the arcology and the emacs environment

WAITING goal: Arcology localapi running

WAITING your own dynamic home-manager.org file

then we can set up a nixos VM somewhere......

NEXT Arroyo Generators

NEXT Bootstrap Phase 3: Arroyo NixOS and the Wobserver

NEXT The Tailnet

Contents
Pages Linking Here
Page Tags