The Complete Computer

Archive Archive Tools

Datasette is a tool for exploring and publishing data. It helps people take data of any shape, analyze and explore it, and publish it as an interactive website and accompanying API.

Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world. It is part of a wider ecosystem of 40 tools and 93 plugins dedicated to making working with structured data as productive as possible.

These could be used to generate org-mode files for The Arcology Project and my Knowledge Base ....

Slurp: shell:inaturalist-to-sqlite ~/DataExports/inaturalist.db rrix &

nix source: :tangle ~/arroyo-nix/pkgs/inaturalist-to-sqlite.nix
{ pkgs ? import <nixpkgs>,
  python ? pkgs.python3,
  ... }:

python.pkgs.buildPythonPackage {
  pname = "inaturalist-to-sqlite";
  version = pkgs.lib.pkgVersions.inaturalist-to-sqlite.version;
  src = pkgs.callPackage pkgs.lib.pkgVersions.inaturalist-to-sqlite.src {};
  pyproject = true;
  build-system = [ python.pkgs.setuptools ];

  propagatedBuildInputs = with python.pkgs; [
    pip
    sqlite-utils
    click
    requests
  ];

  # postUnpack = ''
  #   replace py
  # '';

  checkInputs = with python.pkgs; [
    pytest
  ];

  meta = with pkgs.lib; {
    description = "Create a SQLite database containing your observation history from iNaturalist";
    homepage = "https://github.com/dogsheep/inaturalist-to-sqlite";
    platforms = platforms.unix;
    maintainers = with maintainers; [rrix];
    license = with licenses; [ asl20 ];
  };
}

• Slurp favorites: shell:twitter-to-sqlite favorites --auth ~/sync/auth.json ~/"DataExports/twitter-faves.db" &

• Slurp followers: shell:twitter-to-sqlite followers --auth ~/sync/auth.json ~/"DataExports/twitter-follows.db" &

• Track Twitter Following Deltas

sqlite> SELECT id, full_text FROM tweets WHERE ROWID IN (SELECT ROWID FROM tweets_fts WHERE tweets_fts MATCH 'quick take' ORDER BY rank);
1545048960699551744|@deejaygeejaygee quick take https://twitter.com/MartinLundfall/status/1545048960699551744/video/1

nix source: :tangle ~/arroyo-nix/pkgs/twitter-to-sqlite.nix
{ pkgs ? import <nixpkgs>,
  python ? pkgs.python3,
  ... }:

python.pkgs.buildPythonPackage {
  pname = "twitter-to-sqlite";

  version = pkgs.lib.pkgVersions.twitter-to-sqlite.version;
  src = pkgs.callPackage pkgs.lib.pkgVersions.twitter-to-sqlite.src {};
  pyproject = true;
  build-system = [ python.pkgs.setuptools ];

  propagatedBuildInputs = with python.pkgs; [
    pip
    click-default-group
    sqlite-utils
    requests-oauthlib
    python-dateutil
  ];

  patchPhase = ''
    substituteInPlace setup.py --replace '"requests-oauthlib~=1.2.0",' '"requests-oauthlib~=1.3.0",'
  '';

  checkInputs = with python.pkgs; [
    pytest
  ];

  meta = with pkgs.lib; {
    description = "Save data from Twitter to a SQLite database.";
    homepage = "https://github.com/dogsheep/twitter-to-sqlite";
    platforms = platforms.unix;
    maintainers = with maintainers; [rrix];
    license = with licenses; [ asl20 ];
  };
}

Use these in home-manager like so:

nix source: :tangle ~/arroyo-nix/hm/datasette.nix
{ pkgs, ... }:

{
  home.packages = [
    pkgs.inaturalist-to-sqlite
    # pkgs.twitter-to-sqlite
  ];
}