The Complete Computing Environment

Datasette: An open source multi-tool for exploring and publishing data

LifeTechEmacsTopicsArcology

Archive Tools

Datasette is a tool for exploring and publishing data. It helps people take data of any shape, analyze and explore it, and publish it as an interactive website and accompanying API.

Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world. It is part of a wider ecosystem of 40 tools and 93 plugins dedicated to making working with structured data as productive as possible.

These could be used to generate org-mode files for The Arcology Project and my Knowledge Base….

iNaturalist to Sqlite

Slurp: shell:inaturalist-to-sqlite ~/DataExports/inaturalist.db rrix &

{ pkgs ? import <nixpkgs>,
  python ? pkgs.python3,
  ... }:

python.pkgs.buildPythonPackage {
  pname = "inaturalist-to-sqlite";
  version = "0.2.1";
  src = pkgs.fetchFromGitHub {
    owner = "dogsheep";
    repo = "inaturalist-to-sqlite";
    rev = "d888c7c2f02aa0dfb1559603f02357cd0089da11";
    sha256 = "sha256-2YKGGzkRJJ68kj1h8Di8weY0cDGTnOkI1DZ2aqNsy0c=";
  };

  propagatedBuildInputs = with python.pkgs; [
    sqlite-utils
    click
    requests
  ];

  checkInputs = with python.pkgs; [
    pytest
  ];

  meta = with pkgs.lib; {
    description = "Create a SQLite database containing your observation history from iNaturalist";
    homepage = "https://github.com/dogsheep/inaturalist-to-sqlite";
    platforms = platforms.unix;
    maintainers = with maintainers; [rrix];
    license = with licenses; [ asl20 ];
  };
}

Twitter to Sqlite

Slurp favorites: shell:twitter-to-sqlite favorites --auth ~/sync/auth.json ~/"DataExports/twitter-faves.db" & Slurp followers: shell:twitter-to-sqlite followers --auth ~/sync/auth.json ~/"DataExports/twitter-follows.db" &

sqlite> SELECT id, fulltext FROM tweets WHERE ROWID IN (SELECT ROWID FROM tweetsfts WHERE tweetsfts MATCH 'quick take' ORDER BY rank); 1545048960699551744|@deejaygeejaygee quick take https://twitter.com/MartinLundfall/status/1545048960699551744/video/1

{ pkgs ? import <nixpkgs>,
  python ? pkgs.python3,
  ... }:

python.pkgs.buildPythonPackage {
  pname = "twitter-to-sqlite";
  version = "0.2.1";
  src = pkgs.fetchFromGitHub {
    owner = "dogsheep";
    repo = "twitter-to-sqlite";
    rev = "f09d611782a8372cfb002792dfa727325afb4db6";
    sha256 = "sha256-ICAs/DOcyMA0DBEaMk/KG2tsMIDl3MKfP1CdtVXQIls=";
  };

  propagatedBuildInputs = with python.pkgs; [
    sqlite-utils
    requests-oauthlib
    python-dateutil
  ];

  patchPhase = ''
    substituteInPlace setup.py --replace '"requests-oauthlib~=1.2.0",' '"requests-oauthlib~=1.3.0",'
  '';

  checkInputs = with python.pkgs; [
    pytest
  ];

  meta = with pkgs.lib; {
    description = "Save data from Twitter to a SQLite database.";
    homepage = "https://github.com/dogsheep/twitter-to-sqlite";
    platforms = platforms.unix;
    maintainers = with maintainers; [rrix];
    license = with licenses; [ asl20 ];
  };
}

NEXT move these to rixpkgs

home-manager availability

Use these in home-manager like so:

{ pkgs, ... }:

let
  inat-sql = import ../pkgs/inaturalist-to-sqlite.nix { inherit pkgs; };
  twitter-sql = import ../pkgs/twitter-to-sqlite.nix { inherit pkgs; };
in
{
  home.packages = [ inat-sql twitter-sql ];
}