Setup flake info extraction

Prepare data model fro derivations (#1) Add flake info data (#1) Implement fetching general flake info (#1) Expose CLI (#1) Keep cargo happy Add some doc comments Pin to local nixpkgs to excessive downloads Extend visibility of some data objects Add command to extract infomation about defivations (#1) Add call new feature in main (#1) Include more information in derivation (#1) Add log access Always debug log stderr from nix Format nix script Collect systems per package Remove unnecessary imports Create flake Remove top level version field Represent collected systems/version pairs in rust Fix quotation marks in tests Add correct cargo hash Add iconv dependency Return a list from nix script Export as json Undo version by platform distinction Remove nixpkgs override Apply cargo fmt Flatten export structure Allow for complex licenses Prepare using a central nix file Implement nix part o accessing apps Include the correct filename Add accessor for `all` key Access all available information by default Track more information about Apps Run cargo fmt Fix: allow local builds Prepare next version of the flake info tool Include examples and pull script Expose flake info as library Include thiserror for custom errors Define a source data type Collects source types and their metadata, collected in a json file Add command line argument for input files Mutually exclusive with --flake Refactor functions to extract information given a flake identifier Add kind specifier as CLI argument Amend Argument parsing to require eiteher flake or targets to be defined Run extraction for specified flake or list of flakes as specified in a json file Resolves #5 References #7 Use internal tag to distnguich target types Include target falg usage in examples Set include provided source if available (resolves #9) Resolve flake name Update examples Dont include empty license or description Fix a misfomatting in cargot.toml Add elastic dependencies Implement a wrapper around the elasticsearch client Implements pushing exports (#4) Temporarily skip serializing an unimplemented field in elastic output Extract reading source list files from binary Add lazy_static as dependency Implement createing and pushing to elastic index Add elastic options Provide default name and env falbac for elastic index Modify app binary and type as optionals App can be a derivation too Update examples Add more elastic commands Supported: - ensure - clear - push Rename elastic search config struct Add elastic push support to binary Rename flag to enable elastic push Imporve error messages and format binary source Fix nix file incorrectly expecting meta fields Changing flake descriotions to an optional field deserialize git_ref as hash Implement temporary stores and gc of these prevents flakes from accessing store paths Pass extra arguments to nix Update cargo hash and skip integration tests Move flake.nix to root folder and add apps for all components Fix command invocation that fails test Update README(s) Add help for extra arguments (cherry picked from commit be4bc3dd929178bef66114c2201aaa88e47e9add)
ysndr · Jun 7, 2021 · fdfedf0 · fdfedf0
1 parent d1085b8
commit fdfedf0
Show file tree

Hide file tree

Showing 25 changed files with 3,402 additions and 0 deletions.
diff --git a/flake-info/Cargo.lock b/flake-info/Cargo.lock
diff --git a/flake-info/Cargo.toml b/flake-info/Cargo.toml
@@ -0,0 +1,27 @@
+[package]
+name = "flake-info"
+version = "0.2.0"
+authors = ["Yannik Sander <me@ysndr.de>"]
+edition = "2018"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+clap = "^2.33"
+serde = {version="1.0",  features = ["derive"]}
+serde_json = "1.0"
+anyhow = "1.0"
+thiserror = "1.0"
+structopt = "0.3"
+command-run = "0.13"
+env_logger = "0.8"
+log = "0.4"
+tempfile = "3"
+lazy_static = "1.4"
+
+elasticsearch = "7.12.0-alpha.1"
+tokio = { version = "*", features = ["full"] }
+
+[lib]
+name = "flake_info"
+path = "./src/lib.rs"
diff --git a/flake-info/README.md b/flake-info/README.md
@@ -0,0 +1,114 @@
+# Flake Info
+
+A tool that fetches packages and apps from nix flakes.
+
+## Usage
+
+```
+flake-info 0.2.0
+Extracts various information from a given flake
+
+USAGE:
+    flake-info [FLAGS] [OPTIONS] [extra]...
+
+FLAGS:
+        --elastic-recreate-index    Elasticsearch instance url
+        --push                      Push to Elasticsearch (Configure using FI_ES_* environment variables)
+        --gc                        Whether to use a temporary store or not. Located at /tmp/flake-info-store
+    -h, --help                      Prints help information
+        --temp-store                Whether to use a temporary store or not. Located at /tmp/flake-info-store
+    -V, --version                   Prints version information
+
+OPTIONS:
+        --elastic-index-name <elastic-index-name>
+            Name of the index to store results to [env: FI_ES_INDEX=]  [default: flakes_index]
+
+    -p, --elastic-pw <elastic-pw>                    Elasticsearch password (unimplemented) [env: FI_ES_PASSWORD=]
+        --elastic-url <elastic-url>
+            Elasticsearch instance url [env: FI_ES_URL=]  [default: http://localhost:9200]
+
+    -u, --elastic-user <elastic-user>                Elasticsearch username (unimplemented) [env: FI_ES_USER=]
+    -f, --flake <flake>                              Flake identifier passed to nix to gather information about
+    -k, --kind <kind>                                Kind of data to extract (packages|options|apps|all) [default: all]
+    -t, --targets <targets>                          Points to a JSON file containing info targets
+
+ARGS:
+    <extra>...    Extra arguments that are passed to nix as it
+```
+
+### flake/targets
+
+Use either of these options to define which flake you want to query.
+
+`--flake | -f`: takes a flake reference in the same format as nix
+
+> use git+<url> to checkout a git repository at <url>
+> use /local/absolute/path or ./relative/path to load a local source
+> use gitlab:<user>/<repo>/github:<user>/<repo> to shortcut gitlab or github repositories
+
+`--targets | -t`: refers to a json file that contains a list of queried repositories:
+
+```json
+[
+    {
+        "type": "git",
+        "url": "./."
+    },
+    {
+        "type": "git",
+        "url": "github:fluffynukeit/adaspark"
+    },
+    {
+        "type": "github",
+        "owner": "ngi-nix",
+        "repo": "offen",
+        "hash": "4052febf151d60aa4352fa1960cf3ae088f600aa",
+        "description": "Hier könnte Ihre Werbung stehen"
+    }
+]
+```
+
+Currently `github` and `gitlab` can be used as source repos with hash. the `hash` attribute defines the fetched git reference (branch, commit, tag, etc).
+
+## Installation
+
+### Preparations
+
+This tool requires your system to have Nix installed!
+
+You can install nix using this installer: https://nixos.org/guides/install-nix.html
+Also, see https://nixos.wiki/wiki/Nix_Installation_Guide if your system is ✨special✨.
+
+### Preparations (Docker)
+
+If you do not want to install nix on your system, using Docker is an alternative.
+
+Enter the [nixos/nix](https://hub.docker.com/u/nixos/) docker image and proceed
+
+### Setup nix flakes
+
+Note that you also need to have nix flakes support.
+
+Once you have nix installed run the following commands:
+
+1. ```
+   $ nix-shell -I nixpkgs=channel:nixos-21.05 -p nixFlakes
+   ```
+   to enter a shell with the preview version of nix flakes installed.
+2. ```
+   $ mkdir -p ~/.config/nix
+   $ echo "experimental-features = nix-command flakes" > .config/nix/nix.conf
+   ```
+   to enable flake support
+
+### Installation, finally
+
+This project is defined as a flake therefore you can build the tool using
+
+```
+$ nix build <project root>
+or
+$ nix build github:miszkur/github-search
+```
+
+Replace `build` with run if you want to run the tool directly.
diff --git a/flake-info/examples/.-..json b/flake-info/examples/.-..json
@@ -0,0 +1,48 @@
+[
+  {
+    "flake_description": "Extracting information from flakes",
+    "flake_resolved": {
+      "type": "git",
+      "url": "file:///Volumes/projects/Uni/courses/kth/DD2476-Search-Engines-and-Information-Retrieval-Systems/project?dir=flake-info"
+    },
+    "flake_name": "",
+    "flake_source": {
+      "type": "git",
+      "url": "./."
+    },
+    "package_attr_name": "flake-info",
+    "package_pname": "flake-info",
+    "package_pversion": "",
+    "package_platforms": [
+      "x86_64-linux",
+      "x86_64-darwin",
+      "i686-linux",
+      "aarch64-linux"
+    ],
+    "package_outputs": [
+      "out"
+    ],
+    "package_license": {}
+  },
+  {
+    "flake_description": "Extracting information from flakes",
+    "flake_resolved": {
+      "type": "git",
+      "url": "file:///Volumes/projects/Uni/courses/kth/DD2476-Search-Engines-and-Information-Retrieval-Systems/project?dir=flake-info"
+    },
+    "flake_name": "",
+    "flake_source": {
+      "type": "git",
+      "url": "./."
+    },
+    "app_bin": "/nix/store/4akx0is6fgh9ci2ak5sbskwzykr0xj85-flake-info/bin/flake-info",
+    "app_attr_name": "flake-info",
+    "app_platforms": [
+      "x86_64-linux",
+      "x86_64-darwin",
+      "i686-linux",
+      "aarch64-linux"
+    ],
+    "app_type": "app"
+  }
+]