Skip to content

Commit

Permalink
Setup flake info extraction
Browse files Browse the repository at this point in the history 
Prepare data model fro derivations (#1)

Add flake info data (#1)

Implement fetching general flake info (#1)

Expose CLI (#1)

Keep cargo happy

Add some doc comments

Pin to local nixpkgs to excessive downloads

Extend visibility of some data objects

Add command to extract infomation about defivations (#1)

Add call new feature in main (#1)

Include more information in derivation (#1)

Add log access

Always debug log stderr from nix

Format nix script

Collect systems per package

Remove unnecessary imports

Create flake

Remove top level version field

Represent collected systems/version pairs in rust

Fix quotation marks in tests

Add correct cargo hash

Add iconv dependency

Return a list from nix script

Export as json

Undo version by platform distinction

Remove nixpkgs override

Apply cargo fmt

Flatten export structure

Allow for complex licenses

Prepare using a central nix file

Implement nix part o accessing apps

Include the correct filename

Add accessor for `all` key

Access all available information by default

Track more information about Apps

Run cargo fmt

Fix: allow local builds

Prepare next version of the flake info tool

Include examples and pull script

Expose flake info as library

Include thiserror for custom errors

Define a source data type

Collects source types and their metadata, collected in a json file

Add command line argument for input files

Mutually exclusive with --flake

Refactor functions to extract information given a flake identifier

Add kind specifier as CLI argument

Amend Argument parsing to require eiteher flake or targets to be defined

Run extraction for specified flake or list of flakes as specified in a json file

Resolves #5
References #7

Use internal tag to distnguich target types

Include target falg usage in examples

Set include provided source if available (resolves #9)

Resolve flake name

Update examples

Dont include empty license or description

Fix a misfomatting in cargot.toml

Add elastic dependencies

Implement a wrapper around the elasticsearch client

Implements pushing exports (#4)

Temporarily skip serializing an unimplemented field in elastic output

Extract reading source list files from binary

Add lazy_static as dependency

Implement createing and pushing to elastic index

Add elastic options

Provide default name and env falbac for elastic index

Modify app binary and type as optionals

App can be a derivation too

Update examples

Add more elastic commands

Supported:
- ensure
- clear
- push

Rename elastic search config struct

Add elastic push support to binary

Rename flag to enable elastic push

Imporve error messages and format binary source

Fix nix file incorrectly expecting meta fields

Changing flake descriotions to an optional field

deserialize git_ref as hash

Implement temporary stores and gc of these

prevents flakes from accessing store paths

Pass extra arguments to nix

Update cargo hash and skip integration tests

Move flake.nix to root folder and add apps for all components

Fix command invocation that fails test

Update README(s)

Add help for extra arguments

(cherry picked from commit be4bc3dd929178bef66114c2201aaa88e47e9add)
  • Loading branch information
@ysndr
ysndr committed Aug 17, 2021
1 parent 5ad7136 commit 783aabf
Show file tree
Hide file tree
Showing 17 changed files with 501 additions and 1,466 deletions.
236 changes: 6 additions & 230 deletions flake-info/Cargo.lock
View file

Large diffs are not rendered by default.

9 changes: 3 additions & 6 deletions flake-info/Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "flake-info"
version = "0.3.0"
version = "0.2.0"
authors = ["Yannik Sander <me@ysndr.de>"]
edition = "2018"

Expand All @@ -18,12 +18,9 @@ env_logger = "0.8"
log = "0.4"
tempfile = "3"
lazy_static = "1.4"
fancy-regex = "0.6"
tokio = { version = "*", features = ["full"] }
reqwest = { version = "0.11", features = ["json", "blocking"] }
sha2 = "0.9"

elasticsearch = {git = "https://github.com/elastic/elasticsearch-rs", features = ["rustls-tls"]}
elasticsearch = "7.12.0-alpha.1"
tokio = { version = "*", features = ["full"] }

[lib]
name = "flake_info"
Expand Down
132 changes: 20 additions & 112 deletions flake-info/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,105 +5,48 @@ A tool that fetches packages and apps from nix flakes.
## Usage

```
flake-info 0.3.0
flake-info 0.2.0
Extracts various information from a given flake
USAGE:
flake-info [FLAGS] [OPTIONS] [extra]... <SUBCOMMAND>
flake-info [FLAGS] [OPTIONS] [extra]...
FLAGS:
--push Push to Elasticsearch (Configure using FI_ES_* environment variables)
-h, --help Prints help information
--json Print ElasticSeach Compatible JSON output
-V, --version Prints version information
--elastic-recreate-index Elasticsearch instance url
--push Push to Elasticsearch (Configure using FI_ES_* environment variables)
--gc Whether to use a temporary store or not. Located at /tmp/flake-info-store
-h, --help Prints help information
--temp-store Whether to use a temporary store or not. Located at /tmp/flake-info-store
-V, --version Prints version information
OPTIONS:
--elastic-exists <elastic-exists>
How to react to existing indices [env: FI_ES_EXISTS_STRATEGY=] [default: abort] [possible values: Abort,
Ignore, Recreate]
--elastic-index-name <elastic-index-name> Name of the index to store results to [env: FI_ES_INDEX=]
-p, --elastic-pw <elastic-pw>
Elasticsearch password (unimplemented) [env: FI_ES_PASSWORD=]
--elastic-schema-version <elastic-schema-version>
Which schema version to associate with the operation [env: FI_ES_VERSION=]
--elastic-index-name <elastic-index-name>
Name of the index to store results to [env: FI_ES_INDEX=] [default: flakes_index]
-p, --elastic-pw <elastic-pw> Elasticsearch password (unimplemented) [env: FI_ES_PASSWORD=]
--elastic-url <elastic-url>
Elasticsearch instance url [env: FI_ES_URL=] [default: http://localhost:9200]
-u, --elastic-user <elastic-user> Elasticsearch username (unimplemented) [env: FI_ES_USER=]
-k, --kind <kind>
Kind of data to extract (packages|options|apps|all) [default: all]
-u, --elastic-user <elastic-user> Elasticsearch username (unimplemented) [env: FI_ES_USER=]
-f, --flake <flake> Flake identifier passed to nix to gather information about
-k, --kind <kind> Kind of data to extract (packages|options|apps|all) [default: all]
-t, --targets <targets> Points to a JSON file containing info targets
ARGS:
<extra>... Extra arguments that are passed to nix as it
SUBCOMMANDS:
flake
group
help Prints this message or the help of the given subcommand(s)
nixpkgs
```

### flake

Flakes can be imported using the flake subcommand

```
USAGE:
flake-info flake [FLAGS] <flake>

FLAGS:
--gc Whether to gc the store after info or not
-h, --help Prints help information
--temp-store Whether to use a temporary store or not. Located at /tmp/flake-info-store
-V, --version Prints version information
### flake/targets

ARGS:
<flake> Flake identifier passed to nix to gather information about
```
Use either of these options to define which flake you want to query.

The `<flake>` argument should contain a valid reference to a flake. It accepts all formats nix accepts:
`--flake | -f`: takes a flake reference in the same format as nix

> use git+<url> to checkout a git repository at <url>
> use /local/absolute/path or ./relative/path to load a local source
> use gitlab:<user>/<repo>/github:<user>/<repo> to shortcut gitlab or github repositories

Optionally, analyzing can be done in a temporary store enabled by the `--temp-store` option.

#### Example

```
$ flake-info flake github:ngi-nix/offen
```

### nixpkgs

nixpkgs currently have to be imported in a different way. This is what the `nixpkgs` subcommand exists for.

It takes any valid git reference to the upstream [`nixos/nixpkgs`](https://github.com/iixos/nixpkgs/) repo as an argument and produces a complete output.

**This operation may take a short while and produces lots of output**

#### Example

```
$ flake-info nixpkgs nixos-21.05
```

### group

to perform a bulk import grouping multiple inputs under the same name/index use the group command.

It expects a json file as input that contains references to flakes or nixpkgs. If those resources are on github or gitlab they can be extended with more meta information including pinning the commit hash/ref.

The second argument is the group name that is used to provide the index name.

#### Example

An example `targets.json` file can look like the following
`--targets | -t`: refers to a json file that contains a list of queried repositories:

```json
[
Expand All @@ -125,42 +68,7 @@ An example `targets.json` file can look like the following
]
```

```
$ flake-info group ./targets.json small-group
```

### Elasticsearch

A number of flags is dedicated to pushing to elasticsearch.

```
--elastic-exists <elastic-exists>
How to react to existing indices [env: FI_ES_EXISTS_STRATEGY=] [default: abort]
[possible values: Abort, Ignore, Recreate]
--elastic-index-name <elastic-index-name>
Name of the index to store results to [env: FI_ES_INDEX=]
-p, --elastic-pw <elastic-pw>
Elasticsearch password (unimplemented) [env: FI_ES_PASSWORD=]
--elastic-schema-version <elastic-schema-version>
Which schema version to associate with the operation [env: FI_ES_VERSION=]
--elastic-url <elastic-url>
Elasticsearch instance url [env: FI_ES_URL=] [default: http://localhost:9200]
-u, --elastic-user <elastic-user> Elasticsearch username (unimplemented) [env: FI_ES_USER=]
```


#### Example

```
$ flake-info --push \
--elastic-url http://localhost:5555 \
--elastic-index-name latest-21-21.05
--elastic-schema-version 21 group ./examples/ngi-nix.json ngi-nix
```

Currently `github` and `gitlab` can be used as source repos with hash. the `hash` attribute defines the fetched git reference (branch, commit, tag, etc).

## Installation

Expand Down
Loading

0 comments on commit 783aabf

Please sign in to comment.