Tags · HazyResearch/deepdive

v0.9.0rc2

DeepDive 0.9.0 Release Candidate 2

fixing some Python 3 compatibility issues and
discrepancies in the test vs. actual environment

Nov 16, 2016
85f900b
zip
tar.gz

v0.9.0rc1

DeepDive 0.9.0 Release Candidate 1

Nov 13, 2016
5301b41
zip
tar.gz

UNSTABLE

Merge branch 'master' of https://github.com/HazyResearch/deepdive int…

…o pgtsv-unicode

Jul 5, 2016
a10d96a
zip
tar.gz
Notes
Downloads

v0.8-STABLE

Updates mindbender for ddlite

Apr 26, 2016
28a58de
zip
tar.gz
Notes
Downloads

v0.8.0

DeepDive 0.8.0

==============

A completely re-architected version of DeepDive is here.
Now the system [compiles an execution plan](doc/ops-compiling.md) ahead of time, checkpoints at a much finer granularity, and gives users [full visibility and control of the execution](doc/ops-execution.md), so any parts of the computation can be flexibly repeated, resumed, or optimized later.
The new architecture naturally enforces modularity and extensibility, which enables us to innovate most parts independently without having to understand every possible combination of the entire code.
The abstraction layers that encapsulate [database operations](database/db-driver/) as well as [compute resources](runner/compute-driver/) are now clearly established, giving a stable ground for extensions in the future that support more types of database engines and compute clusters such as Hadoop/YARN and ones with traditional job schedulers.

As an artifact of this redesign, exciting performance improvements are now observed:

* The database drivers show more than 20x higher throughput (2MB/s -> 50MB/s, per connection) with zero storage footprint by streaming data in and out of UDFs.
* The grounded factor graphs save up to 100x storage space (12GB -> 180MB) by [employing compression](#450) during the factor graph's grounding and loading, incurring less than 10% overhead in time (400s -> 460s, measuring only the dumping and loading, hence a much smaller fraction in practice).

See the [issues and pull requests for this milestone on GitHub](https://github.com/HazyResearch/deepdive/issues?page=1&q=milestone%3A%22DeepDive+0.8.0%22) (most notably [#445](#445)) for further details.

## New commands and features

An array of new commands have been added to `deepdive`, and existing ones have been rewritten, such as `deepdive initdb` and `deepdive run`.

* [Compilation](doc/ops-compiling.md)/[Execution](doc/ops-execution.md)
    * `deepdive compile`
    * `deepdive plan`
    * `deepdive do`
    * `deepdive redo`
    * `deepdive mark`
    * `deepdive done`

* [Learning/Inference](doc/ops-model.md)
    * `deepdive model`

* [Data management](doc/ops-data.md)
    * `deepdive create`
    * `deepdive load`
    * `deepdive unload`
    * `deepdive query`
    * `deepdive db`

* [UDFs](doc/debugging-udf.md)
    * `deepdive check`
    * `deepdive compute`
    * [`@tsv_extractor`, `@returns` Python decorators](doc/writing-udf-python.md#writing-udfs-in-python) for parsing and formatting in UDFs.

* Interactive tools

    The bundled Mindbender can now automatically construct a [search and browsing interface from DDlog annotations](doc/browsing.md).
    [Documentation for Dashboard](doc/dashboard.md) has been added.

    * `mindbender search`
    * `mindbender dashboard`
    * `mindbender snapshot`
    * `mindbender tagger`

* Miscellaneous
    * `deepdive whereis`

To learn more about individual `deepdive COMMAND`, use the following `deepdive help` command.

```bash
deepdive help COMMAND
```

## Dropped and deprecated features

Scala code base has been completely dropped and rewritten in Bash and [jq](https://stedolan.github.io/jq/).
Many superfluous features have been dropped and are deprecated to be dropped as summarized below:

* All other extractor style than `tsv_extractor`, `sql_extractor`, and `cmd_extractor` have been dropped, namely:
    * `plpy_extractor`
    * `piggy_extractor`
    * `json_extractor`

* Manually writing `deepdive.conf` is strongly discouraged as filling in more fields such as `dependencies:` and `input_relations:` became mandatory.
    [Rewriting them in DDlog](doc/development-cycle.md#1-write) is strongly recommended.

* Database configuration in `deepdive.db.default` is completely ignored.
    [`db.url`](doc/deepdiveapp.md#db-url) must be used instead.

* `deepdive.extraction.extractors.*.input` in `deepdive.conf` should always be SQL queries.
    `TSV(filename.tsv)` or `CSV(filename.csv)` no longer supported.

Feb 19, 2016
ec64859
zip
tar.gz
Notes
Downloads

v0.7.1

DeepDive 0.7.1

* Adds better support for applications written in DDlog.
    `deepdive run` now runs DDlog-based applications (`app.ddlog`).

* Makes PL/Python extension no longer necessary for PostgreSQL.
    It is still needed for Greenplum and PostgreSQL-XL.

* Adds `deepdive sql eval` command now supports `format=json`.

* Adds `deepdive load` command for loading TSV and CSV data.

* Adds `deepdive help` command for quick usage instructions for `deepdive` command.

* Includes the latest Mindbender with the Search GUI for [browsing data produced by DeepDive](../basics/browsing.html).

* Adds various [bug fixes and improvements](https://github.com/HazyResearch/deepdive/issues?q=milestone%3A%22DeepDive+0.7.1%22).