- [KV] transacting data more than one order of magnitude larger than the initial map size in one transaction. #196
- [Pod] serialize TxReport to regular map. #190
- [Server] migrate old sessions that do not have
:last-active
.
- [Datalog]
datalog-index-cache-limit
function to get/set the limit of Datalog index cache. Helpful to disable cache when bulk transacting data. #195 - [Server]
:idle-timeout
option when creating the server, in ms, default is 24 hours. #122
- [Datalog] error when Clojure collections are used as lookup refs. #194
- [Datalog] correctly handle retracting then transacting the same datom in the same transaction. #192
- [Datalog] error deleting entities that were previously transacted as part of some EDN data. #191.
- [Lib] update deps.
- [KV] added tuple data type that accepts a vector of scalar values. This supports range queries, i.e. having expected ordering by first element, then second element, and so on. This is useful, for example, as path keys for indexing content inside documents. When used in keys, the same 511 bytes limitation applies.
- [Datalog] added heterogeneous tuple
:db/tupleTypes
and homogeneous tuples:db/tupleType
type. Unlike Datomic, the number of elements in a tuple are not limited to 8, as long as they fit inside a 496 bytes buffer. In addition, instead of usingnil
to indicate minimal value like in Datomic, one can use:db.value/sysMin
or:db.value/sysMax
to indicate minimal or maximal values, useful for range queries. #167 - [Main] dynamic var
*datalevin-data-readers*
to support loading custom tag literals. (thx @respatialized)
- [Main] dump and load big integers.
- [Datalog] avoid unnecessary caching, improve transaction speed up to 25% for large transactions.
- [Native] upgrade Graalvm to 22.3.1
- [Native] static build on Linux. #185
- [Lib] update deps.
- [Datalog] error when large
:db/fulltext
value is added then removed in the same transaction.
- [Search]
search-utils/create-ngram-token-filter
now works. #164 - [Datalog] large datom value may throw off full-text indexing. #177
- [Datalog] intermittent
:db/fulltext
values transaction error. #177
- [Search] Breaking search index storage format change. Data re-indexing is necessary.
- [Search] significant indexing speed and space usage improvement: for default
setting, 5X faster bulk load speed; 2 orders of magnitude faster
remove-doc
and 10X disk space reduction; when term positions and offsets are indexed: 3X faster bulk load and 40 percent space reduction. - [Search] added caching for term and document index access, resulting in 5 percent query speed improvement on average, 35 percent improvement at median.
- [Search]
:index-position?
option to indicate whether to record term positions inside documents, defaultfalse
. - [Search]
:check-exist?
argument toadd-doc
indicate whether to check the existence of the document in the index, defaulttrue
. Set it tofalse
when importing data to improve ingestion speed.
- [Datalog] increasing indexing time problem for
:db/fulltext
values. #151 - [Search] error when indexing huge documents.
- [KV] spillable results exception in certain cases.
- [Search]
doc-refs
function. - [Search]
search-index-writer
as well as relatedwrite
andcommit
functions for client/server, as it makes little sense to bulk load documents across network.
- [Native] allow native compilation on apple silicon
- [Datalog] db print-method. (thx @den1k)
- [Datalog] intermittent concurrent transaction problems
- [CI] adjust CI workflow for the latest Graalvm
- [Datalog] moved entity and transaction ids from 32 bits to 64 bits integers, supporting much larger DB. #144
- [Datalog] wrapped
transact!
insidewith-transaction
to ensure ACID and improved performance - [Native] updated to the latest Graalvm 22.3.0. #174
- [KV]
get-range
regression when results are used insequence
. #172
- [Datalog] Ported all applicable Datascript improvements since 0.8.13 up to now (1.4.0). Notably, added composite tuples feature, new pull implementation, many bug fixes and performance improvements. #3, #57, #168
- bump deps
- [Server] error when granting permission to a db created by
create-database
instead of being created by opening a connection URI
- [Datalog] avoid printing all datoms when print a db
- [Doc] clarify that
db-name
is unique on the server. (thx @dvingo)
- avoid
(random-uuid)
, since not every one is on Clojure 1.11 yet.
- typo prevent build on CI
- [KV] spill test that prevents tests on MacOS CI from succeeding.
- [KV] broken deleteOnExit for temporary files
- [KV] clean up spill files
- [Platform] embedded library support for Apple Silicon.
- [KV] A new range function
range-seq
that has similar signature asget-range
, but returns aSeqable
, which lazily reads data items into memory in batches (controlled by:batch-size
option). It should be used insidewith-open
for proper cleanup. #108 - [KV] The existent eager range functions,
get-range
andrange-filter
, now automatically spill to disk when memory pressure is high. The results, though mutable, still implementIPersistentVector
, so there is no API level change. The spill-to-disk behavior is controlled byspill-opts
option map when opening the db, allowing:spill-threshold
and:spill-root
options.
- [KV] write performance improvement
- [KV] Upgrade LMDB to 0.9.29
- [Client]
:client-opts
option map that is passed to the client when opening remote databases.
- [KV]
with-transaction-kv
does not drop prior data when DB is resizing. - [Datalog]
with-transaction
does not drop prior data when DB is resizing.
- [Native] Add github action runner image ubuntu-20.04 to avoid using too new a glibc version (2.32) that does not exist on most people's machines.
- [KV]
with-transaction-kv
does not crash when DB is resizing.
- [KV]
with-transaction-kv
macro to expose explicit transactions for KV database. This allows arbitrary code within a transaction to achieve atomicity, e.g. to implement compare-and-swap semantics, etc, #110 - [Datalog]
with-transaction
macro, the same as the above for Datalog database - [KV]
abort-transact-kv
function to rollback writes from within an explicit KV transaction. - [Datalog]
abort-transact
function, same for Datalog transaction. - [Pod] Missing
visit
function
- [Server] Smaller memory footprint
- bump deps
- [Datalog]
fulltext-datoms
function that return datoms found by full text search query, #157
- [Search] Don't throw for blank search query, return
nil
instead, #158 - [Datalog] Correctly handle transacting empty string as a full text value, #159
- Datalevin is now usable in deps.edn, #98 (thx @ieugen)
- [Datalog] Caching issue introduced in 0.6.20 (thx @cgrand)
- [Pod]
entity
andtouch
function to babashka pod, these return regular maps, as theEntity
type does not exist in a babashka script. #148 (thx @ngrunwald) - [Datalog]
:timeout
option to terminate on deadline for query/pull. #150 (thx @cgrand).
- [Datalog] Entity equality requires DB identity in order to better support reactive applications, #146 (thx @den1k)
- bump deps
- [Search] corner case of search in document collection containing only one term, #143
- [Datalog] entity IDs has smaller than expected range, now they cover full 32 bit integer range, #140
- [Datalog] Persistent
max-tx
, #142
- [Datalog]
tx-data->simulated-report
to obtain a transaction report without actually persisting the changes. (thx @TheExGenesis) - [KV] Support
:bigint
and:bigdec
data types, corresponding tojava.math.BigInteger
andjava.math.BigDecimal
, respectively. - [Datalog] Support
:db.type/bigdec
and:db.type/bigint
, correspondingly, #138.
- Better documentation so that cljdoc can build successfully. (thx @lread)
- [Datalog] Additional arity to
update-schema
to allow renaming attributes. #131 - [Search]
clear-docs
function to wipe out search index, as it might be faster to rebuild search index than updating individual documents sometimes. #132 datalevin.constants/*data-serializable-classes*
dynamic var, which can be used forbinding
if additional Java classes are to be serialized as part of the default:data
data type. #134
- [Datalog] Allow passing option map as
:kv-opts
to underlying KV store whencreate-conn
- bump deps
- [Datalog]
clear
function on server. #133
- [Datalog] Changed
:search-engine
option map key to:search-opts
for consistency [Breaking]
- [Search] Handle empty documents
- [Datalog] Handle safe schema migration, #1, #128
- bump deps
- [KV] visitor function for
visit
can return a special value:datalevin/terminate-visit
to stop the visit.
- Fixed adding created-at schema item for upgrading Datalog DB from prior 0.6.4 (thx @jdf-id-au)
- [breaking] Simplified
open-dbi
signature to take an option map instead
:validate-data?
option foropen-dbi
,create-conn
etc., #121
- Schema update regression. #124
:domain
option tonew-search-engine
, so multiple search engines can coexist in the samedir
, each with its own domain, a string. #112
- Server failure to update max-eid regression, #123
- Added an arity to
update-schema
to allow removal of attributes if they are not associated with any datoms, #99
- Search add-doc error when alter existing docs
- Persistent server session that survives restarts without affecting clients, #119
- More robust server error handling
- Query cache memory leak, #118 (thx @panterarocks49)
- Entity retraction not removing
:db/updated-at
datom, #113
datalevin.search-utils
namespace with some utility functions to customize search, #105 (thx @ngrunwald)
- Add
visit
KV function tocore
name space - Handle concurrent
add-doc
for the same doc ref
- Bump deps
- Handle Datalog float data type, #88
- Allow to use all classes in Babashka pods
- Dump and load Datalog option map
- Dump and load
inter-fn
- Dump and load regex
- Pass search engine option map to Datalog store
- Make configurable analyzer available to client/server
- Dot form Java interop regression in query, #103
- Option to pass an analyzer to search engine, #102
:auto-entity-time?
Datalog DB creation option, so entities can optionally have:db/created-at
and:db/updated-at
values added and maintained automatically by the system during transaction, #86
- Dependency bump
doc-count
function returns the number of documents in the search indexdoc-refs
function returns a seq ofdoc-ref
in the search index
datalevin.core/copy
function can copy Datalog database directly.
doc-indexed?
function
add-doc
can update existing docopen-kv
function allows LMDB flags, #100
- Built-in full-text search engine, #27
- Key-value database
visit
function to do arbitrary things upon seeing a value in a range
- [breaking]
:instant
handles dates before 1970 correctly, #94. The storage format of:instant
type has been changed. For existing Datalog DB containing:db.type/instant
, dumping as a Datalog DB using the old version of dtlv, then loading the data is required; For existing key-value DB containing:instant
type, specify:instant-pre-06
instead to read the data back in, then write them out as:instant
to upgrade to the current format.
- Improve read performance by adding a cursor pool and switch to a more lightweight transaction pool
- Dependency bump
- Create pod client side
defpodfn
so it works in non-JVM.
load-edn
for dtlv, useful for e.g. loading schema from a file, #101
- Serialized writes for concurrent transactions, #83
defpodfn
macro to define a query function that can be used in babashka pod, #85
- Update
max-aid
after schema update (thx @den1k)
- Updated dependencies, particularly fixed sci version (thx @borkdude)
- occasional LMDB crash during multiple threads write or abrupt exit
- Update graalvm version
- Exception handling in copy
- Handle scalar result in remote queries
- Server asks client to reconnect if the server is restarted and client reconnects automatically when doing next request
- Bump versions of all dependency
- More robust handling of abrupt network disconnections
- Automatically maintain the required number of open connections, #68
- Options to specify the number of connections in the client connection pool and to set the time-out for server requests
- Backport the dump/load fix from 0.5.20
- Dumping/loading Datalog store handles raw bytes correctly
- Remove client immediately when
disconnect
message is received, clean up resources afterwards, so a logically correct number of clients can be obtained in the next API call on slow machines.
- Occasional server message write corruptions in busy network traffic on Linux.
- JVM uberjar release for download
- JVM library is now Java 8 compatible, #69
- Auto switch to local transaction preparation if something is wrong with remote preparation (e.g. problem with serialization)
- Do most of transaction data preparation remotely to reduce traffic
- Handle entity serialization, fix #66
- Allow a single client to have multiple open databases at the same time
- Client does not open db implicitly, user needs to open db explicitly
- New
create-conn
should override the old, fix #65
DTLV_LIB_EXTRACT_DIR
environment variable to allow customization of native libraries extraction location.
- Use clj-easy/graal-build-time, in anticipation of GraalVM 22.
- Better robust jar layout for
org.clojars.huahaiy/datalevin-native
- Release artifact
org.clojars.huahaiy/datalevin-native
on clojars, for depending on Datalevin while compiling GraalVM native image. User no longer needs to manually compile Datalevin C libraries.
- Only check to refersh db cache at user facing namespaces, so internal db calls work with a consistent db view
- Replace unnecessary expensive calls such as
db/-search
ordb/-datoms
with cheaper calls to improve remote store access speed. - documentation
- More robust build
- Wrap all LMDB flags as keywords
- Don't do AOT in library, to avoid deps error due to exclusion of graal
- Expose all LMDB flags in JVM version of kv store
- Transparent networked client/server mode with role based access control. #46 and #61
dtlv exec
takes input from stdin when no argument is given.
- When open db, throw exception when lacking proper file permission
- Transactable entity [Thanks @den1k, #48]
clear
function to clear Datalog db
- Native uses the same version of LMDB as JVM, #58
- Remove GraalVM and dtlv specific deps from JVM library jar
- Update deps
- More robust dependency management
- Replacing giant values, this requires Java 11 [#56]
- Transaction of multiple instances of bytes [#52, Thanks @den1k]
- More reflection config in dtlv
- Benchmark deps
- Correct handling of rule clauses in dtlv
- Documentation clarification that we do not support "db as a value"
- Datafy/nav for entity [Thanks @den1k]
- Some datom convenience functions, e.g.
datom-eav
,datom-e
, etc.
- Talk to Babashka pods client in transit+json
- Exposed more functions to Babashka pod
- Native Datalevin can now work as a Babashka pod
- Compile to native on Windows and handle Windows path correctly
close-db
convenience function to close a Datalog db
- Compile to Java 8 bytecode instead of 11 to have wider compatibility
- Use UTF-8 throughout for character encoding
- Improve dtlv REPL (doc f) display
- Provide Datalevin C source as a zip to help compiling native Datalevin dependency
- Minor improvement on the command line tool
- Native image now bundles LMDB
- Handle list form in query properly in command line shell [#42]
- Consolidated all user facing functions to
datalevin.core
, so users don't have to understand and require different namespaces in order to use all features.
- [Breaking] Removed AEV index, as it is not used in query. This reduces storage and improves write speed.
- [Breaking] Change VAE index to VEA, in preparation for new query engine. Now all indices have the same order, just rotated, so merge join is more likely.
- [Breaking] Change
open-lmdb
andclose-lmdb
toopen-kv
andclose-kv
,lmdb/transact
tolmdb/transact-kv
, so they are consistent, easier to remember, and distinct from functions indatalevin.core
.
- GraalVM native image specific LMDB wrapper. This wrapper allocates buffer memory in C and uses our own C comparator instead of doing these work in Java, so it is faster.
- Native command line shell,
dtlv
- Improve Java interop call performance
- Allow Java interop calls in where clauses, e.g.
[(.getTime ?date) ?timestamp]
,[(.after ?date1 ?date2)]
, where the date variables are:db.type/instance
. [#32]
- Changed default LMDB write behavior to use writable memory map and asynchronous msync, significantly improved write speed for small transactions (240X improvement for writing one datom at a time).
- Read
:db.type/instant
value asjava.util.Date
, not aslong
[#30]
- Fixed error when transacting different data types for an untyped attribute [#28, thx @den1k]
- proper exception handling in
lmdb/open-lmdb
- Fixed schema update when reloading data from disk
- Fixed
core/get-conn
schema update
- Remove unnecessary locks in read transaction
- Improved error message and documentation for managing LMDB connection
core/get-conn
andcore/with-conn
- Correctly handle
init-max-eid
for large values as well.
- Fixed regression introduced by 0.3.6, where :ignore value was not considered [#25]
- Add headers to key-value store keys, so that range queries work in mixed data tables
- Expose all data types to key-value store API [#24]
- thaw error for large values of
:data
type. [#23]
- portable temporary directory. [#20, thx @joinr]
- Properly initialize max-eid in
core/empty-db
- Add value type for
:db/ident
in implicit schema
- [Breaking] Change argument order of
core/create-conn
,db/empty-db
etc., and putdir
in front, since it is more likely to be specified thanschema
in real use, so users don't have to putnil
forschema
.
- correct
core/update-schema
- correctly handle
false
value as:data
- always clear buffer before putting data in
- thaw exception when fetching large values
- clearer error messages for byte buffer overflow
- correct schema update
core/schema
andcore/update-schema
core/closed?
db/entid
allows 0 as eid
- fix test
- correct results when there are more than 8 clauses
- correct query result size
- automatically re-order simple where clauses according to the sizes of result sets
- change system dbi names to avoid potential collisions
- miss function keywords in cache keys
- hash-join optimization submitted PR #362 to Datascript
- caching DB query results, significant query speed improvement
- fix invalid reuse of reader locktable slot #7
- remove MDB_NOTLS flag to gain significant small writes speed
- update existing schema instead of creating new ones
- Reset transaction after getting entries
- Only use 24 reader slots
- avoid locking primitive #5
- create all parent directories if necessary
- long out of range error during native compile
- apply query/join-tuples optimization
- use array get wherenever we can in query, saw significant improvement in some queries.
- use
db/-first
instead of(first (db/-datom ..))
,db/-populated?
instead of(not-empty (db/-datoms ..)
, as they do not realize the results hence faster. - storage test improvements
- use only half of the reader slots, so other processes may read
- add an arity for
bits/read-buffer
andbits/put-buffer
- add
lmdb/closed?
,lmdb/clear-dbi
, andlmdb/drop-dbi
- code samples
- API doc
core/close
- Port datascript 0.18.13