invalid reuse of reader locktable slot

[huahaiy]

Keep an old connection, use a new connection to transact, close the db. Try query on old connection, this error appears. Run the query second time, "transaction has not been reset" is thrown.

[Update] This is an expected behavior. See below.

[huahaiy]

http://www.lmdb.tech/doc/starting.html

LMDB uses POSIX locks on files, and these locks have issues if one process opens a file multiple times. Because of this, do not mdb_env_open() a file multiple times from a single process. Instead, share the LMDB environment that has opened the file across all threads. Otherwise, if a single process opens the same environment multiple times, closing it once will remove all the locks held on it, and the other instances will be vulnerable to corruption from other processes.

So making multiple lmdb/open-lmdb calls on the same db is not recommended. Particularly, if one closes, the rests are impacted as well, and will likely throw this error. So if you have to open multiple times, don't close them, until the program exits. Best yet, don't open multiple times.

[vkz]

Hi. Been pulling my hair until thought of checking the issues. This may still be a problem and IMO at least a note in the docs may help others. Easiest way to reproduce is something like this code (as you mentioned at the beginning):

(let [conn (store/connect "bob")]
  (let [conn (store/connect "bob")]
    (store/transact! conn [{:foo/boo "boo"}])
    (store/close conn))
  (store/transact! conn [{:foo/baz "baz"}])
  (store/close conn))

While one is unlikely to write this by hand, the following is a typical way to manage resources:

(store/with [conn "bob"]
  (store/with [conn "bob"]
    (store/transact! conn [{:foo/boo "boo"}]))
  (store/transact! conn [{:foo/baz "baz"}]))

and will roughly expand into the first snippet.

In fact even these macros are likely to occur in some function calls:

(store/with [conn "bob"]
  (perform-query that calls store/with itself)
  ... other queries and transactions)

With few databases one can have a service that simply holds on to connections. With lots of them however you need to manage resources somehow. Argument can be made that it isn't Datalevin's place to deal with it, and I think that's fair. People may choose to do their own connection pooling or ref counting or smth, but they need to know they have to do it. Another option is to provide some lower level API to LMDB "environment" but I know nothing about LMDB to make reasonable suggestions.

[huahaiy]

Thank you for mentioning this. I have updated the doc string and exception message to make it more clear.

with cannot be used like a with-open though, so your sample code won't work anyway. However, there is a danger that people may think with is like with-open. These functions are inherited from Datascript and I don't think they make sense for Datalevin, I am inclined to remove them, or at least disappear them from the API doc, so people don't them.

[vkz]

Thank you.

Re with I should've phrased it better. It has nothing to do with datalevin's with. It is my own macro roughly similar to with-open e.g.

(defmacro with
  ""
  {:style/indent [1 [[:defn]] :form]}
  [[id conn] & body]
  `(let [conn# ~conn
         ~id (connect conn#)]
     (let [res# (do ~@body)
           forced# (if (coll? res#) (doall res#) res#)]
       ;; Only close a connection that we open here. Connection established outside "with" and passed
       ;; in as an argument should remain opened.
       (when-not (d/conn? conn#)
         ;; only close if still opened
         (when (d/conn? ~id)
           (d/close ~id)))
       forced#)))

issue I wanted to bring up had to do with someone, like me, writing something similar to the above thinking that you could open and close connections at will, when in fact semantically managing connections as resource is not straightforward with limitations and requirements imposed by LMDB implementation.

[huahaiy]

Thank you for the clarification. However, I don't think Datalevin or LMDB is special in anyway compared with any other stateful database.

It might be sufficient to tell people that Datalevin should be managed like any other stateful resources and give recommendation on what to use, i.e. a component like life cycle management tool that is commonly used for these use cases.

I would submit that only a very small minority of Clojure users would think to write their own macros to do this kind of things. If they can write their own macro to manage a database connection, it is not that hard to implement something simple to reuse the connections.

On the other hand, Datalevin can probably provide a with-conn macro, so that people do not have to roll their own. This will be similar to what clojure jdbc provides.

[huahaiy]

Release 0.3.9 added with-conn macro and get-conn function to reuse the same connection to the same database, if desired.