Org 9.7's deferred element property resolution breaks dynamic blocks

[jezcope]

OS/platform

Linux

Emacs version and provenance

emacs 29.3 (PGTK) installed from nixpkgs

Emacs command

emacs

Org version and provenance

org mode 9.7, installed via straight with doom emacs

org-ql package version and provenance

org-ql 0.8.6 (c937098), installed via straight with doom emacs

Actions taken

Create the following minimal example file:

* Dynamic block

#+begin: org-ql :query "todo:"
#+end:

* Tasks

** TODO First thing
** DONE Thing the second
** DOING Thing 3

Place the point on the dynamic block and press C-c C-x C-u.

Observed results

A corrupted table is inserted into the dynamic blog:

* Dynamic block

#+begin: org-ql :query "todo:"
| Heading | Todo | P |   |   |
|---------+------+---+---+---|
| [[P        | ][P    | ]]  |   |   |
| [[sks][sks]]     |      |   |   |   |
#+end:

* Tasks

** TODO First thing
** DONE Thing the second
** DOING Thing 3

In this example, no messages are shown in the echo area other than Updating dynamic block ‘org-ql’ at line 3...done. Depending on the content of the file, errors may occur. The one I ran into was "Empty link" from org-ql-search--org-make-link-string at org-ql-search.el#L334.

Expected results

Dynamic block is correctly filled with the matching tasks:

* Dynamic block

#+begin: org-ql :query "todo:"
| Heading                      | Todo  | P |
|------------------------------+-------+---|
| [[First thing][First thing]] | TODO  |   |
| [[Thing 3][Thing 3]]         | DOING |   |
#+end:

* Tasks

** TODO First thing
** DONE Thing the second
** DOING Thing 3

Backtrace

No response

Etc.

I ran into this completely incidentally while doing a minimal implementation of #151. The cause seems to be this sequence of events:

1. User runs org-dblock-update
2. Org then deletes the content of the dynamic block, per https://orgmode.org/manual/Dynamic-Blocks.html
3. org-dblock-write:org-ql obtains a list of elements using org-ql-query: note that these are stored with references to character positions in the buffer, and some element properties are deferred until requested
4. org-dblock-write:org-ql inserts the table header, which changes the character positions of all text following it
5. org-dblock-write:org-ql loops over the elements fetching properties using character positions which have been invalidated by (4), with unpredictable results

The fix seems to be to build the table as a string first and then insert it all together, so that the character positions used in the element structures remain valid. Alternatively, org-element-properties-resolve could be called on each element at some point before the first call to insert. I have implemented a quick fix so I'm happy to make a PR.

[alphapapa]

Indeed, I can confirm that this happens when using Org 9.7, but not with Org 9.6. So this is some of the fallout from the deferred-element-properties change. I've been wondering how many of these kinds of bugs are going to get reported on my packages now.

It's additionally complicated by the fact that we must also support older Org versions that don't defer property resolution. @yantar92 What is the recommended solution to this problem?

I have implemented a quick fix so I'm happy to make a PR.

You're welcome to propose something, but please see https://github.com/alphapapa/org-ql?tab=readme-ov-file#copyright-assignment

[yantar92]

Adam Porter ***@***.***> writes:

Indeed, I can confirm that this happens when using Org 9.7, but not with Org 9.6. So this is some of the fallout from the deferred-element-properties change. I've been wondering how many of these kinds of bugs are going to get reported on my packages now. It's additionally complicated by the fact that we must also support older Org versions that don't defer property resolution. @yantar92 What is the recommended solution to this problem?

You can filter the element list through `org-element-properties-resolve', when it is available. That way, you will get identical behavior with Org 9.6. Note, however, that using values returned by `org-element-at-point' is generally not reliable once you modify the original buffer - the positions will no longer be valid.

…

-- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at <https://orgmode.org/>. Support Org development at <https://liberapay.com/org-mode>, or support my work at <https://liberapay.com/yantar92>

[alphapapa]

@yantar92 Thank you.

[alphapapa]

This should be fixed on v0.8.7 and master now. Thanks.