Skip to content

Latest commit

 

History

History
1055 lines (927 loc) · 68.1 KB

notes.org

File metadata and controls

1055 lines (927 loc) · 68.1 KB

Tasks

Move/rename repo to org-ql

Since org-sidebar uses it, it probably makes more sense for org-ql to be the primary focus of the repo. But it also uses org-agenda-ng--format-element, so org-agenda-ng probably should be part of that package, too.

Document matchers/selectors/predicates

And maybe pick a single name for them…

deadline
Be sure to explain how it works with regard to the implied date and today.
date, scheduled
No implied date, but supports today.

Document sorters

Add more sorters?

  • [ ] category
[ ] Any date
e.g. it would search for timestamps (active/inactive?) anywhere in an entry

MAYBE Date predicate that searches entire entry

Because the existing ones only search the special date property line.

Document/figure out tag inheritance

I think it should probably be enabled in most cases, to avoid missing results that users would expect to find, but it will reduce performance in some cases, so users should be able to turn it off when they don’t need it.

[2018-06-12 Tue 14:32] The docstring for org-map-entries says:

If your function needs to retrieve the tags including inherited tags at the current entry, you can use the value of the variable ‘org-scanner-tags’ which will be much faster than getting the value with ‘org-get-tags-at’. If your function gets properties with ‘org-entry-properties’ at the current entry, bind ‘org-trust-scanner-tags’ to t around the call to ‘org-entry-properties’ to get the same speedup. Note that if your function moves around to retrieve tags and properties at a different entry, you cannot use these techniques.

Ideas

SOMEDAY Dual matching with regexp and predicates

Searching and matching could be sped up by constructing a regexp that searches directly to the next possible match, and then matching with predicate functions.

For example, a search like:

(org-ql (org-agenda-files)
  (and (regexp "lisp")
       (scheduled < today)))

Only entries that contain the word lisp can be matches, and searching each entry for that word is wasteful. Instead, we could search the buffer for the next occurrence of lisp, then check the scheduled date for that entry.

This would require processing the predicate to pull out matchers that can be done as buffer-wide regexps, e.g. regexp, heading-regexp, todo, and possibly tags. Org has some regexp-building functions that might make this fairly easy, and then we could probably use rx to make an optimized version of the regexp. It would also require some refactoring to the searching that would go directly to regexp matches when possible, rather than checking every entry with the predicate.

Operate on list of heading positions

  • State “DONE” from [2018-05-10 Thu 15:02]

[2017-12-31 Sun 17:54] I wonder if, instead of parsing the whole buffer with org-element-parse-buffer, we could simply work on a list of heading positions, e.g. a loop would search forward to the next heading position, then call whatever predicates it needed at the heading’s position, using save-excursion around each function call. The predicates would need to be updated to get their data from the buffer, instead of using org-element-property, but that wouldn’t be hard.

[2018-05-10 Thu 15:01] I already changed to using buffer-parsing predicates instead of org-element-parse-buffer.

Use macros for date

  • State “DONE” from [2018-05-10 Thu 14:59]

If I made the date selector a macro, I could avoid the need to quote the comparator.

Also, maybe instead of having a single date selector, I should have scheduled, deadline, etc.

Byte-compile lambdas

  • State “DONE” from [2018-05-09 Wed 17:30]

elfeed-search--update-list byte-compiles lambdas returned by elfeed-search-compile-filter. Maybe I could do something like this too.

If I can get this working, I should profile it to see what difference it makes.

Profiling

Going to try byte-compiling the predicate function:

(elp-profile 10 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
                 (and (or (date :date '= (org-today))
                          (date :deadline '<= (+ org-deadline-warning-days (org-today)))
                          (date :scheduled '<= (org-today)))
                      (not (apply #'todo org-done-keywords-for-agenda)))))
FunctionTimes calledTotal timeAverage time
org-agenda-ng–agenda100.83705810390.0837058104
org-agenda-finalize-entries100.6528866080.0652886608
org-super-agenda–filter-finalize-entries100.6417945010.0641794501
org-super-agenda–group-items100.6360570060.0636057006
org-super-agenda–group-dispatch1300.6319118490.0048608603
org-super-agenda–group-tag500.5928838690.0118576773
list27200.57927951690.0002129704
mapcar3310.23335919200.0007050126
org-agenda-ng–filter-buffer100.092476260.009247626
org-agenda-ng–format-element1500.06493204790.0004328803
org-entry-get8600.04082853494.747…e-05
org-agenda-ng–date-p9100.03856462494.237…e-05
org-element-headline-parser1500.03744174700.0002496116
org-is-habit-p2700.02901073890.0001074471
org–property-local-values2700.02686159799.948…e-05
org-get-property-block2700.02446133099.059…e-05
org-get-tags-at1500.0178758640.0001191724
org-super-agenda–group-habit100.0159106560.0015910655
mapc25400.01586162906.244…e-06
org-agenda-ng–add-faces1500.01433296709.555…e-05

Now the same thing without byte-compiling:

(elp-profile 10 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
                 (and (or (date :date '= (org-today))
                          (date :deadline '<= (+ org-deadline-warning-days (org-today)))
                          (date :scheduled '<= (org-today)))
                      (not (apply #'todo org-done-keywords-for-agenda)))))
FunctionTimes calledTotal timeAverage time
org-agenda-ng–agenda100.8466455370.0846645537
org-agenda-finalize-entries100.6628968050.0662896805
sort400.5911232560.0147780814
org-entries-lessp4000.59012016200.0014753004
mapcar2010.23182705990.0011533684
org-agenda-ng–filter-buffer100.0925197870.0092519787
org-super-agenda–filter-finalize-entries100.06642780400.0066427804
org-agenda-ng–format-element1500.0646589940.0004310599
org-super-agenda–group-items100.06025040890.0060250408
org-super-agenda–group-dispatch1300.05619044700.0004322342
org-entry-get8600.04374588895.086…e-05
org-agenda-ng–date-p9100.03826234094.204…e-05
org-element-headline-parser1500.03746629200.0002497752
org-is-habit-p2700.03208610790.0001188374
org–property-local-values2700.02986904300.0001106260
org-get-property-block2700.02747166490.0001017469
org-super-agenda–group-habit100.0191179010.0019117901
org-get-tags-at1500.01789589300.0001193059
mapc24700.01503611306.087…e-06
org-agenda-ng–add-faces1500.01430921699.539…e-05

Virtually indistinguishable. Going to try moving the byte-compile call from the org-agenda-ng macro to other places…

(elp-profile 10 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
                 (and (or (date :date '= (org-today))
                          (date :deadline '<= (+ org-deadline-warning-days (org-today)))
                          (date :scheduled '<= (org-today)))
                      (not (apply #'todo org-done-keywords-for-agenda)))))
FunctionTimes calledTotal timeAverage time
org-agenda-ng–agenda100.84763167790.0847631678
mapcar3310.81594522200.0024650913
org-agenda-ng–filter-buffer100.6742179120.0674217912
org-element-headline-parser1500.61711958890.0041141305
line-beginning-position6200.58025796800.0009358999
org-agenda-finalize-entries100.0820651570.0082065157
org-super-agenda–filter-finalize-entries100.07087722790.0070877227
org-super-agenda–group-items100.0655231030.0065523103
org-agenda-ng–format-element1500.06527837400.0004351891
org-super-agenda–group-dispatch1300.06142535890.0004725027
org-entry-get8600.04940230295.744…e-05
org-agenda-ng–date-p9100.03884355194.268…e-05
org-is-habit-p2700.03756875490.0001391435
org–property-local-values2700.03538929290.0001310714
org-get-property-block2700.03297004400.0001221112
org-super-agenda–group-habit100.0244686010.0024468601
re-search-backward15000.01863440891.242…e-05
org-get-tags-at1500.01800388090.0001200258
mapc25400.01565180996.162…e-06
org-agenda-ng–add-faces1500.01441410809.609…e-05

Doesn’t seem to make any difference.

Examples / testing

(org-agenda-ng org-agenda-files
  (and (or (date :deadline '<= (org-today))
           (date :scheduled '<= (org-today)))
       (not (apply #'todo org-done-keywords-for-agenda)))
  ((group (tags "bills"))
   (group (todo "SOMEDAY"))))

(org-agenda-ng org-agenda-files
  (and (or (date :deadline '<= (org-today))
           (date :scheduled '<= (org-today)))
       (not (apply #'todo org-done-keywords-for-agenda))))

(org-agenda-ng "~/org/main.org"
  (and (or (date :deadline '<= (org-today))
           (date :scheduled '<= (org-today)))
       (not (apply #'todo org-done-keywords-for-agenda))))

(org-ql org-agenda-files
  (and (todo "SOMEDAY")
       (tags "Emacs")))
(org-ql org-agenda-files
  (and (todo "SOMEDAY")
       (tags "Emacs")
       (priority >= "B")))
(org-ql "~/org/main.org"
  (and (or (tags "Emacs")
           (priority >= "B"))
       (not (done))))
(org-ql "~/org/main.org"
  (and (or (tags "Emacs")
           (priority >= "B"))
       (done)))

Sorting

(org-ql "~/src/emacs/org-super-agenda/test/test.org"
  (regexp "over")
  :sort (priority deadline scheduled))

(org-ql "~/src/emacs/org-super-agenda/test/test.org"
  (regexp "over")
  :sort (date))

(org-ql "~/src/emacs/org-super-agenda/test/test.org"
  (todo)
  :sort (todo))

Regexp matching

(org-ql "~/src/emacs/org-super-agenda/test/test.org"
  (regexp "over"))

(org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
  (regexp "over"))

Property matching

(org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
  (property "agenda-group"))

(org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
  (property "agenda-group" "plans"))

Screenshot code

(org-super-agenda--test-with-org-today-date "2017-07-08 00:00"
  (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
    (and (or (date = today)
             (deadline <=)
             (scheduled <= today))
         (not (done)))))

Profiling

Using org-element-parse-buffer

This basically works, as a very basic kind of agenda view, but we can already see that it’s much slower (at least, for single-day views) because org-element-parse-buffer is slow compared to the agenda code.

[2018-05-10 Thu 15:03] Note: This is the old, much slower code that used org-element-parse-buffer.

Macro

(defmacro elp-profile (times prefixes &rest body)
  (declare (indent defun))
  (let ((prefixes (append '(org- string- s- buffer- append delq map
                                 list car save- outline- delete-dups
                                 sort line- nth concat char-to-string
                                 rx- goto- when search- re-)
                          prefixes)))
    `(let (output)
       (dolist (prefix ',prefixes)
         (elp-instrument-package (symbol-name prefix)))
       (dotimes (x ,times)
         ,@body)
       (elp-results)
       (elp-restore-all)
       (point-min)
       (forward-line 20)
       (delete-region (point) (point-max))
       (setq output (buffer-substring-no-properties (point-min) (point-max)))
       (kill-buffer)
       (delete-window)
       (let ((rows (s-lines output)))
         (append (list (list "Function" "Times called" "Total time" "Average time")
                       'hline)
                 (cl-loop for row in rows
                          collect (s-split (rx (1+ space)) row 'omit-nulls)))))))

[2018-05-09 Wed 17:31] Note: I seem to have misplaced the org-agenda-ng--test-agenda-today function I used in these tests.

ng

(elp-profile 1 (org-agenda-ng--test-agenda-today))

orig

Make sure to kill any existing agenda buffers first.

(elp-profile 1 (org-agenda-list nil nil 'week))

Profile org-element-map

(elp-profile 1 (with-current-buffer (find-buffer-visiting "~/org/main.org")
                 (org-element-parse-buffer 'headline)))

Profiling position-based

Macro

(defmacro elp-profile (times &rest body)
  (declare (indent defun))
  `(let ((prefixes '("org-" "string-" "s-" "buffer-" "append" "delq" "map"
                     "list" "car" "save-" "outline-" "delete-dups"
                     "sort" "line-" "nth" "concat" "char-to-string"
                     "rx-" "goto-" "when" "search-" "re-"))
         output)
     (dolist (prefix prefixes)
       (elp-instrument-package prefix))
     (dotimes (x ,times)
       ,@body)
     (elp-results)
     (elp-restore-all)
     (point-min)
     (forward-line 20)
     (delete-region (point) (point-max))
     (setq output (buffer-substring-no-properties (point-min) (point-max)))
     (kill-buffer)
     (delete-window)
     output))

orig

Make sure to kill any existing agenda buffers first.

(elp-profile 1 (org-agenda-list nil nil 'week))

ng-funcall

(elp-profile 5 (org-agenda-ng--test-agenda-today))

ng-flet

(elp-profile 5 (org-agenda-ng--test-agenda-today))

Profiling org-trust-scanner-tags

[2018-05-10 Thu 12:59] Turned on org-trust-scanner-tags, going to try profiling again:

;; (elp-profile 1 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
;;                      (tags "world")))

(elp-profile 10 nil (org-agenda-ng org-agenda-files
                     (tags "Emacs")))
FunctionTimes calledTotal timeAverage time
org-agenda-ng–agenda1044.0925982824.4092598282
mapcar28240.2345167070.1426755911
org-agenda-ng–filter-buffer8026.8954924710.3361936558
org-element-headline-parser398010.3876143620.0026099533
org-agenda-finalize-entries109.1944582520.9194458252
org-agenda-ng–tags-p702508.18973798490.0001165799
org-agenda-ng–format-element39806.59443256790.0016568926
outline-next-heading703206.11901804908.701…e-05
re-search-forward970505.87064678296.049…e-05
org-get-tags-at742305.40781580597.285…e-05
org-super-agenda–filter-finalize-entries105.23201234000.5232012340
org-super-agenda–group-items105.12609592100.5126095921
org-super-agenda–group-dispatch1305.1193336240.0393794894
sort203.82043685690.1910218428
org-element–parse-objects61803.53865789290.0005725983
org-is-habit-p59703.24977559200.0005443510
org-entry-get59703.23479640490.0005418419
org–property-local-values59703.17963573190.0005326023
org-get-property-block59703.07679196800.0005153755
org-entries-lessp200202.65639600790.0001326871

Now trying again without it:

;; (elp-profile 1 nil (org-agenda-ng "~/src/emacs/org-super-agenda/test/test.org"
;;                      (tags "world")))

(elp-profile 10 nil (org-agenda-ng org-agenda-files
                     (tags "Emacs")))
FunctionTimes calledTotal timeAverage time
mapcar179157.0963045380.0318795670
org-agenda-ng–agenda1054.2321335065.4232133505
org-agenda-ng–filter-buffer8030.0651670400.3758145880
org-get-tags-at7423013.8402024950.0001864502
org-agenda-ng–format-element398013.4292977970.0033741954
org-element-headline-parser398012.7717766520.0032089891
org-agenda-finalize-entries109.14394339900.9143943399
org-agenda-ng–tags-p702509.02496537300.0001284692
org-super-agenda–filter-finalize-entries107.3005158590.7300515859
outline-next-heading703207.23844356490.0001029357
org-super-agenda–group-items104.9185858550.4918585855
org-super-agenda–group-dispatch1304.91258935090.0377891488
re-search-forward1010204.62948238504.582…e-05
org-up-heading-safe73704.46298856200.0006055615
org-is-habit-p59604.27723519100.0007176569
org-entry-get59604.25953508000.0007146870
org-super-agenda–group-tag503.89420449290.0778840898
re-search-backward261503.36600834900.0001287192
org–property-local-values59603.17934763290.0005334475
org-get-property-block59603.06624259790.0005144702

Wow, using org-trust-scanner-tags saves a lot of time.

Profiling flet across all agenda files

Without flet

(elp-profile 5 (org-agenda-ng--agenda
                :files org-agenda-files
                :pred (lambda ()
                        (and (org-agenda-ng--todo-p)
                             (or (org-agenda-ng--date-p :deadline '<= (org-today))
                                 (org-agenda-ng--date-p :scheduled '<= (org-today)))
                             (not (apply #'org-agenda-ng--todo-p org-done-keywords-for-agenda))))))

With flet

(elp-profile 5 (org-agenda-ng--agenda
                :files org-agenda-files
                :pred (lambda ()
                        (and (todo)
                             (or (date :deadline '<= (org-today))
                                 (date :scheduled '<= (org-today)))
                             (not (apply #'todo org-done-keywords-for-agenda))))))

Profiling flet on a single file

This shows that the difference between them, if any, is so small as to be irrelevant. The convenience and clarity are a big win.

Without flet

(elp-profile 5 (org-agenda-ng--agenda
                :files "~/org/main.org"
                :pred (lambda ()
                        (and (org-agenda-ng--todo-p)
                             (or (org-agenda-ng--date-p :deadline '<= (org-today))
                                 (org-agenda-ng--date-p :scheduled '<= (org-today)))
                             (not (apply #'org-agenda-ng--todo-p org-done-keywords-for-agenda))))))

With flet

(elp-profile 5 (org-agenda-ng--agenda
                :files "~/org/main.org"
                :pred (lambda ()
                        (and (todo)
                             (or (date :deadline '<= (org-today))
                                 (date :scheduled '<= (org-today)))
                             (not (apply #'todo org-done-keywords-for-agenda))))))

Profiling tags matching

ng

(elp-profile 1 nil
  (org-agenda-ng "~/org/main.org"
    (tags "computer")))
FunctionTimes calledTotal timeAverage time
mapcar421712.6127164550.0029909216
org-agenda-ng–agenda19.7214106519.721410651
org-get-tags-at18457.47938603890.0040538677
org-up-heading-safe93616.46226740190.0006903394
re-search-backward250015.33998662390.0002135909
org-agenda-ng–filter-buffer14.8745988544.874598854
org-agenda-ng–tags-p12384.80676234300.0038826836
org-agenda-ng–format-element6073.63256266090.0059844524
org-outline-level174841.02989244595.890…e-05
org-add-props20740.83055492590.0004004604
org-element-headline-parser6070.20926648290.0003447553
org-back-to-heading118130.12521129601.059…e-05
outline-back-to-heading118130.11006937809.317…e-06
org-end-of-subtree6070.07219863400.0001189433
outline-on-heading-p118130.06752610305.716…e-06
outline-next-heading12390.06279809995.068…e-05
re-search-forward32730.06124466201.871…e-05
org-agenda-finalize-entries10.0418462740.041846274
buffer-substring-no-properties63290.03087169794.877…e-06
line-end-position9030.02804849503.106…e-05

ng without inheritance

(elp-profile 1 nil
  (org-agenda-ng "~/org/main.org"
    (tags "computer")))
FunctionTimes calledTotal timeAverage time
mapcar421712.5802468390.0029832219
org-agenda-ng–agenda18.7777760598.777776059
org-get-tags-at18458.28535032990.0044907047
org-up-heading-safe93617.27109818890.0007767437
re-search-backward250015.33600820600.0002134317
org-agenda-ng–filter-buffer14.8656026894.865602689
org-agenda-ng–tags-p12384.79837543100.0038759090
org-agenda-ng–format-element6073.62738251000.0059759184
org-outline-level174841.02844179195.882…e-05
org-back-to-heading118130.93905344797.949…e-05
org-split-string49400.8338250870.0001687905
string-match91020.82316291009.043…e-05
org-element-headline-parser6070.20343058190.0003351409
outline-back-to-heading118130.10961201899.278…e-06
org-end-of-subtree6070.07108025590.0001171009
outline-on-heading-p118130.06700293595.671…e-06
outline-next-heading12390.06223235195.022…e-05
re-search-forward32730.06031025191.842…e-05
org-agenda-finalize-entries10.0372864960.037286496
buffer-substring-no-properties63290.02858186894.516…e-06

original

  (elp-profile 1 nil
    (with-current-buffer "main.org"
(org-tags-view nil "computer")))
FunctionTimes calledTotal timeAverage time
org-tags-view12.6205781292.620578129
org-scan-tags11.4488838171.448883817
org-agenda-format-item6070.92738930600.0015278242
org-add-props20420.88772672090.0004347339
org-agenda-finalize10.1445067820.144506782
re-search-forward21540.13670466506.346…e-05
string-match87420.10025172591.146…e-05
org-get-priority6070.09619962200.0001584837
org-agenda-align-tags10.0951664950.095166495
org-agenda-prepare10.0817244720.081724472
org-outline-level12460.07710331706.188…e-05
org-agenda-finalize-entries10.0717074040.071707404
org-agenda-prepare-buffers10.0579039210.057903921
org-get-heading6070.05177843698.530…e-05
mapcar37380.04186411101.119…e-05
org-agenda-highlight-todo6070.02731230704.499…e-05
mapconcat6090.0247433054.062…e-05
sort20.021170690.010585345
org-activate-plain-links1320.02035589800.0001542113
org-activate-bracket-links780.01985896800.0002546021

More profiling

[2018-05-10 Thu 15:02] I think these are decent improvements.

(elp-profile 1 nil (org-agenda-ng "~/org/main.org"
                      (or (habit)
                          (and (or (date '= (org-today))
                                   (deadline '<=)
                                   (scheduled '<= (org-today)))
                               (not (apply #'todo org-done-keywords-for-agenda)))
                          (and (todo "DONE" "CANCELLED")
                               (closed '= (org-today))))))
FunctionTimes calledTotal timeAverage time
mapcar1641.50045852900.0091491373
org-agenda-ng–agenda11.3482312471.348231247
org-agenda-ng–filter-buffer11.13911898791.1391189879
org-agenda-ng–date-plain-p12670.61985710400.0004892321
org-entry-get39830.29793373707.480…e-05
org-is-habit-p13650.20491011090.0001501172
org–property-local-values13650.19406141500.0001421695
org-agenda-ng–habit-p12720.19110091790.0001502365
org-agenda-ng–format-element520.1779654110.0034224117
org-get-property-block13650.17600045190.0001289380
org-get-tags-at520.13628249690.0026208172
org-agenda-ng–date-p38800.13511766293.482…e-05
org-up-heading-safe2260.12767476090.0005649325
re-search-backward20280.11442110705.642…e-05
org-entry-properties26180.08486609993.241…e-05
org-agenda-ng–todo-p13190.0819526536.213…e-05
org-get-todo-state13190.07968368106.041…e-05
re-search-forward37540.07398037391.970…e-05
org-inlinetask-in-task-p13650.06578293304.819…e-05
org-agenda-ng–scheduled-p12470.06194978504.967…e-05