* org-agenda-files-track

Fine-track =org-agenda-files= to speed-up =org-agenda=.

When an agenda buffer is built, Emacs visits each file listed in =org-agenda-files=. In case your tasks or events are recorded in an ever-extending journal and/or roam directories, =org-agenda= can become sluggish.

This package aims to dynamically update the =org-agenda-files= variable by appending/deleting a candidate org file when it is saved. This limits the number of files to visit when building the agenda. The agenda buffer thus builds faster.

Two different packages are provided :

- =org-agenda-files-track= uses the native =org-element= API. In this case, candidate selection is governed by the =org-agenda-files-track-predicate=.

- =org-agenda-files-track-ql= uses the =org-ql= backend. In this case, candidate selection logic is extracted from =org-agenda-custom-commands= and =org-ql-views=. Once the cache of =org-ql= is built, using =org-ql-view= commands instead of =org-agenda= should additionally speed-up approximately 5 times. See [[https://github.com/alphapapa/org-ql/blob/master/README.org#agenda-like-views][org-ql's readme]] for details.

** Configuration

*** =org-agenda-files-track=

*You should redefine the* =org-agenda-files-track-predicate= *to align with your desired* =org-agenda-files= *.* Typically, this involves matching content compatible with =org-agenda= and =org-agenda-custom-commands=. Test the predicate using =pp-eval-expression= on =(org-agenda-files-track-file-p)= with an opened candidate org file.

*** =org-agenda-files-track-ql=

If you want to continue using =org-agenda= alongside =org-ql=, rather than =org-ql-view=, you should =(setq org-agenda-include-diary nil)= and use pure =org-ql-block= commands in your =org-agenda-custom-commands=. With =org-ql= integration, there's no need for explicit predicate definition.

*** Initially defining or redefining =org-agenda-files=

Redefining =org-agenda-files= is probably necessary when =org-agenda-files-predicate= or =org-agenda-custom-commands= is changed. One simple method is to record all possible org files and then cleanup which files are agenda-files. This is the part that will take time. For instance:

#+begin_src elisp
(defun my/org-agenda-files-track-init ()
  "(Re)initialize dynamic agenda files.

This can take a long time, so it is recommended to run this only
on installation and when first tasks are added to many files via
methods the save hook cannot detect, like file synchronization."
  (interactive)
  ;; ;; uncomment if storing org-agenda-files in file
  ;; (make-empty-file org-agenda-files 'force)
  (org-store-new-agenda-file-list
   (directory-files-recursively
    org-directory (rx ".org" eos) nil
    ;; ignore hidden directories like .git and .attach
    (lambda (subdir)
      (not (eq ?. (string-to-char (file-name-nondirectory subdir)))))))
  ;; use ql here if desired
  (org-agenda-files-track-cleanup-files 'full)
  (message "Initialized agenda files"))
#+end_src

Both values of =org-agenda-files= (a file path or a list of files) are supported, since we use a native =org-agenda= saving method. You will likely want to set =org-agenda-files= to a single filename to persist =org-agenda-files= across Emacs sessions:

#+begin_src elisp
(setq org-agenda-files
      (expand-file-name "org-agenda-files.txt" (xdg-cache-home)))
#+end_src

** Changelog

It's only getting started!

MAYBE TODO: redefine =org-agenda-files-track-predicate= to match all possible =org-agenda= commands, current default config is quite minimalist. Maybe that could be done automatically by parsing =org-agenda-custom-commands= and =org-agenda=, but then it amounts to re-create a subset of =org-ql=.

MAYBE TODO: investigate integration with =org-super-agenda=.

MAYBE TODO: investigate integration with =consult-org-agenda=.

** Relation to other files / packages

The original source of inspiration for this package is the post [[https://d12frosted.io/posts/2021-01-16-task-management-with-roam-vol5.html][Task management with org-roam Vol. 5: Dynamic and fast agenda]] by Boris Buliga. But since I'd like to use my =org-roam= files for knowledge management and keep task management separate, I decided to make a package out of it to build it.

In the process of writing the package, I discovered [[https://www.armindarvish.com/en/post/emacs_workflow_dynamically_adding_files_to_org-agenda-files/][Armin Darvish's implementation]] which basically does the same thing, on which I added a few improvements (namely: native save, org-ql integration, more general utility functions). 

This package functions both on builtin functionality and with =org-ql= clarity and speed improvements.

At some point, this package family prefix was =org-dynamic-agenda= but settled for =org-agenda-files-track= for clarity.

Finally, the same package could be done with =org-roam= and =org-roam-ql= (with some more work, could be faster (you might not need more speed than here though) since you can read from the database without having to parse files), but it mingles knowledge and task management.