\input texinfo    @c -*- texinfo -*-
@c %**start of header
@setfilename org-mem.info
@settitle Org-mem
@documentencoding UTF-8
@documentlanguage en
@c %**end of header

@dircategory Emacs
@direntry
* Org-Mem: (org-mem).   Quick database of Org file contents.
@end direntry

@finalout
@titlepage
@title Org-mem
@author Martin Edström
@end titlepage

@ifnottex
@node Top
@top Org-mem

A cache of metadata about the contents of all your Org files -- headings@comma{} links@comma{} timestamps and so on.

Builds quickly@comma{} so that there is no need to persist data across sessions.

My @samp{M-x org-mem-reset}:

@example
Org-mem saw 2418 files@comma{} 8388 headings@comma{} 8624 links (3418 IDs@comma{} 4874 ID-links) in 1.95s
@end example

This library came from asking myself "what could I move out of @uref{https://github.com/meedstrom/org-node, org-node}@comma{} that'd make sense in core?"  Maybe a proposal for upstream@comma{} or at least a proof-of-concept.

Many notetaking packages now reinvent the wheel@comma{} when it comes to keeping track of some or many files and what may be in them.

Example: org-roam's DB@comma{} org-node's hash tables@comma{} and other packages just re-run grep all the time@comma{} which still leads to writing elisp to cross-reference the results.

And they must do this@comma{} because Org ships no tool to query across lots of files.

Well---Org does ship the agenda@comma{} which has tools.  But you know what happens if you put 2@comma{}000 files into @samp{org-agenda-files}!  It needs to open each file in real-time to check anything in them@comma{} so everyday commands grind to a halt@comma{} or even crash: many OSes have a cap of 1@comma{}024 simultaneous file handles.

@end ifnottex

@menu
* Quick start::
* Two APIs::
* Data only::
* No Org at init::
* A SQLite database@comma{} for free::
* Elisp API::
* SQL API::
* Tips::
* Current limitations::
* Future work::

@detailmenu
--- The Detailed Node Listing ---

A SQLite database@comma{} for free

* Without org-roam installed::

Elisp API

* Example Let org-agenda cast its net wide::
* Example Warn about dangling clocks at init::
* Force an update::
* Entries and links::
* Full list of functions::

SQL API

* With org-roam installed::
* To developers::
* Known issue::
* View what info is in the DB::

Tips

* Encrypted files (@samp{.org.gpg} @samp{.org.age}): Encrypted files (@samp{orggpg} @samp{orgage}). 

Current limitations

* Limitation TRAMP::
* Limitation Encrypted entries::
* Limitation SETUPFILE::

@end detailmenu
@end menu

@node Quick start
@chapter Quick start

Example setup:

@lisp
(setq org-mem-watch-dirs '("~/org" "/mnt/stuff/notes"))
(org-mem-updater-mode)
@end lisp

That's it -- give it a couple of seconds@comma{} and now evalling @samp{(org-mem-all-entries)}@comma{} @samp{(org-mem-all-links)} and variants should return a lot of results.  See examples of how to use them at section @uref{https://github.com/meedstrom/org-mem#elisp-api, Elisp API}!

See an example in detail by typing @samp{M-x org-mem-list-example}.

You do not even need to configure @samp{org-mem-watch-dirs}@comma{} if Org is loaded.  Org-mem will guess where to look for Org files anyway!  That is controlled by the user option @samp{org-mem-do-look-everywhere} (default t).

@node Two APIs
@chapter Two APIs

You get two different APIs to pick from@comma{} to access the same data.

@itemize
@item
Emacs Lisp
@item
SQL
@end itemize

Why two?  It's free.  When the data has been gathered anyway@comma{} there is no reason to @emph{only} insert it into a SQLite db@comma{} nor @emph{only} put it in a hash table.

Famously@comma{} org-roam uses a SQLite DB@.  My package org-node used simple hash tables.  Now you get both@comma{} without having to install either.

@node Data only
@chapter Data only

A design choice: Org-mem @strong{only} delivers data.  It could easily ship conveniences like@comma{} let's call it a function "org-mem-goto":

@lisp
(defun org-mem-goto (entry)
  (cl-assert (org-mem-entry-p entry))
  (find-file (org-mem-entry-file entry))
  (goto-char (org-mem-entry-pos entry))
@end lisp

but in my experience@comma{} that will spiral into dozens of lines over time@comma{} to handle a variety of edge cases.  Since you may prefer to handle edge cases different than I do@comma{} or have different needs@comma{} it ceases to be universally applicable.

So@comma{} it is up to you to write your own "goto" function@comma{} and all else to do with user interaction.

@node No Org at init
@chapter No Org at init

A design choice: Org-mem does not use Org code to analyze your files@comma{} but a @uref{https://github.com/meedstrom/org-mem/blob/main/org-mem-parser.el, custom@comma{} more dumb parser}.  That's for three reasons:

@enumerate
@item
@strong{Fast init}.  Since I want Emacs init to be fast@comma{} it's not allowed to load Org.  Yet@comma{} I want to be able to use a command like @samp{org-node-find} to browse my Org stuff immediately after init.

Or be warned about a deadline even if I don't.

That means the data must exist before Org has loaded.

@item
@strong{Robustness.}  Many users heavily customize Org@comma{} so no surprise that it sometimes breaks.  In my experience@comma{} it's very nice then to have an alternative way to browse@comma{} that does not depend on a functional Org setup.

@item
@strong{Fast rebuild.}  As they say@comma{} there are two hard things in computer science: cache invalidation and naming things.

Org-mem must update its cache as the user saves@comma{} renames and deletes files.  Not difficult@comma{} until you realize that files and directory listings may change due to a Git operation@comma{} OS file operations@comma{} a @samp{mv} or @samp{cp -a} command on the terminal@comma{} edits by another Emacs instance@comma{} or remote edits by Logseq.

A robust approach to cache invalidation is to avoid trying: ensure that a full rebuild is fast enough that you can just do @emph{that} instead.

In fact@comma{} @samp{org-mem-updater-mode} does a bit of both@comma{} because it is still important that saving a file does not lag;  it does its best to update only the necessary tables on save@comma{} and an idle timer triggers a full reset every now and then.
@end enumerate

@node A SQLite database@comma{} for free
@chapter A SQLite database@comma{} for free

Included is a drop-in for @uref{https://github.com/org-roam/org-roam, org-roam's} @samp{(org-roam-db)}@comma{} called @samp{(org-mem-roamy-db)}.

In the future we may also create something that fits @uref{https://github.com/ndwarshuis/org-sql/blob/80bea9996de7fa8bc7ff891a91cfaff91111dcd8/org-sql.el#L141, org-sql}'s DB schemata@comma{} or something custom@comma{} but we'll see!

@menu
* Without org-roam installed::
@end menu

@node Without org-roam installed
@section Without org-roam installed

Activating the mode creates an in-memory database by default.

@lisp
(org-mem-roamy-db-mode)
@end lisp

Test that it works:

@lisp
(emacsql (org-mem-roamy-db) [:select * :from files :limit 10])
@end lisp

For more tips@comma{} see section @uref{https://github.com/meedstrom/org-mem#sql-api, SQL API}.

@node Elisp API
@chapter Elisp API

@menu
* Example Let org-agenda cast its net wide::
* Example Warn about dangling clocks at init::
* Force an update::
* Entries and links::
* Full list of functions::
@end menu

@node Example Let org-agenda cast its net wide
@section Example: Let org-agenda cast its net wide

You can't put 2@comma{}000 files in @samp{org-agenda-files}@comma{} but most contain nothing of interest for the agenda anyway@comma{} right?

Turns out I have only about 30 files worth checking@comma{} the challenge was always knowing @emph{which} files ahead-of-time.  Now it's easy:

@lisp
(defun my-set-agenda-files (&rest _)
  (setq org-agenda-files
        (cl-loop
         for file in (org-mem-all-files)
         ;; Exclude *.org_archive or *archive/2025.org or similar
         unless (string-search "archive" file)
         when (seq-find (lambda (entry)
                          (or (org-mem-entry-active-timestamps entry)
                              (and (not (org-mem-entry-closed entry))
                                   (or (org-mem-entry-todo-state entry)
                                       (org-mem-entry-scheduled entry)
                                       (org-mem-entry-deadline entry)))))
                        (org-mem-entries-in file))
         collect file)))
(add-hook 'org-mem-post-full-scan-functions #'my-set-agenda-files)
(org-mem-updater-mode)
@end lisp

@node Example Warn about dangling clocks at init
@section Example: Warn about dangling clocks at init

While Org can warn about dangling clocks through the @samp{org-clock-persist} setting@comma{} that requires loading Org at some point during your session.  Which means that if it is a matter of concern for you to forget you had a clock going@comma{} that you effectively have to put @samp{(require 'org)} in your initfiles@comma{} @emph{just in case}.

(Actually you're told to call @samp{(org-clock-persistence-insinuate)} which does load Org for you@comma{} but you might forget that if you put it in a @samp{(use-package org :config ...)} form and lazy-load it.)

Now the following is an alternative:

@lisp
(defun my-warn-dangling-clock (&rest _)
  (let ((entries (seq-filter #'org-mem-entry-dangling-clocks
                             (org-mem-all-entries))))
    (when entries
      (warn "Didn't clock out in files: %S"
            (seq-uniq (mapcar #'org-mem-entry-file entries))))))
(add-hook 'org-mem-initial-scan-hook #'my-warn-dangling-clock)
(org-mem-updater-mode)
@end lisp

@node Force an update
@section Force an update

If you want to force-update the cache in synchronous code@comma{} you should now (since version 0.28.0) be able to do it like this.

@lisp
(org-mem-updater-update t)
@end lisp

To do a full reset:

@lisp
(org-mem-reset t)
(org-mem-await "Manually resetting..." 60)
@end lisp

@node Entries and links
@section Entries and links

We use two types of objects to help represent file contents: @samp{org-mem-entry} objects and @samp{org-mem-link} objects.  They involve some simplifications:

@itemize
@item
An @samp{org-mem-link} object corresponds either to a proper Org link@comma{} or to a citation fragment like @samp{@@key1} in @samp{[cite:@@key1;@@key2;@@key3]}.

@itemize
@item
Check which it is with @samp{org-mem-link-citation-p}.
@end itemize

@item
The content before the first heading also counts as an "entry"@comma{} with heading level zero.

@itemize
@item
Some predictable differences from normal entries: the zeroth-level entry obviously cannot have a TODO state@comma{} so @samp{org-mem-entry-todo-state} always returns nil@comma{} and so on.

@item
Check with @samp{org-mem-entry-subtree-p}.

@itemize
@item
Or if you're looking at the list of entries output by @samp{(org-mem-entries-in-file FILE)}@comma{} the first element (the @samp{car}) is always a zeroth-level entry@comma{} no matter if it's empty or not.  The rest (the @samp{cdr}) are subtrees.
@end itemize

@item
Some functions behave specially if the zeroth-level entry is empty@comma{} such that the first proper Org heading is on line 1: @samp{(org-mem-entry-at-lnum-in-file 1 FILE)} then returns the entry for that heading instead of the zeroth-level entry. Ditto for @samp{(org-mem-entry-at-pos-in-file 1 FILE)}.

That is hopefully intuitive.  Opinions on API design are very welcome!
@end itemize
@end itemize

@node Full list of functions
@section Full list of functions

Updated @emph{[2026-02-26 Thu]}.

Getters taking no arguments

@itemize
@item
@samp{org-mem-all-entries}
@item
@samp{org-mem-all-entries-with-active-timestamps}
@item
@samp{org-mem-all-entries-with-dangling-clock}
@item
@samp{org-mem-all-files}
@item
@samp{org-mem-all-files-expanded}
@item
@samp{org-mem-all-files-with-active-timestamps}
@item
@samp{org-mem-all-file-truenames}
@item
@samp{org-mem-all-id-links}
@item
@samp{org-mem-all-id-nodes}
@item
@samp{org-mem-all-ids}
@item
@samp{org-mem-all-links}
@item
@samp{org-mem-all-roam-reflinks}
@item
@samp{org-mem-all-roam-refs}
@item
@samp{org-mem-entry-at-point}
@end itemize

Getters operating on an entry object

@itemize
@item
@samp{org-mem-entry-active-timestamps}
@item
@samp{org-mem-entry-active-timestamps-int}
@item
@samp{org-mem-entry-children}
@item
@samp{org-mem-entry-clocks}
@item
@samp{org-mem-entry-clocks-int}
@item
@samp{org-mem-entry-closed}
@item
@samp{org-mem-entry-closed-int}
@item
@samp{org-mem-entry-dangling-clocks}
@item
@samp{org-mem-entry-deadline}
@item
@samp{org-mem-entry-deadline-int}
@item
@samp{org-mem-entry-file}
@item
@samp{org-mem-entry-file-truename}
@item
@samp{org-mem-entry-id}
@item
@samp{org-mem-entry-keywords}
@item
@samp{org-mem-entry-level}
@item
@samp{org-mem-entry-lnum}
@item
@samp{org-mem-entry-olpath}
@item
@samp{org-mem-entry-olpath-with-file-title}
@item
@samp{org-mem-entry-olpath-with-file-title-or-basename}
@item
@samp{org-mem-entry-olpath-with-self}
@item
@samp{org-mem-entry-olpath-with-self-with-file-title}
@item
@samp{org-mem-entry-olpath-with-self-with-file-title-or-basename}
@item
@samp{org-mem-entry-pos}
@item
@samp{org-mem-entry-priority}
@item
@samp{org-mem-entry-properties-inherited}
@item
@samp{org-mem-entry-properties-local} --- Alias @samp{org-mem-entry-properties}
@item
@samp{org-mem-entry-property}
@item
@samp{org-mem-entry-property-with-inheritance}
@item
@samp{org-mem-entry-pseudo-id}
@item
@samp{org-mem-entry-roam-aliases}
@item
@samp{org-mem-entry-roam-refs}
@item
@samp{org-mem-entry-scheduled}
@item
@samp{org-mem-entry-scheduled-int}
@item
@samp{org-mem-entry-stats-cookies}
@item
@samp{org-mem-entry-subtree-p}
@item
@samp{org-mem-entry-tags}
@item
@samp{org-mem-entry-tags-inherited}
@item
@samp{org-mem-entry-tags-local}
@item
@samp{org-mem-entry-text} --- Always nil unless @samp{org-mem-do-cache-text} is t
@item
@samp{org-mem-entry-title}
@item
@samp{org-mem-entry-title-maybe}
@item
@samp{org-mem-entry-todo-state}
@item
@samp{org-mem-id-links-to-entry}
@item
@samp{org-mem-links-in-entry}
@item
@samp{org-mem-next-entry}
@item
@samp{org-mem-previous-entry}
@item
@samp{org-mem-roam-reflinks-to-entry}
@end itemize

Getters operating on a file name

@itemize
@item
@samp{org-mem-entries-in-file} --- Alias @samp{org-mem-file-entries}
@item
@samp{org-mem-entries-in-files}
@item
@samp{org-mem-entry-at-lnum-in-file} --- Near-alias @samp{org-mem-entry-at-file-lnum}
@item
@samp{org-mem-entry-at-pos-in-file} --- Near-alias @samp{org-mem-entry-at-file-pos}
@item
@samp{org-mem-file-attributes}
@item
@samp{org-mem-file-char-count}
@item
@samp{org-mem-file-coding-system}
@item
@samp{org-mem-file-id-strict}
@item
@samp{org-mem-file-id-topmost}
@item
@samp{org-mem-file-keywords}
@item
@samp{org-mem-file-known-p}
@item
@samp{org-mem-file-line-count}
@item
@samp{org-mem-file-mtime}
@item
@samp{org-mem-file-mtime-floor}
@item
@samp{org-mem-file-ptmax}
@item
@samp{org-mem-file-size}
@item
@samp{org-mem-file-title-or-basename}
@item
@samp{org-mem-file-title-strict}
@item
@samp{org-mem-file-title-topmost}
@end itemize

Getters operating on a link object

@itemize
@item
@samp{org-mem-id-link-p}
@item
@samp{org-mem-link-citation-p}
@item
@samp{org-mem-link-description}
@item
@samp{org-mem-link-entry}
@item
@samp{org-mem-link-entry-pseudo-id}
@item
@samp{org-mem-link-file}
@item
@samp{org-mem-link-file-truename}
@item
@samp{org-mem-link-nearby-id}
@item
@samp{org-mem-link-pos}
@item
@samp{org-mem-link-supplement}
@item
@samp{org-mem-link-target}
@item
@samp{org-mem-link-type}
@item
@samp{org-mem-roam-reflink-p}
@end itemize

Getters that return entries

@itemize
@item
@samp{org-mem-all-entries}
@item
@samp{org-mem-all-entries-with-active-timestamps}
@item
@samp{org-mem-all-entries-with-dangling-clock}
@item
@samp{org-mem-all-id-nodes}
@item
@samp{org-mem-entries-in-file} --- Alias @samp{org-mem-file-entries}
@item
@samp{org-mem-entries-in-files}
@item
@samp{org-mem-entry-at-lnum-in-file} --- Near-alias @samp{org-mem-entry-at-file-lnum}
@item
@samp{org-mem-entry-at-pos-in-file} --- Near-alias @samp{org-mem-entry-at-file-pos}
@item
@samp{org-mem-entry-by-id}
@item
@samp{org-mem-entry-by-pseudo-id}
@item
@samp{org-mem-entry-by-roam-ref}
@item
@samp{org-mem-entry-children}
@item
@samp{org-mem-id-node-by-title}
@item
@samp{org-mem-id-nodes-in-files}
@item
@samp{org-mem-link-entry}
@end itemize

Getters that return links

@itemize
@item
@samp{org-mem-all-id-links}
@item
@samp{org-mem-all-links}
@item
@samp{org-mem-all-roam-reflinks}
@item
@samp{org-mem-id-links-from-id}
@item
@samp{org-mem-id-links-into-file}
@item
@samp{org-mem-id-links-to-entry}
@item
@samp{org-mem-id-links-to-id}
@item
@samp{org-mem-links-from-id}
@item
@samp{org-mem-links-in-entry}
@item
@samp{org-mem-links-in-file}
@item
@samp{org-mem-links-of-type}
@item
@samp{org-mem-links-to-roam-ref}
@item
@samp{org-mem-links-to-target}
@item
@samp{org-mem-links-with-type-and-path}
@item
@samp{org-mem-roam-reflinks-into-file}
@item
@samp{org-mem-roam-reflinks-into-files}
@item
@samp{org-mem-roam-reflinks-to-entry}
@item
@samp{org-mem-roam-reflinks-to-id}
@end itemize

Getters that return file names

@itemize
@item
@samp{org-mem-all-files}
@item
@samp{org-mem-all-files-expanded}
@item
@samp{org-mem-all-files-with-active-timestamps}
@item
@samp{org-mem-all-file-truenames}
@item
@samp{org-mem-entry-file}
@item
@samp{org-mem-entry-file-truename}
@item
@samp{org-mem-file-by-id}
@item
@samp{org-mem-file-known-p}
@item
@samp{org-mem-link-file}
@item
@samp{org-mem-link-file-truename}
@end itemize

Misc

@itemize
@item
@samp{org-mem-all-ids}
@item
@samp{org-mem-all-roam-refs}
@item
@samp{org-mem-entry-at-point}
@item
@samp{org-mem-id-by-title}
@end itemize

Some API for actually activating & using org-mem as a library:

@itemize
@item
@samp{org-mem-await}
@item
@samp{org-mem-reset}
@item
@samp{org-mem-updater-update}
@end itemize

@node SQL API
@chapter SQL API

@menu
* With org-roam installed::
* To developers::
* Known issue::
* View what info is in the DB::
@end menu

@node With org-roam installed
@section With org-roam installed

You can use this to end your dependence on @samp{org-roam-db-sync}.  Set the following to let org-mem overwrite the "org-roam.db" file.

@lisp
(setq org-roam-db-update-on-save nil)
(setq org-mem-roamy-do-overwrite-real-db t)
(org-mem-roamy-db-mode)
@end lisp

Now@comma{} you have a new@comma{} all-fake org-roam.db!  Test that org-roam's @samp{org-roam-db-query} works:

@lisp
(org-roam-db-query [:select * :from files :limit 10])
@end lisp

If it works@comma{} then various packages that depend on org-roam's DB should also work without issue.  However@comma{} you'll have to be content with them pulling in org-roam even if you never turn it on.

N/B: because @samp{(equal (org-roam-db) (org-mem-roamy-db))}@comma{} the above is equivalent to these expressions:

@lisp
(emacsql (org-roam-db) [:select * :from files :limit 10])
(emacsql (org-mem-roamy-db) [:select * :from files :limit 10])
@end lisp

@node To developers
@section To developers

If you write a package that actually only needed to use org-roam's DB and not its other utilities like @samp{org-roam-capture}@comma{} you should be able to stop requiring org-roam@comma{} and do e.g.

@lisp
(defun my-db ()
  (cond ((fboundp 'org-roam-db) (org-roam-db))
        ((fboundp 'org-mem-roamy-db) (org-mem-roamy-db))
        (t (error "Enable either org-roam-db-autosync-mode or org-mem-roamy-db-mode"))
@end lisp

and then rewrite all your @samp{(org-roam-db-query ...)} forms to:

@lisp
(emacsql (my-db) ...)
@end lisp

In theory@comma{} anyway ;-)

Please report any issues!

@node Known issue
@section Known issue

Error "attempt to write a readonly database" can happen when when you run multiple Emacs instances.  Get unstuck with @samp{M-: (org-roam-db--close-all)}.

@node View what info is in the DB
@section View what info is in the DB

Use @samp{M-x org-mem-list-db-contents}.

Or the new @samp{M-x org-roam-db-explore}@comma{} with exactly the same UI@.

@node Tips
@chapter Tips

@menu
* Encrypted files (@samp{.org.gpg} @samp{.org.age}): Encrypted files (@samp{orggpg} @samp{orgage}). 
@end menu

@node Encrypted files (@samp{orggpg} @samp{orgage})
@section Encrypted files (@samp{.org.gpg} @samp{.org.age})

Suppose you use @uref{https://github.com/anticomputer/age.el, age.el} to keep files encrypted.

This needs extra config to tell org-mem's subprocesses how to open them. The following should do the trick:

@lisp
(setq org-mem-inject-vars '(age-default-identity age-default-recipient age-program))
(setq org-mem-eval-forms '((require 'age) (age-file-enable)))
(add-to-list 'org-mem-suffixes ".org.age")
@end lisp

@node Current limitations
@chapter Current limitations

@menu
* Limitation TRAMP::
* Limitation Encrypted entries::
* Limitation SETUPFILE::
@end menu

@node Limitation TRAMP
@section Limitation: TRAMP

Files over TRAMP are excluded from org-mem's database@comma{} so as far as org-mem is concerned@comma{} it is as if they do not exist.

This limitation comes from the fact that org-mem parses your files in many parallel subprocesses that do not inherit your TRAMP connections.  It is fixable in theory.

@node Limitation Encrypted entries
@section Limitation: Encrypted entries

The body text of specific entries may be encrypted by @samp{org-crypt}.   If so@comma{} org-mem cannot find links nor timestamps inside.

@node Limitation SETUPFILE
@section Limitation: SETUPFILE

No support yet for buffer settings from @samp{#+SETUPFILE:}.

@node Future work
@chapter Future work

Use Org's own parser.

@itemize
@item
@uref{https://lists.gnu.org/archive/html/emacs-orgmode/2025-05/msg00288.html}
@item
@uref{https://github.com/meedstrom/org-mem/issues/29}
@end itemize

@bye
