\input texinfo    @c -*- texinfo -*-
@c %**start of header
@setfilename el-job.info
@settitle el-job
@documentencoding UTF-8
@documentlanguage en
@c %**end of header

@dircategory Emacs
@direntry
* El-Job: (el-job).     Async multicore mapcar.
@end direntry

@finalout
@titlepage
@title el-job
@author Martin Edström
@end titlepage

@ifnottex
@node Top
@top el-job

Imagine you have a function you'd like to run on a long list of inputs.  You could run @samp{(mapcar #'FN INPUTS)}@comma{} but that hangs Emacs until done.

This library lets you run the same function in many subprocesses (one per CPU core)@comma{} each with their own split of the @samp{INPUTS} list@comma{} then merge their outputs and pass it back to the current Emacs.

In the meantime@comma{} current Emacs does not hang at all.

Best of all@comma{} it completes @emph{faster} than @samp{(mapcar #'FN INPUTS)}@comma{} owing to the use of all CPU cores!

That's it in a nutshell.  You can look at real-world usage by searching for "el-job" in these packages:
@itemize
@item
@uref{https://raw.githubusercontent.com/meedstrom/org-mem/refs/heads/main/org-mem.el, org-mem.el}
@item
@uref{https://raw.githubusercontent.com/meedstrom/org-roam-async/refs/heads/main/org-roam-async.el, org-roam-async.el}
@end itemize

@end ifnottex

@menu
* Since 2.5.0: Since 250. 
* README for 2.4.8: README for 248. 

@detailmenu
--- The Detailed Node Listing ---

Since 2.5.0

* Future work::

README for 2.4.8

* Design rationale::
* News 2.4: News 24. 
* News 2.3: News 23. 
* News 2.1: News 21. 
* News 2.0: News 20. 
* News 1.1: News 11. 
* News 1.0: News 10. 
* Limitations::

Design rationale

* Processes stay alive::
* Emacs 30 @samp{fast-read-process-output}::

@end detailmenu
@end menu

@node Since 250
@chapter Since 2.5.0

Released @emph{<2025-Oct-06>}@comma{} v2.5.0 comes with a variant library "el-job-ng".

I find it simpler and easier to reason about.  400 lines of code instead of 800.

Some differences:

@itemize
@item
Does @emph{not} ever keep a process alive
@item
Does @emph{not} merge the subprocesses' outputs in a bespoke way@comma{} just uses @samp{append}.
@item
Does @emph{not} look up @samp{load-history} to try hard to find an .eln variant of your libraries@comma{} instead it lets @samp{require} look up @samp{load-path} normally.
@item
Removed argument @samp{:if-busy}@comma{} you can manage this yourself with @samp{el-job-ng-busy-p} and @samp{el-job-ng-kill}.
@item
Argument @samp{:funcall-per-input} now takes a function of two (2) arguments@comma{} not one.
@item
Argument @samp{:callback} now takes a function of one (1) argument@comma{} not two.
@item
Added optional argument @samp{:eval}.
@end itemize

The design is always changing in response to my needs@comma{} so feel free to file an issue or email!  It's instructive to hear about other people's needs.

@menu
* Future work::
@end menu

@node Future work
@section Future work

I may write yet another variant.

Something that came with experience is that it's best to make a new variant library for a narrow use-case@comma{} rather than complicate one library with different code flows.  When it comes to this type of library@comma{} you really want to keep it easy to reason about!

Ideas as of @emph{<2025-Oct-06>}:

@table @asis
@item File IPC
Sending input and output by writing and reading files@comma{} instead of through the pipe connection.

Theory: performance can sometimes be a lot better@comma{} with large inputs and outputs.  I would guess it depends a lot on the machine.

Drawback: if the data sent is sensitive@comma{} those files probably should be encrypted@comma{} and that could negate any performance benefit.

@item Worker daemons
Keeping subprocesses alive forever@comma{} so they are available at beck and call -- think worker daemons.

That's basically implemented in "el-job-old"@comma{} and partly why it got so hairy@comma{} but it could be remade to put this usage front-and-center@comma{} with a "happy path" UX@.

At this time@comma{} I do not have a demanding use-case to experiment with@comma{} to discover what that happy path should be@comma{} how the affordances should be.

If you need this now@comma{} I recommend @uref{https://github.com/jwiegley/emacs-async, async.el}.
@end table

@node README for 248
@chapter README for 2.4.8

@menu
* Design rationale::
* News 2.4: News 24. 
* News 2.3: News 23. 
* News 2.1: News 21. 
* News 2.0: News 20. 
* News 1.1: News 11. 
* News 1.0: News 10. 
* Limitations::
@end menu

@node Design rationale
@section Design rationale

I wanted to shorten the round-trip as much as possible@comma{} @strong{between the start of an async task and having the results}.

For example@comma{} say you have some lisp that collects completion candidates@comma{} and you want to run it asynchronously because the lisp you wrote isn't always fast enough to avoid the user's notice@comma{} but you'd still like it to return as soon as possible.

@menu
* Processes stay alive::
* Emacs 30 @samp{fast-read-process-output}::
@end menu

@node Processes stay alive
@subsection Processes stay alive

In the above example@comma{} a user might only delay a fraction of a second between opening the minibuffer and beginning to type@comma{} so there's scant room for overhead like spinning up subprocesses that load a bunch of libraries before getting to work.

Thus@comma{} el-job keeps idle subprocesses for up to 30 seconds after a job finishes@comma{} awaiting more input.

An aesthetic drawback is cluttering your task manager with many processes named "emacs".

Users who tend to run system commands such as @samp{pkill emacs} may find that the command occasionally "does not work"@comma{} because it actually killed an el-job subprocess@comma{} instead of the Emacs they see on screen.

@node Emacs 30 @samp{fast-read-process-output}
@subsection Emacs 30 @samp{fast-read-process-output}

Some other libraries@comma{} like the popular @uref{https://github.com/jwiegley/emacs-async/, async.el}@comma{} are designed around a custom process filter.

Since Emacs 30@comma{} it's a good idea to instead use the @emph{built-in} process filter when performance is critical@comma{} and el-job does so.  Quoting @uref{https://github.com/emacs-mirror/emacs/blob/master/etc/NEWS.30, NEWS.30}:

@example
** The default process filter was rewritten in native code.
The round-trip through the Lisp function
'internal-default-process-filter' is skipped when the process filter is
the default one.  It is reimplemented in native code@comma{} reducing GC churn.
To undo this change@comma{} set 'fast-read-process-output' to nil.
@end example

@node News 24
@section News 2.4

@itemize
@item
Jobs must now have @samp{:inputs}.  If @samp{:inputs} nil and there was nothing queued@comma{} @samp{el-job-launch} will no-op and return the symbol @samp{inputs-were-empty}.
@end itemize

@node News 23
@section News 2.3

@itemize
@item
Some renames to follow Elisp convention
@itemize
@item
@samp{el-job:timestamps} and friends now @samp{el-job-timestamps}.
@end itemize
@end itemize

@node News 21
@section News 2.1

@itemize
@item
DROP SUPPORT Emacs 28
@itemize
@item
It likely has not been working for a while anyway.  Maybe works on the @uref{https://github.com/meedstrom/el-job/tree/v0.3, v0.3 branch}@comma{} from 0.3.26+.
@end itemize
@end itemize

@node News 20
@section News 2.0

@itemize
@item
Jobs must now have @samp{:id} (no more anonymous jobs).
@item
Pruned many code paths.
@end itemize

@node News 11
@section News 1.1

@itemize
@item
Changed internals so that all builds of Emacs can be expected to perform similarly well.
@end itemize

@node News 10
@section News 1.0

@itemize
@item
No longer keeps processes alive forever.  All jobs are kept alive for up to 30 seconds of disuse@comma{} then reaped.
@item
Pruned many code paths.
@item
Many arguments changed@comma{} and a few were removed.  Consult the docstring of @samp{el-job-launch} again.
@end itemize

@node Limitations
@section Limitations

@enumerate
@item
The return value from the @samp{:funcall-per-input} function must always be a list with a fixed length@comma{} where the elements are also lists.

For example@comma{} org-mem passes @samp{:funcall-per-input #'org-mem-parser--parse-file} to el-job@comma{} and if you look in @uref{https://github.com/meedstrom/org-mem/blob/main/org-mem-parser.el, org-mem-parser.el} for the defun of @samp{org-mem-parser--parse-file}@comma{} it always returns a list of 5 items:

@lisp
(list (if missing-file (list missing-file)) ; List of 0 or 1 item
      (if file-mtime (list file-mtime))     ; List of 0 or 1 item
      found-entries                         ; List of many items
      org-node-parser--found-links          ; List of many items
      (if problem (list problem))))         ; List of 0 or 1 item
@end lisp

It may look clunky to return sub-lists of only one item@comma{} but you could consider it a minor expense in exchange for simpler library code.

@item
Some data types cannot be exchanged with the children: those whose printed form look like @samp{#<...>}.  For example@comma{} @samp{#<buffer notes.org>}@comma{} @samp{#<obarray n=94311>}@comma{} @samp{#<marker at 3102 in README.org>}.

IIUC@comma{} this sort of data only has meaning within the current process -- so even if you could send it@comma{} it would not be usable by the recipient anyway.

@item
For now@comma{} this library tends to be applicable only to a narrow set of use-cases@comma{} since you can only pass one @samp{:inputs} list which would tend to contain a single kind of thing@comma{} e.g. it could be a list of files to visit@comma{} to be split between child processes.  In many potential use-cases@comma{} you'd actually want multiple input lists and split them differently@comma{} and that's not supported yet.
@end enumerate

@bye
