Shadowgraph
TrendingAbout
Sign in

melisgl/mgl-pax

PAX, the Common Lisp documentation system

Common Lisp

94

988 commits

updated Aug 3, 2026

fixnum.com

See the code

README

PAX Manual

Table of Contents

  • [1 Introduction][685e]

  • [2 Links and Systems][ba90]

  • [3 Emacs Setup][8541]

    • [3.1 Functionality Provided][d4a9]

    • [3.2 Installing from Quicklisp][f3f4]

    • [3.3 Loading PAX][d3fc]

    • [3.4 Setting up Keys][cfab]

  • [4 Background][f74b]

  • [5 Basics][94c7]

  • 6 PAX Locatives

  • 7 Navigating Sources in Emacs

    • 7.1 M-. Defaulting

    • [7.2 M-. Prompting][ed46]

      • [7.2.1 M-. Minibuffer Syntax][8106]

      • [7.2.2 M-. Completion][e444]

  • 8 Generating Documentation

    • [8.1 The DOCUMENT Function][dc0a]

      • 8.1.1 DOCUMENTABLE

      • [8.1.2 Return Values][7dc7]

      • [8.1.3 PAGES][9c7d]

      • [8.1.4 Package and Readtable][ab7e]

    • [8.2 Browsing Live Documentation][a595]

      • [8.2.1 Browsing with w3m][83d5]

      • [8.2.2 Browsing with Other Browsers][c434]

      • [8.2.3 Apropos][b7fc]

      • [8.2.4 PAX Live Home Page][9d50]

    • [8.3 Markdown Support][c2d3]

    • [8.4 Codification][f1ab]

    • 8.5 Linking

      • [8.5.1 Reflink][cbc4]

      • [8.5.2 Autolink][ec7a]

      • [8.5.3 Linking to the HyperSpec][7cc3]

      • 8.5.4 Linking to Sections

      • [8.5.5 Filtering Links][b2e4]

      • [8.5.6 Link Format][c0d2]

    • [8.6 Local Definition][9db9]

    • 8.7 Overview of Escaping

    • [8.8 Indexing][adf8]

    • [8.9 Output Formats][8d9b]

      • [8.9.1 Plain Output][c879]

      • [8.9.2 Markdown Output][dd29]

      • 8.9.3 PDF Output

      • [8.9.4 Dummy Output][f7e6]

    • [8.10 Documentation Generation Implementation Notes][d1ca]

    • 8.11 Utilities for Generating Documentation

  • [9 Transcripts][6300]

    • [9.1 Transcribing in Documentation][9bd5]

    • [9.2 Transcribing with Emacs][f5bd]

    • [9.3 Controlling the Dynamic Environment][6b59]

    • [9.4 Transcript API][9dbc]

    • 9.5 Transcript Consistency Checking

      • [9.5.1 Finer-Grained Consistency Checks][6e18]

      • [9.5.2 Utilities for Consistency Checking][8423]

  • 10 Parsing

    • [10.1 Parsing Names][e65d]

    • [10.2 Parsing Locatives][ab38]

  • [11 Writing Extensions][c4ce]

[in package MGL-PAX with nicknames PAX]

1 Introduction

What if documentation were code?

If some narrative glued docstrings together, we'd be able to develop and explore the code alongside its documentation. The main tool that PAX provides for this is [DEFSECTION][72b4]:

(defsection @foo-random-manual (:title "Foo Random manual")
  "Foo Random is a random number generator library."
  (foo-random-state class)
  (uniform-random function)
  (@foo-random-examples section))

Like the one above, sections can have docstrings and references to definitions (e.g. (UNIFORM-RANDOM FUNCTION)). These are the glue. To support interactive development, PAX

  • makes [SLIME][6be7]'s [M-.][cb15] work with references and

  • adds a documentation browser.

See [Emacs Setup][8541].

Beyond interactive workflows, Generating Documentation from sections and all the referenced items in Markdown or HTML format is also implemented.

PAX [emphasizes the narrative][bbc6] much like Literate Programming, but it is an untangled LP system.

Docstrings

PAX automatically recognizes and [marks up code][f1ab] with backticks and links names in code to their definitions. Take, for instance, SBCL's ABORT function, whose docstring is written in the usual style, uppercasing names of symbols:

(docstring #'abort)
=> "Transfer control to a restart named ABORT, signalling
a CONTROL-ERROR if none exists."

Note how in the generated documentation, ABORT is set with a monospace font, while CONTROL-ERROR is [Autolink][ec7a]ed:

  • [function] ABORT &OPTIONAL CONDITION

    Transfer control to a restart named ABORT, signalling a CONTROL-ERROR if none exists.

The following [transcript][6300] shows the raw Markdown for the previous example.

(document #'abort :format :markdown)
.. - [function] **ABORT** *&OPTIONAL CONDITION*
..
..     Transfer control to a restart named `ABORT`, signalling
..     a [`CONTROL-ERROR`][7c2c] if none exists.
..
..   [7c2c]: http://www.lispworks.com/documentation/HyperSpec/Body/e_contro.htm "CONTROL-ERROR (MGL-PAX:CLHS CONDITION)"
..
A Complete Example

Here is an example of how it all works together:

(mgl-pax:define-package :foo-random
  (:documentation "This package provides various utilities for random.
  See FOO-RANDOM:@FOO-RANDOM-MANUAL.")
  (:use #:common-lisp #:mgl-pax))

(in-package :foo-random)

(defsection @foo-random-manual (:title "Foo Random manual")
  "FOO-RANDOM is a random number generator library inspired by CL:RANDOM.
  Functions such as UNIFORM-RANDOM use *FOO-STATE* and have a
  :RANDOM-STATE keyword arg."
  (foo-random-state class)
  (state (reader foo-random-state))
  "Hey we can also print states!"
  (print-object (method (foo-random-state t)))
  (*foo-state* variable)
  (gaussian-random function)
  (uniform-random function)
  ;; This is a subsection.
  (@foo-random-examples section))

(defclass foo-random-state ()
  ((state :reader state)))

(defmethod print-object ((object foo-random-state) stream)
  (print-unreadable-object (object stream :type t)))

(defvar *foo-state* (make-instance 'foo-random-state)
  "Much like *RANDOM-STATE* but uses the FOO algorithm.")

(defun uniform-random (limit &key (random-state *foo-state*))
  "Return a random number from the uniform distribution between 0 and
  LIMIT (exclusive)."
  nil)

(defun gaussian-random (stddev &key (random-state *foo-state*))
  "Return a random number from a zero mean normal distribution with
  STDDEV."
  nil)

(defsection @foo-random-examples (:title "Examples")
  "Let's see the transcript of a real session of someone working
  with FOO:

  ```cl-transcript
  (values (princ :hello) (list 1 2))
  .. HELLO
  => :HELLO
  => (1 2)

  (make-instance 'foo-random-state)
  ==> #<FOO-RANDOM-STATE >
  ```")

Note how (VARIABLE *FOO-STATE*) in the [DEFSECTION][72b4] form both exports *FOO-STATE* and includes its documentation in @FOO-RANDOM-MANUAL. The symbols VARIABLE and FUNCTION are just two instances of locatives, used in DEFSECTION to refer to definitions tied to symbols.

(DOCUMENT @FOO-RANDOM-MANUAL) generates fancy Markdown or HTML output with [automatic markup][f25f] and [Autolink][ec7a]s uppercase [word][d7b0]s found in docstrings, numbers sections, and creates a table of contents.

One can even generate documentation for different but related libraries at the same time with the output going to different files but with cross-page links being automatically added for symbols mentioned in docstrings. In fact, this is what PAX World does. See Generating Documentation for some convenience functions to cover the most common cases.

The [transcript][6300] in the code block tagged with cl-transcript is automatically checked for up-to-dateness when documentation is generated.

The official repository is https://github.com/melisgl/mgl-pax, and this document is available in various formats on https://fixnum.com for the latest version. There is also a PAX channel on YouTube with a couple of videos.

PAX is built on top of the DRef library. See https://github.com/melisgl/dref.

  • Installation for deployment

    The base system is [mgl-pax][6fdb]. It has very few dependencies and is sufficient as a dependency for systems using the [Basics][94c7] to add documentation. This is to keep deployed code small. To install only the bare minimum, with no intention of using Navigating Sources in Emacs, Generating Documentation, [Browsing Live Documentation][a595] or using [Transcripts][6300], under Quicklisp for example, PAX could be installed as:

      (ql:quickload "mgl-pax")

  • Installation for development

    The heavier dependencies are on the other systems, which correspond to the main functionalities provided, intended to be used primarily during development. To install the dependencies for all features under Quicklisp, do

      (ql:quickload "mgl-pax/full")

    Having thus installed the dependencies, it is enough to load the base system, which will autoload the other systems as necessary.

  • [system] "mgl-pax"

  • [system] "mgl-pax/navigate"

    • Description: Support for Navigating Sources in Emacs via Slime's [M-.][cb15] in MGL-PAX.

    • Depends on: alexandria, autoload-doc, dref/full, [mgl-pax][6fdb], swank(?)

    • Defsystem depends on: autoload

  • [system] "mgl-pax/document"

    • Description: Support for Generating Documentation in MGL-PAX.

    • Depends on: 3bmd, 3bmd-ext-code-blocks, 3bmd-ext-math, alexandria, autoload-doc, colorize, md5, [mgl-pax/navigate][f155], [mgl-pax/transcribe][5825], trivial-utf-8

    • Defsystem depends on: autoload

  • [system] "mgl-pax/web"

    • Description: Web server for [Browsing Live Documentation][a595] in MGL-PAX.

    • Depends on: hunchentoot, mgl-pax/document

    • Defsystem depends on: autoload

  • [system] "mgl-pax/transcribe"

    • Description: Support for [Transcripts][6300] in MGL-PAX.

    • Depends on: alexandria, [mgl-pax/navigate][f155]

    • Defsystem depends on: autoload

  • [system] "mgl-pax/full"

    • Description: The [mgl-pax][6fdb] system with all features preloaded.

    • Depends on: mgl-pax/document, [mgl-pax/navigate][f155], [mgl-pax/transcribe][5825], [mgl-pax/web][a8c5]

    • Defsystem depends on: autoload

3 Emacs Setup

Here is a quick recipe for setting up PAX for use via [SLIME][6be7] to take advantage of the [conveniences on offer][d4a9]. Conversely, there is no need to do any of this just to use [DEFSECTION][72b4], write docstrings and for Generating Documentation.

If PAX was installed from Quicklisp, then evaluate this in CL to install the Elisp code in a stable location:

(mgl-pax:install-pax-elisp "~/quicklisp/")

Assuming the Elisp file is in the ~/quicklisp/ directory, add something like this to your .emacs:

(add-to-list 'load-path "~/quicklisp/")
(require 'mgl-pax)
(global-set-key (kbd "C-.") 'mgl-pax-document)
(global-set-key (kbd "s-x t") 'mgl-pax-transcribe-last-expression)
(global-set-key (kbd "s-x r") 'mgl-pax-retranscribe-region)

Alternatively, with use-package:

(use-package mgl-pax
  :load-path "~/quicklisp"
  :after slime
  :demand t
  :bind (("C-." . mgl-pax-document)
         ("s-x t" . mgl-pax-transcribe-last-expression)
         ("s-x r" . mgl-pax-retranscribe-region))
  :config
  (mgl-pax-hijack-slime-doc-keys))

When [Browsing with Other Browsers][c434], for clicking on the locative next to a definition to visit the corresponding source location in Emacs, permission needs to be given:

(setq slime-enable-evaluate-in-emacs t)

3.1 Functionality Provided

  • For Navigating Sources in Emacs, loading mgl-pax extends slime-edit-definitions ([M-.][cb15]) by adding mgl-pax-edit-definitions to slime-edit-definition-hooks. There are no related variables to customize.

  • For [Browsing Live Documentation][a595], mgl-pax-browser-function and mgl-pax-web-server-port can be customized in Elisp. To browse within Emacs, choose w3m-browse-url (see [w3m][7439]), and make sure both the w3m binary and the w3m Emacs package are installed. On Debian, simply install the w3m-el package. With other browser functions, a HUNCHENTOOT web server is started.

  • See [Transcribing with Emacs][f5bd] for how to use the transcription features. There are no related variables to customize.

Independently from the Common Lisp side, the Elisp functions mgl-pax-hideshow-documentation and mgl-pax-hideshow-comments help focus on the code only by folding or unfolding [MGL-PAX:DEFSECTION][72b4], [MGL-PAX:DEFINE-GLOSSARY-TERM][8ece] forms and long strings, or comments.

3.2 Installing from Quicklisp

If you installed PAX with Quicklisp, the location of mgl-pax.el may change with updates, and you may want to copy the current version of mgl-pax.el to a stable location by evaluating this in CL:

(mgl-pax:install-pax-elisp "~/quicklisp/")

If working from, say, a git checkout, there is no need for this step.

  • [function] INSTALL-PAX-ELISP TARGET-DIR

    Install mgl-pax.el distributed with this package in TARGET-DIR.

3.3 Loading PAX

Assuming the Elisp file is in the ~/quicklisp/ directory, add something like this to your .emacs:

(add-to-list 'load-path "~/quicklisp/")
(require 'mgl-pax)

If the Elisp variable mgl-pax-autoload is true (the default), then PAX will be loaded in the connected Lisp on-demand via [SLIME][6be7].

If loading fails, mgl-pax will be unloaded from Emacs and any [overridden Slime key bindings][cfab] restored.

3.4 Setting up Keys

The recommended key bindings are

(global-set-key (kbd "C-.") 'mgl-pax-document)
(global-set-key (kbd "s-x t") 'mgl-pax-transcribe-last-expression)
(global-set-key (kbd "s-x r") 'mgl-pax-retranscribe-region)

The global key bindings above are global because their commands work in any mode. If that's not desired, one may bind C-. locally in all Slime related modes like this:

(slime-bind-keys slime-parent-map nil '(("C-." mgl-pax-document)))

If the customizable variable mgl-pax-hijack-slime-doc-keys is true, then upon loading mgl-pax, the following changes are made to slime-doc-map (assuming it's bound to C-c C-d):

  • C-c C-d a: replaces slime-apropos with mgl-pax-apropos

  • C-c C-d z: replaces slime-apropos-all with mgl-pax-apropos-all

  • C-c C-d p: replaces slime-apropos-package with mgl-pax-apropos-package

  • C-c C-d f: replaces slime-describe-function with mgl-pax-document

  • C-c C-d d: replaces slime-describe-symbol with mgl-pax-hideshow-documentation

  • C-c C-d c: installs mgl-pax-hideshow-comments

  • C-c C-d u: installs mgl-pax-edit-parent-section

Calling mgl-pax-unhijack-slime-doc-keys reverts these changes.

4 Background

As a user, I frequently run into documentation that's incomplete and out of date, so I tend to stay in the editor and explore the code by jumping around with [SLIME][6be7]'s [M-.][cb15] (slime-edit-definition). As a library author, I spend a great deal of time polishing code but precious little writing documentation.

In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual, and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.

That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programmer turned inside out. The original prototype, which did almost everything I wanted, was this:

(defmacro defsection (name docstring)
  `(defun ,name () ,docstring))

Armed with this DEFSECTION, I soon found myself organizing code following the flow of user-level documentation and relegated comments to implementation details entirely. However, some parts of DEFSECTION docstrings were just listings of all the functions, macros and variables related to the narrative, and this list was repeated in the [DEFPACKAGE][9b43] form complete with little comments that were like section names. A clear violation of [OAOO][7d18], one of them had to go, so DEFSECTION got a list of symbols to export.

That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class were named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type, which was later generalized into the concept of locatives:

(defsection @introduction ()
  "A single line for one man ..."
  (foo class)
  (bar function))

After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol, and everything was good for a while.

Then, I realized that sections could refer to other sections if there were a [SECTION][672f] locative. Going down that path, I soon began to feel the urge to generate pretty documentation as all the necessary information was available in the DEFSECTION forms. The design constraint imposed on documentation generation was that following the typical style of upcasing symbols in docstrings, there should be no need to explicitly mark up links: if M-. works, then the documentation generator shall also be able to figure out what's being referred to.

I settled on [Markdown][a317] as a reasonably non-intrusive format, and a few thousand lines later PAX was born. Since then, locatives and references were factored out into the DRef library to let PAX focus on M-. and documentation.

5 Basics

Now let's examine the most important pieces.

  • [macro] DEFSECTION NAME (&KEY (PACKAGE '*PACKAGE*) (READTABLE '*READTABLE*) (EXPORT T) TITLE LINK-TITLE-TO (DISCARD-DOCUMENTATION-P *DISCARD-DOCUMENTATION-P*) CONCEPTS) &BODY ENTRIES

    Define a documentation section and maybe export referenced symbols. A bit behind the scenes, a global variable with NAME is defined and is bound to a [SECTION][5fac] object. By convention, section names start with the character @. See [Introduction][685e] for an example.

    Entries

    ENTRIES consists of docstrings and references in any order. Docstrings are arbitrary strings in Markdown format.

    References are XREFs given in the form (NAME LOCATIVE). For example, (FOO FUNCTION) refers to the function FOO, (@BAR SECTION) says that @BAR is a subsection of this one. (BAZ (METHOD (T T T))) refers to the default method of the three argument generic function BAZ. (FOO FUNCTION) is equivalent to (FOO (FUNCTION)). See the DRef Introduction for more.

    The same name may occur in multiple references, typically with different locatives, but this is not required.

    The references are not LOCATEd until documentation is generated, so they may refer to things yet to be defined.

    Exporting

    If EXPORT is true (the default), NAME and the [name][88cf]s of references among ENTRIES which are [SYMBOL][e5af]s are candidates for exporting. A candidate symbol is exported if

    • it is accessible in PACKAGE, and

    • there is a reference to it in the section being defined which is approved by [EXPORTABLE-REFERENCE-P][e51f].

    See [DEFINE-PACKAGE][63f3] if you use the export feature. The idea with conflating documentation and exporting is to force documentation of all exported symbols and to reduce duplication.

    Misc

    • TITLE is a string containing Markdown or NIL. If non-NIL, it determines the text of the heading in the generated output. LINK-TITLE-TO is a reference given as an (NAME LOCATIVE) pair or NIL, to which the heading will link when generating HTML. If not specified, the heading will link to its own anchor.

    • When DISCARD-DOCUMENTATION-P (defaults to [*DISCARD-DOCUMENTATION-P*][730f]) is true, ENTRIES will not be recorded to save memory.

    • CONCEPTS is a list of concept names and [concept key][b19d]s, for which this section is a primary source. See [Indexing Concepts][c001].

  • [variable] *DISCARD-DOCUMENTATION-P* NIL

    The default value of [DEFSECTION][72b4]'s DISCARD-DOCUMENTATION-P argument. One may want to set *DISCARD-DOCUMENTATION-P* to true before building a binary application.

  • [macro] DEFINE-PACKAGE PACKAGE &BODY OPTIONS

    This is like [CL:DEFPACKAGE][9b43] but silences warnings and errors signalled when the redefined package is at variance with the current state of the package. Typically this situation occurs when symbols are exported by calling EXPORT (as is the case with [DEFSECTION][72b4]) as opposed to adding :EXPORT forms to the DEFPACKAGE form and the package definition is subsequently reevaluated. See the section on package variance in the SBCL manual.

    The bottom line is that if you rely on DEFSECTION to do the exporting, then you'd better use DEFINE-PACKAGE.

  • [macro] DEFINE-GLOSSARY-TERM NAME (&KEY TITLE URL (DISCARD-DOCUMENTATION-P *DISCARD-DOCUMENTATION-P*) CONCEPTS) &BODY DOCSTRING

    Define a global variable with NAME, and set it to a [GLOSSARY-TERM][8251] object. TITLE, URL and DOCSTRING are Markdown strings or NIL. Glossary terms are DOCUMENTed in the lightweight bullet + locative + name/title style. See the glossary entry [name][88cf] for an example.

    • When a glossary term is linked to in documentation, its TITLE will be the link text instead of the name of the symbol (as with [SECTION][5fac]s).

    • Glossary terms with non-NIL URLs are like external links: they are linked to their URL in the generated documentation. These offer a more reliable alternative to using Markdown reference links and are usually not included in SECTIONs.

    • When DISCARD-DOCUMENTATION-P (defaults to [*DISCARD-DOCUMENTATION-P*][730f]) is true, DOCSTRING will not be recorded to save memory.

    • CONCEPTS is a list of concept names and [concept key][b19d]s, for which this term is a primary source. See [Indexing Concepts][c001].

    By [default][8894], glossary terms are indexed alongside concepts.

  • [macro] NOTE &BODY ARGS

    Define a note with an optional NAME and an optional DOCSTRING. The DOCSTRING of the note is its own docstring concatenated with docstrings of other notes in the lexical scope of BODY.

    ARGS has the form [NAME] [DOCSTRING] BODY, where the square brackets indicate optional arguments. See below for the details of parsing ARGS.

    Note: NOTE is experimental and, as such, subject to change.

    NOTE can occur in any evaluated position without changing its BODY's run-time behaviour or introducing any run-time overhead. Top level forms remain top level when wrapped in NOTE. The names of notes live in the same global namespace regardless of nesting or whether they are top level forms. These properties come at the price of NOTE being weird: it defines named notes at macro-expansion time (or load time). But the definitions are idempotent, so it's fine to macroexpand NOTE any number of times.

    Notes are similar to Lisp comments, but they can be included in the documentation. In fact, notes are auto-included: a Specific Link to a note is equivalent to including it with the [DOCSTRING][ce75] locative. Thus, NOTE can double as a tool to avoid boilerplate. An [Unspecific Link][8e71] to a note has no effect.

    Notes are intended to help reduce the distance between code and its documentation when there is no convenient definition docstring to use nearby.

    (note @xxx "We change the package."
  (in-package :mgl-pax))
==> #<PACKAGE "MGL-PAX">
(values (docstring (dref '@xxx 'note)))
=> "We change the package."

    Here is an example of how to overdo things:

    (note @1+*
  "This is a seriously overdone example."
  (defun 1+* (x)
    "[@1+* note][docstring]"
    (if (stringp x)
        (note (@1+*/1 :join #\Newline)
          "- If X is a STRING, then it is parsed as a REAL number."
          (let ((obj (read-from-string x)))
            (note "It is an error if X does not contain a REAL."
              (unless (realp obj)
                (assert nil)))
            (1+ obj)))
        (note "- Else, X is assumed to be a REAL number, and we simply
                 add 1 to it."
          (1+ x)))))

(1+* "7")
=> 8

(values (docstring (dref '@1+* 'note)))
=> "This is a seriously overdone example.

- If X is a STRING, then it is parsed as a REAL number.
It is an error if X does not contain a REAL.

- Else, X is assumed to be a REAL number, and we simply
add 1 to it."

    The parsing of ARGS:

    • If the first element of ARGS is not a string, then it is a NAME (a non-NIL [SYMBOL][e5af]) or name with options, currently destructured as (NAME &KEY JOIN). As in [DEFSECTION][72b4] and [DEFINE-GLOSSARY-TERM][8ece], the convention is that NAME starts with a @ character.

      JOIN is [PRINC][676d]ed before the docstring of a child note is output. Its default value is a string of two newline characters.

    • The next element of ARGS is a Markdown docstring. See [Markdown in Docstrings][7bf5].

    • The rest of ARGS is the BODY. If BODY is empty, then NIL is returned.

    Note that named and nameless notes can contain other named or nameless notes without restriction, but nameless notes without a lexically enclosing named note are just an [implicit progn][c2fd] with BODY, and their docstring is discarded.

    If NOTE occurs as a top level form, then its SOURCE-LOCATION is reliably recorded. Else, the quality of the source location varies, but it is at least within the right top level form on all implementations. On SBCL, exact source location is supported.

6 PAX Locatives

In addition to DRef's own, PAX defines a few locative types using the facilities described in Adding New Locatives. Locatives allow referenceing definitions, which is used in [DEFSECTION][72b4], Navigating Sources in Emacs and docstrings (see [Codification][f1ab] and Linking in the context of Generating Documentation).

  • [locative] SECTION

    • Direct locative supertypes: VARIABLE

    • Direct locative subtypes: DYNAMIC-SECTION

    Refers to a [SECTION][5fac] defined by [DEFSECTION][72b4].

    SECTION is [EXPORTABLE-LOCATIVE-TYPE-P][c930] but not exported by default (see [EXPORTABLE-REFERENCE-P][e51f]).

  • [locative] GLOSSARY-TERM

    • Direct locative supertypes: VARIABLE

    Refers to a [GLOSSARY-TERM][8251] defined by [DEFINE-GLOSSARY-TERM][8ece].

    GLOSSARY-TERM is [EXPORTABLE-LOCATIVE-TYPE-P][c930] but not exported by default (see [EXPORTABLE-REFERENCE-P][e51f]).

  • [locative] CONCEPT

    • Direct locative supertypes: VARIABLE

    Refers to a named group of [concept key][b19d]s defined by [DEFINE-CONCEPT][b80d] for multiplexing its keys to definitions that link to it in their docstrings. See [Indexing Concepts][c001] for more.

    CONCEPT is [EXPORTABLE-LOCATIVE-TYPE-P][c930] but not exported by default (see [EXPORTABLE-REFERENCE-P][e51f]).

    CONCEPTs have no ARGLIST or DOCSTRING.

  • [locative] NOTE

    Refers to named notes defined by the [NOTE][e2ae] macro.

    When Generating Documentation, if a single link would be made to a NOTE (be it either a Specific Link or an unambiguous [Unspecific Link][8e71]), then the NOTE's DOCSTRING is included as if with the [DOCSTRING][ce75] locative.

    NOTE is [EXPORTABLE-LOCATIVE-TYPE-P][c930] but not exported by default (see [EXPORTABLE-REFERENCE-P][e51f]).

  • [locative] DISLOCATED

    Refers to a symbol in a non-specific context. Useful for suppressing [Unspecific Autolink][e2a4]ing. For example, if there is a function called FOO then

      `FOO`

    will be linked (if [*DOCUMENT-LINK-CODE*][d9ee]) to its definition. However,

      [`FOO`][dislocated]

    will not be. With a dislocated locative, LOCATE always fails with a LOCATE-ERROR condition. Also see [Escaping Autolinking][a270].

    DISLOCATED references do not RESOLVE.

  • [locative] ARGUMENT

    An alias for [DISLOCATED][e391], allowing one to refer to an argument of a macro without accidentally linking to a class that has the same name as that argument. In the following example, FORMAT may link to CL:FORMAT (if we generated documentation for it):

    "See FORMAT in DOCUMENT."

    Since ARGUMENT is a locative, we can prevent that linking by writing:

    "See the FORMAT argument of DOCUMENT."

    ARGUMENT references do not RESOLVE.

  • [locative] INCLUDE SOURCE &KEY LINE-PREFIX HEADER FOOTER HEADER-NL FOOTER-NL

    This PSEUDO locative refers to a region of a file. SOURCE can be a [STRING][b93c] or a PATHNAME, in which case the whole file is being pointed to, or it can explicitly supply START, END locatives. INCLUDE is typically used to include non-lisp files in the documentation (say Markdown or Elisp as in the next example) or regions of Lisp source files. This can reduce clutter and duplication.

    (defsection @example-section ()
  (mgl-pax.el (include #.(asdf:system-relative-pathname
                          :mgl-pax "src/mgl-pax.el")
                       :header-nl "```elisp" :footer-nl "```"))
  (foo-example (include (:start (dref-ext:make-source-location function)
                         :end (dref-ext:source-location-p function))
                        :header-nl "```"
                        :footer-nl "```")))

    In the above example, when documentation is generated, the entire src/mgl-pax.el file is included in the Markdown output surrounded by the strings given as HEADER-NL and FOOTER-NL. The documentation of FOO-EXAMPLE will be the region of the file from the SOURCE-LOCATION of the START reference (inclusive) to the SOURCE-LOCATION of the END reference (exclusive). If only one of START and END is specified, then they default to the beginning and end of the file, respectively.

    Since START and END are literal references, pressing M-. on PAX.EL will open the src/mgl-pax.el file and put the cursor on its first character. M-. on FOO-EXAMPLE will go to the source location of the FOO function.

    With the LAMBDA locative, one can specify positions in arbitrary files.

    • SOURCE is either an absolute pathname designator or a list matching the [destructuring lambda list][6067] (&KEY START END), where START and END must be NIL or (<NAME> <LOCATIVE>) lists (not evaluated) like a [DEFSECTION][72b4] entry. Their SOURCE-LOCATIONs constitute the bounds of the region of the file to be included. Note that the file of the source location of START and END must be the same. If SOURCE is a pathname designator, then it must be absolute so that the locative is context independent.

    • If specified, LINE-PREFIX is a string that's prepended to each line included in the documentation. For example, a string of four spaces makes Markdown think it's a code block.

    • HEADER and FOOTER, if non-NIL, are printed before the included string.

    • HEADER-NL and FOOTER-NL, if non-NIL, are printed between two FRESH-LINE calls.

    INCLUDE is not [EXPORTABLE-LOCATIVE-TYPE-P][c930], and INCLUDE references do not RESOLVE.

  • [locative] DOCSTRING

    DOCSTRING is a PSEUDO locative for including the parse tree of the Markdown DOCSTRING of a definition in the parse tree of a docstring when generating documentation. It has no source location information and only works as an explicit link. This construct is intended to allow docstrings to live closer to their implementation, which typically involves a non-exported definition.

    (defun div2 (x)
  "X must be [even* type][docstring]."
  (/ x 2))

(deftype even* ()
  "an even integer"
  '(satisfies evenp))

(document #'div2)
.. - [function] DIV2 X
..
..     X must be an even integer.
..

    There is no way to LOCATE DOCSTRINGs, so nothing to RESOLVE either.

  • [locative] GO (NAME LOCATIVE)

    Redirect to a definition in the context of the reference designated by NAME and LOCATIVE. This PSEUDO locative is intended for things that have no explicit global definition.

    As an example, consider this part of a hypothetical documentation of CLOS:

      (defsection @clos ()
    (defmethod macro)
    (call-next-method (go (defmethod macro))))

    The GO reference exports the symbol [CALL-NEXT-METHOD][6832] and also produces a terse redirection message in the documentation.

    GO behaves as described below.

    • A GO reference RESOLVEs to what NAME with LOCATIVE resolves to:

      (resolve (dref 'xxx '(go (print function))))
==> #<FUNCTION PRINT>
=> T

    • The DOCSTRING of a GO reference is NIL.

    • SOURCE-LOCATION (thus M-.) returns the source location of the embedded reference:

      (equal (source-location (dref 'xxx '(go (print function))))
       (source-location (dref 'print 'function)))
=> T

  • [locative] CLHS &OPTIONAL NESTED-LOCATIVE

    Refers to definitions, glossary entries, sections, issues and issue summaries in the Common Lisp HyperSpec. These have no source location so [M-.][cb15] will not work. What works is linking in documentation, including [Browsing Live Documentation][a595]. The generated links are relative to [*DOCUMENT-HYPERSPEC-ROOT*][f585] and work even if [*DOCUMENT-LINK-TO-HYPERSPEC*][875e] is NIL. All matching is case-insensitive.

    • definitions: These are typically unnecessary as DOCUMENT will produce the same link for e.g. PPRINT, [PPRINT][function], or [PPRINT][] if *DOCUMENT-LINK-TO-HYPERSPEC* is non-NIL and the [PPRINT][6af6] function in the running Lisp is not [linkable][7eb5]. When [Browsing Live Documentation][a595], a slight difference is that everything is linkable, so using the CLHS link bypasses the page with the definition in the running Lisp.

      • unambiguous definition: [pprint][clhs] ([pprint][6af6])

      • disambiguation page: [function][clhs] ([function][aeb6])

      • specific: [function][(clhs class)] (function)

    • glossary terms:

      • [lambda list][(clhs glossary-term)] ([lambda list][98ff])

    • issues:

      • [ISSUE:AREF-1D][clhs] ([ISSUE:AREF-1D][63ef])

      • [ISSUE:AREF-1D][(clhs section)] ([ISSUE:AREF-1D][63ef])

    • issue summaries: These render as ([SUMMARY:CHARACTER-PROPOSAL:2-6-5][935f]):

      • [SUMMARY:CHARACTER-PROPOSAL:2-6-5][clhs]

      • [SUMMARY:CHARACTER-PROPOSAL:2-6-5][(clhs section)]

      Since these summary ids are not particularly reader friendly, the anchor text a [Specific Reflink with Text][fb17] may be used:

      • [see this][SUMMARY:CHARACTER-PROPOSAL:2-6-5 (clhs section)] ([see this][935f]).

    • sections:

      • by section number: [3.4][clhs] or [3.4][(clhs section)] ([3.4][e442])

      • by section title: With the locative (CLHS SECTION), substring matching against the title starting at word boundaries is performed. With the locative [CLHS][ed5f] (where SECTION is not specified explicitly), the name must match the title exactly. For example, [lambda list][(clhs section)] refers to the same definition as [lambda lists][clhs] ([Lambda Lists][e442]).

      • by filename: [03_d][clhs] or [03_d][(clhs section)] ([03_d][e442])

      • by alias

        • [Format directives][d273] are aliases of the sections describing them. Thus, [~c][clhs] is equivalent to [22.3.1.1][clhs] and [Tilde C: Character][clhs]. The full list is ~C [%][7bd6] ~& ~| [~~][65bc] [R][9927] ~D [B][6897] [O][76df] [X][f9fa] [F][cae2] ~E [G][76ab] [$][5b4d] [A][f275] [S][b39f] [W][e6d3] ~_ [<][ed9f] [:>][ed9f] ~I [/][db38] ~T ~< Justification [>][bf38] [*][bdd6] [[][76ea] []][945b] ~{ [}][886c] ~? [(][ac30] ~) [P][60e4] [;][d296] ~^ ~Newline.

        • Similarly, reader macro characters are aliases of the sections describing them. The full list is ( [)][e43c] ['][8e92] [;][56ba] ["][cd66] ` [,][826b] [#][698d] [#\][7b6f] [#'][8a5e] [#(][7a7f] [#*][adf2] [#:][ac5e] [#.][ffd7] [#B][c93e] [#O][68d2] #X #R [#C][bfaa] [#A][7163] [#S][dded] #P [#=][fa43] [##][dd03] [#+][d5e1] [#-][81b3] #| #< [#)][9478].

        • Finally, loop keywords have aliases to the sections describing them. For example, the strings loop:for, for and :for are aliases of CLHS 6.1.2.1. The loop:* aliases are convenient for completion at the prompt when [Browsing Live Documentation][a595], while the other aliases are for defaulting to buffer contents.

    As the above examples show, the NESTED-LOCATIVE argument of the [CLHS][ed5f] locative may be omitted. In that case, definitions, glossary terms, issues, issue summaries, and sections are considered in that order. Sections are considered last because a substring of a section title can be matched by chance easily.

    All examples so far used [Reflink][cbc4]s. [Autolink][ec7a]ing also works if the [name][88cf] is marked up as code or is [codified][f1ab] (e.g. in COS clhs ([COS][c4a3] clhs).

    As mentioned above, M-. does not do anything over CLHS references. Slightly more usefully, the [live documentation browser][a595] understands CLHS links so one can enter inputs like 3.4 clhs, "lambda list" clhs or error (clhs function).

    CLHS references do not RESOLVE.

7 Navigating Sources in Emacs

Integration into [SLIME][6be7]'s [M-.][cb15] (slime-edit-definition) allows one to visit the SOURCE-LOCATION of a definition. PAX extends standard Slime functionality by

  • adding support for all kinds of definitions (see e.g. ASDF:SYSTEM, READTABLE in Basic Locative Types), not just the ones Slime knows about,

  • providing a portable way to refer to even standard definitions,

  • disambiguating the definition based on buffer content, and

  • adding more powerful completions.

The definition is either determined from the buffer content at point or is prompted for. At the prompt, TAB-completion is available for both names and locatives. With a prefix argument (C-u M-.), the buffer contents are not consulted, and M-. always prompts.

The M-. extensions can be enabled by loading src/mgl-pax.el. See [Emacs Setup][8541]. In addition, the Elisp command mgl-pax-edit-parent-section visits the source location of the section containing the definition with point in it.

A close relative of M-. is C-. for [Browsing Live Documentation][a595].

7.1 M-. Defaulting

When [M-.][cb15] is invoked, it first tries to find a [name][88cf] in the current buffer at point. If no name is found, then it [prompts][ed46].

First, (slime-sexp-at-point) is taken as a [word][d7b0], from which the [name][88cf] will be parsed. Then, candidate locatives are looked for before and after the [word][d7b0]. Thus, if a locative is the previous or the next expression, then M-. will go straight to the definition which corresponds to the locative. If that fails, M-. will try to find the definitions in the normal way, which may involve popping up an xref buffer and letting the user interactively select one of possible definitions.

M-. works on parenthesized references, such as those in [DEFSECTION][72b4]:

(defsection @foo ()
  (cos function))

Here, when the cursor is on one of the characters of COS or just after COS, pressing M-. will visit the definition of the function [COS][c4a3].

To play nice with Generating Documentation, forms suitable for [Autolink][ec7a]ing are recognized:

function cos
cos function

... as well as [Reflink][cbc4]s:

[cos][function]
[see this][cos function]

... and [Markdown inline code][68c1]:

cos `function`
`cos` function
`cos` `function`

Everything works the same way in comments and docstrings as in code. In the next example, pressing M-. on RESOLVE* will visit its denoted method:

;;; See RESOLVE* (method (function-dref)) for how this all works.

7.2 M-. Prompting

7.2.1 M-. Minibuffer Syntax

At the minibuffer prompt, the definitions to edit can be specified as follows.

  • NAME: Refers to all DREF:DEFINITIONS of NAME with a Lisp locative type. See these NAME -> DEFINITIONS examples:

      print    ->  PRINT FUNCTION
  PRINT    ->  PRINT FUNCTION
  MGL-PAX  ->  "mgl-pax" ASDF:SYSTEM, "MGL-PAX" package
  pax      ->  "PAX" PACKAGE
  "PAX"    ->  "PAX" PACKAGE

    Note that depending on the Lisp implementation there may be more definitions. For example, SBCL has an UNKNOWN :DEFOPTIMIZER definition for PRINT.

  • NAME LOCATIVE: Refers to a single definition (as in (DREF:DREF NAME LOCATIVE)). Example inputs of this form:

      print function
  dref-ext:docstring* (method (t))

  • LOCATIVE NAME: This has the same form as the previous: two sexps, but here the first one is the locative. If ambiguous, this is considered in addition to the previous one. Example inputs:

      function print
  (method (t)) dref-ext:docstring*

In all of the above NAME is a [raw name][f5af], meaning that print will be recognized as PRINT and pax as "PAX".

The package in which symbols are read is the Elisp slime-current-package. In Lisp buffers, this is the buffer's package, else it's the package of the Slime repl buffer.

7.2.2 M-. Completion

When M-. prompts for the definition to edit, TAB-completion is available in the minibuffer for both names and locatives. To reduce clutter, string names are completed only if they are typed explicitly with an opening quotation mark, and they are case-sensitive. Examples:

  • pri<TAB> invokes the usual Slime completion.

  • print <TAB> (note the space) lists [FUNCTION][a51f] and (PAX:CLHS [FUNCTION][aeb6]) as locatives.

  • class dref:<TAB> lists DREF:XREF and DREF:DREF (all the classes in the package DREF).

  • pax:locative <TAB> lists all locative types (see the CL function DREF:LOCATIVE-TYPES).

  • package "MGL<TAB> lists the names of packages that start with "MGL".

  • package <TAB> lists the names of all packages as strings and also CLASS, MGL-PAX:LOCATIVE because PACKAGE denotes a class and also a locative.

For more powerful search, see [Apropos][b7fc].

8 Generating Documentation

8.1 The DOCUMENT Function

  • [function] DOCUMENT DOCUMENTABLE &KEY (STREAM T) PAGES (FORMAT :PLAIN)

    Write DOCUMENTABLE in FORMAT to STREAM diverting some output to PAGES. FORMAT is one of [:PLAIN][c879], [:MARKDOWN][dd29], :HTML, :PDF or [NIL][f7e6]. STREAM may be a [STREAM][d5a9] object, T or NIL as with [CL:FORMAT][ad78].

    To look up the documentation of the DOCUMENT function itself:

      (document #'document)

    The same with fancy markup:

      (document #'document :format :markdown)

    To document a [SECTION][5fac]:

      (document pax::@pax-manual)

    To generate the documentation for separate libraries with automatic cross-links:

      (document (list pax::@pax-manual dref::@dref-manual) :format :markdown)

    See Utilities for Generating Documentation for more.

    Definitions that do not define a first-class object are supported via DRef:

      (document (dref:locate 'foo 'type))

    There are quite a few special variables that affect how output is generated. See [Codification][f1ab], [Linking to the HyperSpec][7cc3], Linking to Sections, [Link Format][c0d2] and [Output Formats][8d9b].

    For the details, see the following sections, starting with DOCUMENTABLE. Also see [Writing Extensions][c4ce] and [DOCUMENT-OBJECT*][8269].

8.1.1 DOCUMENTABLE

The DOCUMENTABLE argument of DOCUMENT may be a single object (e.g. #'PRINT), a definition such as (DREF 'PRINT 'FUNCTION), a string, or a nested list of these. More precisely, DOCUMENTABLE is one of the following:

  • single definition designator: A DREF or anything else that is LOCATEable. This includes non-DREF XREFs and first-class objects such as FUNCTIONs. The generated documentation typically includes the definition's DOCSTRING. See [Markdown Output][dd29] for more.

  • docstring: A string, in which case it is processed like a docstring in [DEFSECTION][72b4]. That is, with [docstring sanitization][7bf5], [Codification][f1ab], and Linking.

  • list of documentables: A nested list of LOCATEable objects and docstrings. The objects in it are documented in depth-first order. The structure of the list is otherwise unimportant.

  • [variable] *DOCUMENT-TIGHT* NIL

    Normally, DOCUMENT calls [ENSURE-MD-PARAGRAPH][a021] before and after documenting each atomic DOCUMENTABLE. If *DOCUMENT-TIGHT* is true, then this doesn't happen for top-level string documentables.

8.1.2 Return Values

If PAGES are NIL, then DOCUMENT - like [CL:FORMAT][ad78] - returns a string (when [STREAM][d5a9] is NIL) else NIL.

If PAGES, then a list of output designators are returned, one for each non-empty page (to which some output has been written), which are determined as follows.

  • The string itself if the output was to a string.

  • The stream if the output was to a stream.

  • The pathname of the file if the output was to a file.

If the default page given by the STREAM argument of DOCUMENT was written to, then its output designator is the first element of the returned list. The rest of the designators correspond to the non-empty pages in the PAGES argument of DOCUMENT in that order.

8.1.3 PAGES

The PAGES argument of DOCUMENT is to create multi-page documents by routing some of the generated output to files, strings or streams. PAGES is a list of page specs. A page spec is a [property list][b18e] with keys :OBJECTS, :OUTPUT, :URI-FRAGMENT, :SOURCE-URI-FN, :HEADER-FN and :FOOTER-FN.

PAGES may look something like this:

`((;; The section about SECTIONs and everything below it ...
   :objects (, @sections)
   ;; ... is so boring that it's not worth the disk space, so
   ;; send it to a string.
   :output (nil)
   ;; Explicitly tell other pages not to link to these guys.
   :uri-fragment nil)
  ;; Send the @EXTENSION-API section and everything reachable
  ;; from it ...
  (:objects (, @extension-api)
   ;; ... to build/tmp/pax-extension-api.html.
   :output "build/tmp/pax-extension-api.html"
   ;; However, on the web server html files will be at this
   ;; location relative to some common root, so override the
   ;; default:
   :uri-fragment "doc/dev/pax-extension-api.html"
   ;; Set html page title, stylesheet, charset.
   :header-fn 'write-html-header
   ;; Just close the body.
   :footer-fn 'write-html-footer)
  ;; Catch references that were not reachable from the above. It
  ;; is important for this page spec to be last.
  (:objects (, @pax-manual)
   :output "build/tmp/manual.html"
   ;; Links from the extension api page to the manual page will
   ;; be to ../user/pax-manual#<anchor>, while links going to
   ;; the opposite direction will be to
   ;; ../dev/pax-extension-api.html#<anchor>.
   :uri-fragment "doc/user/pax-manual.html"
   :header-fn 'write-html-header
   :footer-fn 'write-html-footer))

Documentation is initially sent to a default stream (the STREAM argument of DOCUMENT), but output is temporary redirected to :OUTPUT for the duration of generating documentation for a docstring or definition that matches :OBJECTS, shadowing any existing redirection if any. This nesting typically happens when a [SECTION][5fac] references another definition and they both have their own page specs.

:OBJECTS is a list of objects whose definitions are LOCATEable. In addition, docstrings can be included. The latter can be useful if DOCUMENTABLE includes a docstring.

  • A docstring matches :OBJECTS if it is [EQ][5a82] to one of its elements.

  • A definition matches :OBJECTS if it is XREF= to one of its DREF elements.

If multiple page specs match, then the first one has precedence.

  • :OUTPUT can be a number of things:

    • If it's NIL, then output will be collected in a string.

    • If it's T, then output will be sent to [*STANDARD-OUTPUT*][e7ee].

    • If it's a stream, then output will be sent to that stream.

    • If it's a list whose first element is a string or a pathname, then output will be sent to the file denoted by that and the rest of the elements of the list are passed on to [CL:OPEN][6547]. One extra keyword argument is :ENSURE-DIRECTORIES-EXIST. If it's true, [ENSURE-DIRECTORIES-EXIST][876d] will be called on the pathname before it's opened.

    Note that even if PAGES is specified, STREAM acts as a catch all, absorbing the generated documentation for references not claimed by any pages.

  • :HEADER-FN, if not NIL, is a function of a single stream argument, which is called just before the first write to the page. Since :FORMAT :HTML only generates HTML fragments, this makes it possible to print arbitrary headers, typically setting the title, CSS stylesheet, or charset.

  • :FOOTER-FN is similar to :HEADER-FN, but it's called after the last write to the page. For HTML, it typically just closes the body.

  • :URI-FRAGMENT is a string such as "doc/manual.html" that specifies where the page will be deployed on a webserver. It defines how links between pages will look. If it's not specified and :OUTPUT refers to a file, then it defaults to the name of the file. If :URI-FRAGMENT is NIL, then no links will be made to or from that page.

  • :SOURCE-URI-FN is a function of a single, DREF argument. If it returns a value other than NIL, then it must be a string representing an URI. This affects [*DOCUMENT-MARK-UP-SIGNATURES*][8fb6] and [*DOCUMENT-FANCY-HTML-NAVIGATION*][6ab0]. Also see [MAKE-GIT-SOURCE-URI-FN][587f].

8.1.4 Package and Readtable

While generating documentation, symbols may be read from docstrings and printed. Our goal in general is to use the [*PACKAGE*][5ed1] and [*READTABLE*][b79a] in effect at the time the docstring was [READ][fe58]. This keeps the correspondence between

  • [M-.][cb15] and Linking, and

  • interned symbols and [Codification][f1ab].

What values of *PACKAGE* and *READTABLE* are used is determined separately for each definition being documented. For a [SECTION][5fac], its [SECTION-PACKAGE][a5b1] and [SECTION-READTABLE][88a7] are in effect.

For non-SECTION definitions, the package and the readtable are given by the following rules. If multiple rules provide a non-NIL package, then the first such rule takes precedence and similarly for the readtable.

  • DOCSTRING: The second value returned by DOCSTRING for the definition provides the package.

    • If the value of *PACKAGE* in effect at the time of definition was captured (e.g. by DEFINE-LOCATIVE-TYPE), then DOCSTRING returns that.

    • The user may provide defaults for the package. See DREF-EXT:DOCSTRING* (especially the package-wide default).

  • Home section: If the definition has a [HOME-SECTION][fda4], then the home section's SECTION-PACKAGE and SECTION-READTABLE are used.

  • Arglist heuristic: If the definition has an argument list, then the package of the first argument that's not external in any package is used.

  • Name heuristic: If the definition is named by a symbol, then its [SYMBOL-PACKAGE][e5ab] is used, and *READTABLE* is set to the standard readtable (NAMED-READTABLES:FIND-READTABLE :COMMON-LISP).

  • Default: *PACKAGE* is set to the CL-USER package and *READTABLE* to the standard readtable.

The values thus determined come into effect after the name itself is printed, for printing of the arglist and the docstring.

CL-USER> (pax:document #'foo)
- [function] FOO <!> X Y &KEY (ERRORP T)

    Do something with X and Y.

In the above, the <!> marks the place where *PACKAGE* and *READTABLE* are bound.

  • [function] HOME-SECTION OBJECT

    The home section of an object is a [SECTION][5fac] that contains the object's definition in its [SECTION-ENTRIES][d850] or NIL. In the overwhelming majority of cases there should be at most one containing section.

    If there are multiple containing sections, the following apply.

    • If the name of the definition is a non-keyword symbol, only those containing sections are considered whose package is closest to the [SYMBOL-PACKAGE][e5ab] of the name, where closest is defined as having the longest common prefix between the two [PACKAGE-NAME][db68]s.

    • If there are multiple sections with equally long matches or the name is not a non-keyword symbol, then it's undefined which one is the home section.

    For example, (MGL-PAX:DOCUMENT FUNCTION) is an entry in the MGL-PAX::@BASICS section. Unless another section that contains it is defined in the MGL-PAX package, the home section is guaranteed to be MGL-PAX::@BASICS because the SYMBOL-PACKAGEs of MGL-PAX:DOCUMENT and MGL-PAX::@BASICS are the same (hence their common prefix is maximally long).

    This scheme would also work, for example, if the home package of DOCUMENT were MGL-PAX/IMPL, and it were reexported from MGL-PAX because the only way to externally change the home package would be to define a containing section in a package like MGL-PAX/IMP.

    Thus, relying on the package system makes it possible to find the intended home section of a definition among multiple containing sections with high probability. However, for names which are not symbols, there is no package system to take advantage of.

  • [variable] *DOCUMENT-NORMALIZE-PACKAGES* T

    Whether to print [in package <package-name>] in the documentation when the package changes.

8.2 Browsing Live Documentation

Documentation for definitions in the running Lisp can be browsed directly without generating documentation in the offline manner. HTML documentation, complete with [Codification][f1ab] and Linking, is generated from docstrings of all kinds of definitions and PAX [SECTION][5fac]s in the running Lisp on the fly. This allows ad-hoc exploration of the Lisp, much like describe-function, apropos-command and other online help commands in Emacs, for which direct parallels are provided.

Still, even without Emacs and [SLIME][6be7], limited functionality is available through the [PAX Live Home Page][9d50] by starting the live documentation web server [manually][72cc].

If [Emacs Setup][8541] has been done, the Elisp function mgl-pax-document (maybe bound to C-.) generates and displays documentation as a single HTML page. If necessary, a disambiguation page is generated with the documentation of all matching definitions. For example, to view the documentation of this very SECTION, one can do:

M-x mgl-pax-document
View Documentation of: pax::@browsing-live-documentation

Alternatively, pressing C-. with point over the text pax::@browsing-live-documentation in a buffer achieves the same effect.

In interactive use, mgl-pax-document behaves similarly to M-. except:

  • It shows the DOCUMENTation of some definition and does not visit its SOURCE-LOCATION.

  • It considers definitions with all LOCATIVE-TYPES not just LISP-LOCATIVE-TYPES because it doesn't need SOURCE-LOCATION.

    This also means that completion works for [CLHS][ed5f] definitions:

    • "lambda list<TAB> lists "lambda list" and "lambda list keywords", both HyperSpec glossary entries. This is similar to common-lisp-hyperspec-glossary-term in Elisp but also works for HyperSpec section titles.

    • "#<TAB> lists all sharpsign reader macros (similar to common-lisp-hyperspec-lookup-reader-macro in Elisp).

    • "~<TAB> lists all [CL:FORMAT][ad78] directives (similar to common-lisp-hyperspec-format in Elisp).

    • "loop:~<TAB> lists all [loop keywords][e016].

  • It works in non-lisp-mode buffers by reinterpreting a few lines of text surrounding point as lisp code (hence the suggested global binding) in package MGL-PAX.

  • It supports fragment syntax at the prompt:

      <name> <locative> <fragment-name> <fragment-locative>

    This is like <NAME> <LOCATIVE>, but the browser scrolls to the definition of <FRAGMENT-NAME> <FRAGMENT-LOCATIVE> within that page.

    For example, entering this at the prompt will generate the entire PAX manual as a single page and scroll to the very section you are reading within it:

      pax::@pax-manual pax:section pax::@browsing-live-documentation pax:section

  • If the empty string is entered at the prompt, and there is no existing w3m buffer or w3m is not used, then [PAX Live Home Page][9d50] is visited. If there is a w3m buffer, then entering the empty string displays that buffer.

The convenience function mgl-pax-current-definition-toggle-view (C-c C-d c) documents the definition with point in it.

  • [variable] *BROWSE-CONTEXT*

    When set to a DOCUMENTABLE, live browsing will link only to definitions reachable from it as opposed to linking to all definitions in the running Lisp when it is unbound. This makes it easier to see how offline generated documentation will turn out.

    For convenience, setting *BROWSE-CONTEXT* to :PAX-WORLD is equivalent to listing all [registered][f4fd] sections.

8.2.1 Browsing with w3m

When the value of the Elisp variable mgl-pax-browser-function is w3m-browse-url (see [Emacs Setup][8541]), the Emacs w3m browser is used without the need for a web server, and also offering somewhat tighter integration than [Browsing with Other Browsers][c434].

With w3m's default key bindings, moving the cursor between links involves TAB and S-TAB (or <up> and <down>). RET and <right> follow a link, while B and <left> go back in history.

In addition, the following PAX-specific key bindings are available:

  • M-. visits the source location of the definition corresponding to the link under the point.

  • Invoking mgl-pax-document on a section title link will show the documentation of that section on its own page.

  • n moves to the next PAX definition on the page.

  • p moves to the previous PAX definition on the page.

  • u follows the first Up: link (to the first containing [SECTION][5fac]) if any.

  • U is like u but positions the cursor at the top of the page.

  • v visits the source location of the current definition (the one under the cursor or the first one above it).

  • V visits the source location of the first definition on the page.

8.2.2 Browsing with Other Browsers

When the value of the Elisp variable mgl-pax-browser-function is not w3m-browse-url (see [Emacs Setup][8541]), requests are served via a web server started in the running Lisp, and documentation is most likely displayed in a separate browser window.

By default, mgl-pax-browser-function is nil, which makes PAX use browse-url-browser-function. You may want to customize the related browse-url-new-window-flag or, for Chrome, set browse-url-chrome-arguments to ("--new-window").

By default, mgl-pax-web-server-port is nil, and PAX will pick a free port automatically.

In the browser, clicking on the locative on the left of the name (e.g. in - [function] PRINT) will raise and focus the Emacs window (if Emacs is not in text mode, and also subject to window manager focus stealing settings), then go to the corresponding source location. For sections, clicking on the lambda link will do the same (see [*DOCUMENT-FANCY-HTML-NAVIGATION*][6ab0]).

Finally, note that the URLs exposed by the web server are subject to change, and even the port used may vary by session if the Elisp variable mgl-pax-web-server-port is nil.

  • [variable] *BROWSE-HTML-STYLE* :CHARTER

    The HTML style to use for browsing live documentation. Affects only non-w3m browsers. See [*DOCUMENT-HTML-DEFAULT-STYLE*][90fa] for the possible values.

    If you change this variable, you may need to do a hard refresh in the browser (often C-<f5>).

8.2.3 Apropos

The Elisp functions mgl-pax-apropos, mgl-pax-apropos-all, and mgl-pax-apropos-package can display the results of DREF-APROPOS in the [live documentation browser][a595]. These extend the functionality of slime-apropos, slime-apropos-all and slime-apropos-package to support more kinds of definitions in an extensible way. The correspondence is so close that the PAX versions might [take over the Slime key bindings][8541].

Note that apropos functionality is also exposed via the [PAX Live Home Page][9d50].

More concretely, the PAX versions supports the following extensions:

  • Definitions with string names. One can search for ASDF:SYSTEMs, PACKAGEs and [CLHS][ed5f] sections, glossary entries, format directives, reader macro characters, loop keywords.

  • Exact or substring matching of the name and the package.

  • Matching only symbol or string names.

On the [PAX Live Home Page][9d50], one may [Browse by Locative Types][659d], which gives access to some of the apropos functionality via the browser without involving Emacs.

On the result page:

  • A DREF-APROPOS form to reproduce the results at the REPL is shown.

  • One may toggle the EXTERNAL-ONLY and CASE-SENSITIVE boolean arguments.

  • One may switch between list, and detailed view. The list view only shows the first, [bulleted line][dd29] for each definition, while the detailed view includes the full documentation of definitions with the exception of [SECTION][5fac]s.

  • The returned references are presented in two groups: those with non-symbol and those with symbol [name][88cf]s. The non-symbol group is sorted by locative type then by name. The symbol group is sorted by name then by locative type.

With mgl-pax-apropos-all and mgl-pax-apropos-package being simple convenience functions on top of mgl-pax-apropos, we only discuss the latter in detail here. For the others, see the Elisp docstrings.

The STRING Argument of mgl-pax-apropos

The STRING argument consists of a name pattern and a DTYPE.

The name pattern has the following forms.

  • :print matches definitions whose names are the string print or a symbol with SYMBOL-NAME print. Vertical bar form as in :|prInt| is also also supported and is useful in when CASE-SENSITIVE is true.

  • "print" matches definitions whose names contain print as a substring.

  • print is like the previous, substring matching case. Use this form to save typing if the pattern does not contain spaces and does not start with a colon.

  • The empty string matches everything.

After the name pattern, STRING may contain a DTYPE that the definitions must match.

  • print t matches definitions with LISP-LOCATIVE-TYPES, which is the default (equivalent to print).

  • print function matches functions whose names contain print (e.g. [CL:PRINT][d451] and [CL:PPRINT][6af6]).

  • :print function is like the previous example but with exact name match (so it matches CL:PRINT but not CL:PPRINT).

  • print variable matches for example [*PRINT-ESCAPE*][ff76].

  • print (or variable function) matches all variables and functions with print in their names.

  • array (or type (not class)) matches [DEFTYPE][7f9a]s and but not CLASSes with the string array in their names.

  •  pax:section (note the leading space) matches all PAX sections (EXTERNAL-ONLY NIL is necessary to see many of them).

  • print dref:pseudo matches definitions with PSEUDO-LOCATIVE-TYPES such as MGL-PAX:CLHS.

  • print dref:top matches definitions with all locative types (LOCATIVE-TYPES).

The PACKAGE Argument of mgl-pax-apropos

When mgl-pax-apropos is invoked with a prefix argument, it prompts for a package pattern among other things. The pattern may be like the following examples.

  • :none restricts matches to non-symbol names.

  • :any restricts matches to symbol names.

  • :cl restricts matches to symbols in the CL package.

  • :|X Y| is similar to the previous, but the vertical bar syntax allows for spaces in names.

  • mgl restricts matches to packages whose name contains mgl as a substring.

  • "x y" is the same as the previous, but the explicit quotes allow for spaces in names.

The above examples assume case-insensitive matching.

8.2.4 PAX Live Home Page

When [Browsing Live Documentation][a595], the home page provides quick access to documentation of the definitions in the system. In Emacs, when mgl-pax-document is invoked with the empty string, it visits the home page.

The home page may also be accessed directly by going to the root page of the web server (if one is started). Here, unless the home page is viewed [with w3m][83d5], one may directly look up documentation and access [Apropos][b7fc] via the input boxes provided.

  • [function] ENSURE-WEB-SERVER &KEY PORT HYPERSPEC-ROOT

    Start or update a web server on PORT of localhost for [Browsing Live Documentation][a595]. Returns the base URL of the server (e.g. http://localhost:32790), which goes to the [PAX Live Home Page][9d50]. If the web server is running already (ENSURE-WEB-SERVER) simply returns its base URL.

    Note that even when using Emacs but [Browsing with Other Browsers][c434], the web server is started automatically. When [Browsing with w3m][83d5], no web server is involved at all. Calling this function explicitly is only needed if the Emacs integration is not used, or to override PORT and HYPERSPEC-ROOT.

    • If PORT is NIL or 0, then the server will use any free port.

    • If there is a server already running and PORT is not NIL or 0, then the server is restarted on PORT.

    • If HYPERSPEC-ROOT is NIL, the HyperSpec pages will be served from any previously provided HYPERSPEC-ROOT or, failing that, from [*DOCUMENT-HYPERSPEC-ROOT*][f585].

    • If HYPERSPEC-ROOT is non-NIL, then pages in the HyperSpec will be served from HYPERSPEC-ROOT. The following command changes the root without affecting the server in any other way:

        (ensure-web-server :hyperspec-root "/usr/share/doc/hyperspec/")

Top-level PAX Sections

The [PAX Live Home Page][9d50] lists the top-level PAX sections: those that have no other [SECTION][5fac]s referencing them (see [HOME-SECTION][fda4]) and those registered in PAX World.

The [PAX Live Home Page][9d50] lists all ASDF:SYSTEMs and PACKAGEs in the Lisp. For easier overview, they are grouped based on their SOURCE-LOCATIONs. Two systems are in the same group if the directory of one (i.e. the directory of the .asd file in which it was defined) is the same or is below the other's.

A PACKAGE is presented under a group of systems if the SOURCE-LOCATION of the package is below the top-most directory among the systems in the group.

Systemless Packages

The [PAX Live Home Page][9d50] lists PACKAGEs [unrelated][b033] to any ASDF:SYSTEM as systemless.

Browse by Locative Types

The [PAX Live Home Page][9d50] provides quick links to [Apropos][b7fc] result pages for all Basic Locative Types which may have definitions.

  • [glossary-term] related

    Two definitions are related if the directory of one's SOURCE-LOCATIONs contains the directory of the other's.

8.3 Markdown Support

[Markdown][a317] in docstrings and titles is processed with the 3BMD library.

8.3.1 Markdown in Docstrings

  • Docstrings can be indented in any of the usual styles. PAX normalizes indentation by stripping the longest run of leading spaces common to all non-blank lines except the first. Thus, the following two docstrings are equivalent:

      (defun foo ()
    "This is
    indented
    differently")
  
  (defun foo ()
    "This is
  indented
  differently")

  • When [Browsing Live Documentation][a595], the page displayed can be of, say, a single function within what would constitute the offline documentation of a library. Because Markdown reference link definitions, for example

      [Daring Fireball]: http://daringfireball.net/

    can be defined anywhere, they wouldn't be resolvable in that case, their use is discouraged. Currently, only reflink definitions within the documentation of the same definition are guaranteed to be resolvable. This is left intentionally vague because the specifics are subject to change.

    See [DEFINE-GLOSSARY-TERM][8ece] for a better alternative to Markdown reference links.

Docstrings of definitions that do not have a [HOME-SECTION][fda4] and are not PAX constructs themselves (e.g [SECTION][5fac], [GLOSSARY-TERM][8251], [NOTE][e2ae]) are assumed to have been written with no knowledge of PAX and to conform to Markdown only by accident. These docstrings are thus sanitized more aggressively.

  • Indentation of what looks like blocks of Lisp code is rounded up to a multiple of 4. More precisely, non-zero indented lines between blank lines or the docstring boundaries are reindented if the first non-space character of the first line is an ( or a ; character.

  • Special HTML characters <& are escaped.

  • Furthermore, to reduce the chance of inadvertently introducing a Markdown heading, if a line starts with a string of # characters, then the first one is automatically escaped. Thus, the following two docstrings are equivalent:

      The characters #\Space, #\Tab and
  #\Return are in the whitespace group.

  The characters #\Space, #\Tab and
  \#\Return are in the whitespace group.

8.3.2 Markdown in Titles

  • [glossary-term] title

    A title is a [STRING][68cc] associated with a definition (e.g. with the TITLE argument of [DEFSECTION][72b4] or [DEFINE-GLOSSARY-TERM][8ece]). Titles are accessible via [DOCTITLE][e619] and processed according to Markdown in Titles.

Titles undergo [Codification][f1ab] and may be a single paragraph containing explicit [Markdown inline code][68c1], [Markdown emphasis][c624], [Markdown image][d534]s, inline [MathJax][a17d] and HTML entities (e.g. &quot;). Other kinds of Markdown markup and block elements are not allowed.

  • [function] DOCTITLE OBJECT

    Return the title of OBJECT if it has one or NIL. For [Codification][f1ab], the title is interpreted in the package returned by DOCSTRING. DOCTITLE can be extended via [DOCTITLE*][c34e].

8.3.3 Syntax Highlighting

For syntax highlighting, GitHub's fenced code blocks Markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:

```elisp
(defun foo ())
```

to get syntactically marked up HTML output. Copy src/style.css from PAX and you are set. The language tag, elisp in this example, is optional and defaults to common-lisp.

See the documentation of 3BMD and Colorize for the details.

8.3.4 MathJax

Displaying pretty mathematics in TeX format is supported via MathJax.

  • Inline

    It can be done inline (within a paragraph):

      Pretty, eh? $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$ Yes.

    which is displayed as

    Pretty, eh? $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$ Yes.

    To avoid rendering between $5 and $6 with inline math, both the opening and the closing $ character must be followed / preceded by a non-space character. This agrees with Pandoc.

    Alternatively, the $`x_0`$ syntax may be used (renders as $x_0$), which has no restriction on spacing.

  • Block

    The $$ is supported as a block element:

      Pretty, eh?
  
  $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$
  
  Yes.

    which will be rendered in its own paragraph:

    Pretty, eh?

    $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

    Yes.

    $$ is also legal to use inline, but it's not recommended as it gets rendered inline on GitHub but as display math in PDF Output.

MathJax will leave inline code (e.g. those between single backticks) and code blocks (triple backticks) alone. Outside code, use <span>$</span> to scare MathJax off.

Escaping all those backslashes in TeX fragments embedded in Lisp strings can be a pain. [Pythonic String Reader][d3fc5] can help with that.

8.4 Codification

  • [variable] *DOCUMENT-UPPERCASE-IS-CODE* T

    When true, [interesting][7445] [name][88cf]s extracted from [codifiable][b89a] [word][d7b0]s are marked up as code with backticks. For example, this docstring

      "T PRINT CLASSes SECTION *PACKAGE* MGL-PAX ASDF
  CaMeL Capital"

    is equivalent to this:

      "`T` `PRINT` `CLASS`es `SECTION` `*PACKAGE*` `MGL-PAX` `ASDF`
  CaMel Capital"

    and renders as

    T [PRINT][d451] CLASSes [SECTION][5fac] MGL-PAX ASDF CaMel Capital

    where the links are added due to [*DOCUMENT-LINK-CODE*][d9ee].

    To suppress codification, add a backslash to the beginning of the a [codifiable][b89a] word or right after the leading * if it would otherwise be parsed as Markdown emphasis:

      "\\SECTION *\\PACKAGE*"

    The number of backslashes is doubled above because that's how the example looks in a docstring. Note that the backslash is discarded even if *DOCUMENT-UPPERCASE-IS-CODE* is false.

  • [glossary-term] codifiable

    A [word][d7b0] is codifiable if

    • it has a single uppercase character (e.g. it's T) and no lowercase characters at all, or

    • there is more than one uppercase character and no lowercase characters between them (e.g. CLASSes, unREADable, CLASS-NAMEs but not Classes or aTe).

  • [glossary-term] interesting

    A [name][88cf] is interesting if

    • it names a symbol external to its package, or

    • it is at least 3 characters long and names an interned symbol, or

    • it names a [Local Definition][9db9].

    See [Package and Readtable][ab7e].

  • [variable] *DOCUMENT-DOWNCASE-UPPERCASE-CODE* NIL

    If true, then all [Markdown inline code][68c1] (e.g. `code`, which renders as code) – including [Codification][f1ab] – which has no lowercase characters is downcased in the output. Characters of literal strings in the code may be of any case. If this variable is :ONLY-IN-MARKUP and the output format does not support markup (e.g. it's :PLAIN), then no downcasing is performed. For example,

      `(PRINT "Hello")`

    is downcased to

      `(print "Hello")`

    because it only contains uppercase characters outside the string. However,

      `MiXed "RESULTS"`

    is not altered because it has lowercase characters.

    If the first two characters are backslashes, then no downcasing is performed, in addition to [Escaping Autolinking][a270]. Use this to mark inline code that's not Lisp.

      Press `\\M-.` in Emacs.

8.5 Linking

PAX supports linking to definitions either with explicit [Reflink][cbc4]s or with [Autolink][ec7a]s.

When generating offline documentation, only the definitions in DOCUMENTABLE may be [linkable][7eb5], but when [Browsing Live Documentation][a595], everything is linkable as documentation is generated on-demand.

Many examples in this section link to standard Common Lisp definitions. In the offline case, these will link to [external URLs][f585], while in the live case to disambiguation pages that list the definition in the running Lisp and in the HyperSpec.

Invoking M-. on WORD or NAME in any of the following examples will disambiguate based on the textual context, determining the locative. This is because navigation and linking use the same Parsing algorithm, although linking is a bit more strict about trimming, depluralization, and it performs [Filtering Links][b2e4]. On the other hand, M-. cannot visit the [CLHS][ed5f] references because there are no associated source locations.

  • [glossary-term] stable printed locative

    The [Link Format][c0d2] relies on definitions having a unique textual representation that doesn't change. More concretely, if [PRIN1][6384] under WITH-STANDARD-IO-SYNTAX but with [*PRINT-READABLY*][8aca] NIL produces the same unique string deterministically, then linking to definitions works even with non-readable locatives. The uniqueness condition requires that if two definitions are different under XREF=, then their textual representations are also different.

    On the other hand, for example, a method involving an EQL specializer with an object printed with [PRINT-UNREADABLE-OBJECT][9439] :IDENTITY T does not produce a stable string and links will break.

The [Markdown reference link][8c00] syntax [label][id] is repurposed for linking to definitions. In the following, we discuss the various forms of reflinks.

Format: [ [WORD][d7b0] ][ LOCATIVE ]

The first [name][88cf] in WORD (with depluralization) that forms a valid DREF with LOCATIVE is determined, and that definition is linked to. If there is no such DREF, then an [UNRESOLVABLE-REFLINK][64be] warning is signalled.

Examples:

  • [`EQL`][type] renders as [EQL][5fd4].

  • [EQL][type] renders as [EQL][5fd4].

The Markdown link definition (i.e. [type][7c9f] above) needs no backticks to mark it as code, but here and below, the second example relies on [*DOCUMENT-UPPERCASE-IS-CODE*][f25f] being true.

Specific Reflink with Text

Format: [LINK TEXT][ [NAME][88cf] LOCATIVE ]

If NAME and LOCATIVE form a valid DREF, then that definition is linked to with link text LINK TEXT. If there is no such DREF, then an [UNRESOLVABLE-REFLINK][64be] warning is signalled.

In this form, if NAME starts with #\", then it's read as a string, else as a symbol.

Examples:

  • [see this][output-label restart] renders as [see this][65e5].

  • [see this]["MGL-PAX" package] renders as see this.

Format: [ [WORD][d7b0] ][]

The first [name][88cf] in WORD (with depluralization, symbols only) that has some DEFINITIONS is determined, and those definitions are linked to. If no [name][88cf] with any definition is found, then an [UNRESOLVABLE-REFLINK][64be] warning is signalled.

Examples:

  • single link: [PRINT][] renders as [PRINT][d451].

  • multiple links: [OUTPUT-LABEL][] renders as OUTPUT-LABEL([0][c818] [1][65e5]).

  • no definitions: [BAD-NAME][] renders as BAD-NAME.

Unspecific Reflink with Text

Format: [LINK TEXT][ [NAME][88cf] ]

The DEFINITIONS of NAME are determined, and those definitions are linked to. If NAME has no definitions, then an [UNRESOLVABLE-REFLINK][64be] warning is signalled.

Examples:

  • [see this][print] renders as [see this][d451].

  • [see this][output-label] renders as see this([0][c818] [1][65e5]).

Format: [label][id]

This is a normal [Markdown reference link][8c00] if id is not a valid locative.

  • [see this][user-defined] renders unchanged.

    (dref:dref 'user-defined 'locative)
.. debugger invoked on LOCATE-ERROR:
..   Could not locate USER-DEFINED LOCATIVE.
..   USER-DEFINED is not a valid locative type or locative alias.

    (document "[see this][user-defined]" :format :markdown)
.. [see this][user-defined]

Use URLs with [DEFINE-GLOSSARY-TERM][8ece] as a better alternative to Markdown reference links (see [Markdown in Docstrings][7bf5]).

  • [condition] UNRESOLVABLE-REFLINK [WARNING][bcb6] CONDITION-CONTEXT-MIXIN

    When DOCUMENT encounters a [Reflink][cbc4] that looks like a PAX construct but has no matching definition, it signals an UNRESOLVABLE-REFLINK warning with the [OUTPUT-REFLINK][ff9c] and OUTPUT-LABEL([0][c818] [1][65e5]) restarts available.

    If the warning is not handled, then it is printed to [*ERROR-OUTPUT*][66c6], and it behaves as if [OUTPUT-LABEL][65e5] was invoked.

  • [restart] OUTPUT-REFLINK

    In the context of an [UNRESOLVABLE-REFLINK][64be], print no warning, and leave the Markdown link unchanged. [MUFFLE-WARNING][9dac] is equivalent to OUTPUT-REFLINK.

  • [restart] OUTPUT-LABEL

    In the context of an [UNRESOLVABLE-REFLINK][64be], print no warning, and replace the Markdown link by its label. For example, [NONEXISTENT][function] becomes NONEXISTENT.

  • [function] OUTPUT-REFLINK &OPTIONAL CONDITION

    Invoke the [OUTPUT-REFLINK][ff9c] restart.

  • [function] OUTPUT-LABEL &OPTIONAL CONDITION

    Invoke the [OUTPUT-LABEL][65e5] restart.

[Markdown inline code][68c1] automatically links to the corresponding definitions without having to use [Reflink][cbc4]s. This works especially well in conjunction with [Codification][f1ab]. The following examples assume that [*DOCUMENT-UPPERCASE-IS-CODE*][f25f] is true. If that's not the case, explicit backticks are required on [WORD][d7b0] (but not on LOCATIVE).

Format: [WORD][d7b0] LOCATIVE or LOCATIVE [WORD][d7b0]

The first [name][88cf] in WORD (with depluralization) that forms a valid DREF with LOCATIVE is determined, and that definition is linked to. If no such name is found, then [Unspecific Autolink][e2a4] is attempted.

Examples:

  • PRINT function renders as [PRINT][d451] function.

  • type EQL renders as type [EQL][5fd4].

  • type EQL function renders as type [EQL][db03] function.

If LOCATIVE has spaces, then it needs to be marked up as code, too. For example,

DREF-NAME `(reader dref)`

renders as DREF-NAME (reader dref).

Format: [WORD][d7b0]

The first [name][88cf] in WORD (with depluralization, symbols only) that has some DEFINITIONS is determined, and those definitions are linked to. If no such name is found or the autolink to this name is suppressed (see below), then WORD is left unchanged. If a locative is found before or after WORD, then Specific Autolink is tried first.

Examples:

  • PRINT renders as [PRINT][d451].

  • EQL renders as EQL.

[Unspecific Autolink][e2a4]ing is suppressed if the name found has a [Local Definition][9db9] or was linked to before in the same docstring:

  • "My other CAR is also a CAR" renders as "My other [CAR][d5a2] is also a CAR".

  • "[COS][] and COS" renders as "[COS][c4a3] and COS".

  • "[EQL][type] and EQL" renders as "[EQL][5fd4] and EQL".

  • "EQ and the EQ function" renders as "[EQ][5a82] and the [EQ][5a82] function".

[Unspecific Autolink][e2a4]ing to T and NIL is also suppressed (see [*DOCUMENT-LINK-TO-HYPERSPEC*][875e]):

  • "T and NIL" renders as "T and NIL".

As an exception, a single link (be it either a Specific Link or an unambiguous [Unspecific Link][8e71]) to a [SECTION][5fac], [GLOSSARY-TERM][8251] or [NOTE][e2ae] is not suppressed to allow their titles to be displayed or their docstring to be included properly:

  • "@NAME and @NAME" renders as "[name][88cf] and [name][88cf]".

Escaping Autolinking

In the common case, when [*DOCUMENT-UPPERCASE-IS-CODE*][f25f] is true, prefixing an uppercase [word][d7b0] with a backslash prevents it from being codified and thus also prevents [Autolink][ec7a]ing from kicking in. For example,

\DOCUMENT

renders as DOCUMENT. If it should be marked up as code but not autolinked, the backslash must be within backticks like this:

`\DOCUMENT`

This renders as DOCUMENT. Alternatively, the [DISLOCATED][e391] or the [ARGUMENT][8710] locative may be used as in [DOCUMENT][dislocated].

8.5.3 Linking to the HyperSpec

  • [variable] *DOCUMENT-LINK-TO-HYPERSPEC* T

    If true, consider definitions found in the Common Lisp HyperSpec for linking. For example,

    • PRINT renders as [PRINT][d451].

    In offline documentation, this would be a link to the hyperspec unless #'PRINT in the running Lisp is DOCUMENTABLE.

    When [Browsing Live Documentation][a595], everything is [linkable][7eb5], so the generated link will go to a disambiguation page that lists the definition in the Lisp and in the HyperSpec.

    Locatives work as expected (see [*DOCUMENT-LINK-CODE*][d9ee]): FIND-IF links to [FIND-IF][5884], FUNCTION links to [FUNCTION][a51f], and [FUNCTION][type] links to FUNCTION.

    [Unspecific Autolink][e2a4]ing to T and NIL is suppressed. If desired, use [Reflink][cbc4]s such as [T][] (that links to T) or [T][constant] (that links to [T][fe21]).

    Note that linking explicitly with the [CLHS][ed5f] locative is not subject to the value of this variable, so

    • [PRINT][clhs] always renders as [PRINT][d451], and

    • PRINT clhs always renders as [PRINT][d451] clhs.

    See also the filtering of [Unspecific Link][8e71]s.

  • [variable] *DOCUMENT-HYPERSPEC-ROOT* "http://www.lispworks.com/documentation/HyperSpec/"

    A URL to the Common Lisp HyperSpec. The URL may not have a scheme, in which case file is assumed.

    The default value is the canonical location. When [invoked from Emacs][a595], the Elisp variable common-lisp-hyperspec-root is in effect.

8.5.4 Linking to Sections

The following variables control how to generate section numbering, table of contents and navigation links.

  • [variable] *DOCUMENT-LINK-SECTIONS* T

    When true, HTML anchors and PDF destinations are generated before the headings (e.g. of sections), which allows the table of contents to contain links and also code-like references to sections (like @FOO-MANUAL) to be translated to links with the [TITLE][72b4] being the link text.

  • [variable] *DOCUMENT-MAX-NUMBERING-LEVEL* 3

    A non-negative integer. In their hierarchy, sections on levels less than this value get numbered in the format of 3.1.2. Setting it to 0 turns numbering off.

  • [variable] *DOCUMENT-MAX-TABLE-OF-CONTENTS-LEVEL* 3

    An integer that determines the depth of the table of contents.

    • If negative, then no table of contents is generated.

    • If non-negative, and there are multiple top-level sections on a page, then they are listed at the top of the page.

    • If positive, then for each top-level section a table of contents is printed after its heading, which includes a nested tree of section titles whose depth is limited by this value.

    If *DOCUMENT-LINK-SECTIONS* is true, then the tables will link to the sections.

  • [variable] *DOCUMENT-TEXT-NAVIGATION* NIL

    If true, then before each heading a line is printed with links to the previous, parent and next section. Needs *DOCUMENT-LINK-SECTIONS* to be on to work.

  • [variable] *DOCUMENT-FANCY-HTML-NAVIGATION* T

    If true and the output format is HTML, then headings get a navigation component that consists of links to the previous, parent, next section, a self-link, and a link to the definition in the source code if available (see :SOURCE-URI-FN in DOCUMENT). This component is normally hidden, it is visible only when the mouse is over the heading. Has no effect if *DOCUMENT-LINK-SECTIONS* is false.

  • [variable] *DOCUMENT-LINK-CODE* T

    Whether definitions of things other than [SECTION][5fac]s are allowed to be [linkable][7eb5].

  • [glossary-term] linkable

    When a reference is encountered to definition D while processing documentation for some page C, we say that definition D is linkable (from C) if

    • D denotes a [SECTION][5fac] and *DOCUMENT-LINK-SECTIONS* is true, or

    • D does not denote a SECTION and [*DOCUMENT-LINK-CODE*][d9ee] is true

    ... and

    • We are [Browsing Live Documentation][a595], or

    • D is an external definition ([CLHS][ed5f] or denotes a [GLOSSARY-TERM][8251] with a [URL][8ece]), or

    • D's page is C, or

    • D's page is relativizable to C.

    In the above, D's page is the last of the pages in the DOCUMENTABLE to which D's documentation is written (see :OBJECTS in [PAGES][9c7d]), and we say that a page is relativizable to another if it is possible to construct a relative link between their :URI-FRAGMENTs.

Specific links are those [Reflink][cbc4]s and [Autolink][ec7a]s that have a single locative and therefore at most a single matching definition. These are [Specific Reflink][f7a5], [Specific Reflink with Text][fb17] and Specific Autolink.

A specific link to a [linkable][7eb5] definition produces a link in the output. If the definition is not linkable, then the output will contain only what would otherwise be the link text.

Unspecific links are those [Reflink][cbc4]s and [Autolink][ec7a]s that do not specify a locative and match all definitions with a name. These are Unspecific Reflink, Unspecific Reflink with Text and [Unspecific Autolink][e2a4].

To make the links predictable and manageable in number, the following steps are taken.

  1. Definitions that are not symbol-based (i.e. whose DREF-NAME is not a symbol) are filtered out to prevent unrelated PACKAGEs, ASDF:SYSTEMs and [CLHS][ed5f] sections from cluttering the documentation without the control provided by importing symbols.

  2. Links to the CLHS are filtered out when the corresponding non-CLHS definition is being documented (and thus among the links).

  3. All references with LOCATIVE-TYPE LOCATIVE are filtered out.

  4. Non-[linkable][7eb5] definitions are removed.

  5. If the definitions include a GENERIC-FUNCTION, then all definitions with LOCATIVE-TYPE METHOD, ACCESSOR, READER and WRITER are removed to avoid linking to a possibly large number of methods.

If at most a single definition remains, then the output is the same as with a Specific Link.

If multiple definitions remain, then the link text is output followed by a number of numbered links, one to each definition. When [Browsing Live Documentation][a595], ambiguities can be much more frequent; and thus a single disambiguation page is linked to instead (but see [*BROWSE-CONTEXT*][805d]).

The following variables control various aspects of links and URLs.

  • [variable] *DOCUMENT-URL-VERSIONS* (2 1)

    A list of versions of PAX URL formats to support in the generated documentation. The first in the list is used to generate links.

    PAX emits HTML anchors before the documentation of [SECTION][5fac]s (see Linking to Sections) and other things (see Linking). For the function FOO, in the current version (version 2), the anchor is <a id="MGL-PAX:FOO%20FUNCTION">, and its URL will end with #MGL-PAX:FOO%20FUNCTION.

    Note: To make the URL independent of whether a symbol is internal or external to their [SYMBOL-PACKAGE][e5ab], single colon is printed where a double colon would be expected. Package and symbol names are both printed verbatim except for escaping colons and spaces with a backslash. For exported symbols with no funny characters, this coincides with how [PRIN1][6384] would print the symbol, while having the benefit of making the URL independent of the Lisp printer's escaping strategy and producing human-readable output for mixed-case symbols. No such promises are made for non-ASCII characters, and their URLs may change in future versions. Locatives are printed with PRIN1.

    Version 1 is based on the more strict HTML4 standard and the id of FOO is "x-28MGL-PAX-3A-3AFOO-20FUNCTION-29". This is supported by GitHub-flavoured Markdown. Version 2 has minimal clutter and is obviously preferred. However, in order not to break external links, by default, both anchors are generated.

    Let's understand the generated Markdown.

    (defun foo (x))

(document #'foo :format :markdown)
=> ("<a id=\"x-28MGL-PAX-3AFOO-20FUNCTION-29\"></a>
<a id=\"MGL-PAX:FOO%20FUNCTION\"></a>

- [function] **FOO** *X*
")

(let ((*document-url-versions* '(1)))
  (document #'foo :format :markdown))
=> ("<a id=\"x-28MGL-PAX-3AFOO-20FUNCTION-29\"></a>

- [function] **FOO** *X*
")

  • [variable] *DOCUMENT-MIN-LINK-HASH-LENGTH* 4

    Recall that [Markdown reference link][8c00]s (like [label][id]) are used for Linking. It is desirable to have ids that are short to maintain legibility of the generated Markdown, but also stable to reduce the spurious diffs in the generated documentation, which can be a pain in a version control system.

    Clearly, there is a tradeoff here. This variable controls how many characters of the MD5 sum of the full link id (the reference as a string) are retained. If collisions are found due to the low number of characters, then the length of the hash of the colliding reference is increased.

    This variable has no effect on the HTML generated from Markdown, but it can make Markdown output more readable.

  • [variable] *DOCUMENT-BASE-URL* NIL

    When *DOCUMENT-BASE-URL* is non-NIL, this is prepended to all Markdown relative URLs. It must be a valid URL without query or fragment parts. For example,

      http://lisp.org/doc/

    but not either of the following:

      http://lisp.org/doc?a=1
  http://lisp.org/doc#fragment.

    Note that intra-page links using only URL fragments (e.g. and explicit HTML links (e.g. <a href="...">) in Markdown are not affected.

8.6 Local Definition

While documentation is generated for a definition, that definition is considered local. Other local definitions may also be established. Local definitions inform [Codification][f1ab] through [interesting][7445] [name][88cf]s and affect [Unspecific Autolink][e2a4]ing.

(defun foo (x)
  "FOO adds one to X."
  (1+ x)

In this example, while the docstring of FOO is being processed, the global definition (DREF 'FOO 'FUNCTION) is also considered local, which suppresses linking FOO in the FOO's docstring back to its definition. If FOO has other definitions, [Unspecific Autolink][e2a4]ing to those is also suppressed.

Furthermore, the purely local definition (DREF 'X 'ARGUMENT) is established, causing the argument name X to be [codified][f1ab] because X is now [interesting][7445].

See [DOCUMENTING-DEFINITION][a412] and WITH-DISLOCATED-NAMES in [Extending DOCUMENT][574a].

8.7 Overview of Escaping

Let's recap how escaping [Codification][f1ab], [downcasing][a5ee], and Linking works.

  • One backslash in front of a [word][d7b0] turns codification off. Use this to prevent codification words such as DOCUMENT, which is all uppercase hence [codifiable][b89a], and it names an exported symbol hence it is [interesting][7445].

  • One backslash right after an opening backtick turns autolinking off.

  • Two backslashes right after an opening backtick turns autolinking and downcasing off. Use this for things that are not Lisp code but which need to be in a monospace font.

In the following examples, capital C/D/A letters mark the presence, and a/b/c the absence of codification, downcasing, and autolinking assuming all these features are enabled by [*DOCUMENT-UPPERCASE-IS-CODE*][f25f], [*DOCUMENT-DOWNCASE-UPPERCASE-CODE*][a5ee], and [*DOCUMENT-LINK-CODE*][d9ee].

DOCUMENT                => [`document`][1234]    (CDA)
\DOCUMENT               => DOCUMENT              (cda)
`\DOCUMENT`             => `document`            (CDa)
`\\DOCUMENT`            => `DOCUMENT`            (Cda)
[DOCUMENT][]            => [`document`][1234]    (CDA)
[\DOCUMENT][]           => [DOCUMENT][1234]      (cdA)
[`\DOCUMENT`][]         => [`document`][1234]    (CDA) *
[`\\DOCUMENT`][]        => [`DOCUMENT`][1234]    (CdA)
[DOCUMENT][dislocated]  => `document`            (CDa)

Note that in the example marked with *, the single backslash, that would normally turn autolinking off is ignored because it is in an explicit link.

8.8 Indexing

PAX can create indices that cross-reference definitions and [concept key][b19d]s with where they are mentioned in the documentation.

  • [glossary-term] referrer

    If the documentation of a definition links to another definition, the former is considered a referrer to the latter. See [referent][ad8e].

  • [glossary-term] referent

    A definition is considered a referent of another definition if the latter's documentation links to it. See [referrer][e7e0].

8.8.1 Indexing Definitions

PAX generates indices with entries like this:

foo [my-lib] (fn)
  ↩ d: My Lib Utilities
  ↩ f: bar, baz

MY-LIB:FOO here is a function whose documentation has been linked to by the My Lib Utilities [SECTION][5fac] as well as the BAR and BAZ functions (grouped together under f). For example, BAR could be defined like this:

(defun bar (x)
  "Extends FOO."
  (1+ (foo x)))

In this example, MY-LIB:FOO is the [referent][ad8e] and the others are the [referrer][e7e0]s. If there are no referrers or they are in the same group, then a single line is printed. Here is an example for the latter:

foo [my-lib] (fn) ↩ f: bar, baz

8.8.2 Indexing Concepts

Let's first [DEFINE-CONCEPT][b80d]s:

(define-concept @albert-einstein (:title "Albert Einstein"
                                  :keys ("Einstein, Albert")))

(define-concept @einstein-summation (:title "Einstein summation"
                                     :keys ("Einstein summation")))

CONCEPTs are definitions, but in the generated documentation, links to them are [hidden][56b9], and they are not meant to be documented themselves. They exist to endow their [referrer][e7e0]s with [concept key][b19d]s:

(defun einsum (x)
  "Implement @EINSTEIN-SUMMATION by @ALBERT-EINSTEIN."
  x)

This makes EINSUM a [referrer][e7e0] to the concept @EINSTEIN-SUMMATION, and thus will be indexed under the concept keys of @EINSTEIN-SUMMATION.

[DEFSECTION][72b4] and [DEFINE-GLOSSARY-TERM][8ece] can associate concepts with the section or glossary term being defined:

(defsection @famous-people (:title "Famous People"
                            :concepts (@albert-einstein))
  "Let's start with one of the most famous, Albert Einstein.")

This is like the mention in EINSUM, but it also makes the section a primary source of information. These are typeset more prominently in the generated documentation. If the section only mentions Einstein in passing, then don't use the :CONCEPTS argument, just mention @ALBERT-EINSTEIN in a section docstring as EINSUM does.

Now, let's define a section to hold them together, configure a minimal index, and see what comes out:

(defsection @test (:title "Test" :export nil)
  (@famous-people section)
  (einsum function))

(defun print-concept-index (documentable)
  (let* ((*document-index-formats* '(:plain))
         (*document-indices* '((:concepts t :title "Concept Index")))
         (output (document documentable :stream nil)))
    (princ (subseq output (search "## Concept Index" output)))))

(print-concept-index @test)
.. ## Concept Index
..
.. - Einstein summation ↩ f: EINSUM
..
.. - Einstein, Albert
..
..     - ↩ d: Famous People
..
..     - ↩ f: EINSUM
..

This is okay, but let's group summation and Albert under Einstein by breaking the string keys into lists of strings:

(define-concept @albert-einstein (:title "Albert Einstein"
                                  :keys (("Einstein" "Albert"))))

(define-concept @einstein-summation (:title "Einstein summation"
                                     :keys (("Einstein" "summation"))))

(print-concept-index @test)
.. ## Concept Index
..
.. - Einstein
..
..     - Albert
..
..         - ↩ d: Famous People
..
..         - ↩ f: EINSUM
..
..     - summation ↩ f: EINSUM
..

See [DEFINE-CONCEPT][b80d] about pure multiplexer concepts (those without :TITLEs) and how concepts can inherit the keys of other concepts.

Also, see [concept subkey][5920] on how to order concepts in the output.

  • [macro] DEFINE-CONCEPT NAME (&KEY TITLE KEYS)

    Define a CONCEPT for [Indexing Concepts][c001]. TITLE is a [STRING][68cc] or NIL, while KEYS is a list of [concept key][b19d]s and [SYMBOL][e5af]s (naming other concepts, see below).

    When Generating Documentation, if a link would be made to a CONCEPT, then:

    • The concept's :KEYS are recorded as [referent][ad8e]s with the definition being documented as the [referrer][e7e0].

      SYMBOLs in :KEYS are resolved to the concepts they name, and they are substituted with their keys recursively (circularities are not allowed, but there are no checks for them).

    • In the docstring being processed, the text denoting the name of the concept is replaced by the concept's :TITLE or the empty string if the title is NIL. In an ambiguous [Unspecific Link][8e71], the links to concepts are simply removed.

    By convention, names of TITLEd concepts start with a #@ character, to indicate that these names are part of the sentence flow, as their titles are substituted for their mentions. Names of unTITLEd concepts start with a #~ character to indicate that they are pure index key multiplexers, and do not appear in the generated documentation.

    See also [DEFINE-GLOSSARY-TERM][8ece].

  • [glossary-term] concept key

    A concept key identifies an entry in an index. It is a [concept subkey][5920] or a list of [concept subkey][5920]s. Concept keys occur in [DEFINE-CONCEPT][b80d], [DEFSECTION][72b4] and [DEFINE-GLOSSARY-TERM][8ece].

    • Single subkey: For example, the key "Einstein, Albert" consists of a single subkey. A single subkey is equivalent to a one-element list.

    • List of subkeys: When Einstein summation is also referenced, we can instead use the lists ("Einstein," "Albert") and ("Einstein," "summation") to ensure that their [referrer][e7e0]s are grouped together under "Einstein,".

      List concept keys can be arbitrarily long, although the PDF Output imposes a limit of 9.

  • [glossary-term] concept subkey

    A concept subkey (a part of a [concept key][b19d]) can be a string or a cons of strings. In the latter form, the [CAR][d5a2] is for printing, and the [CDR][e012] is for sorting. For example, use ("Éclair" . "Eclair") to have "Éclair" printed but sorted as if it were "Eclair". Sorting is done with plain STRING<, which is not Unicode-aware.

    Note that print names are ESCAPE-MARKDOWNed.

8.8.3 Configuring Indices

  • [variable] *DOCUMENT-INDEX-SECTIONS* :HOMELESS-DOCUMENTABLE

    Controls what sections can get an index (if [*DOCUMENT-INDEX-FORMATS*][ba76] also allows it and they are non-empty).

    • NIL: No indices are generated.

    • :DOCUMENTABLE: Sections that appear in DOCUMENTABLE (at any level) can get indices. Their subsections cannot.

    • :HOMELESS-DOCUMENTABLE: Sections that appear in DOCUMENTABLE and have no [HOME-SECTION][fda4] can get indices.

  • [variable] *DOCUMENT-INDEX-FORMATS* (:HTML :PDF)

    The list of [Output Formats][8d9b] for which index generation is allowed.

  • [variable] *DOCUMENT-INDICES* "<value omitted>"

    A nested list of index specifications. Each spec is of the form

      (&key title documentation document-referrer-groups
        dtype concepts children)

    Non-CONCEPT [referent][ad8e]s recorded during documentation generation are assigned to the first (in depth-first order) spec with a matching DTYPE. For [referent][ad8e]s that are concepts, their keys are assigned to the first spec with :CONCEPTS true.

    Specs are processed in the following way and order:

    • If a spec and all its :CHILDREN are empty (nothing was assigned to them above), then they are not output.

    • Else, a dynamically generated [SECTION][5fac] with TITLE (a Markdown string) is entered.

    • If DOCUMENTATION is non-NIL, then it is a Markdown docstring and is written as is to the output.

    • If DOCUMENT-REFERRER-GROUPS is true, then the ABBREVs and DOCUMENTATIONs in [*DOCUMENT-INDEX-REFERRER-GROUPS*][85c9] are listed in the output.

    • The [referent][ad8e]s assigned to this spec are output along with their [referrer][e7e0]s (see the examples in [Indexing Concepts][c001]).

    • The CHILDREN specs are processed.

    • The dynamically generated section is closed.

    The default value specifies separate indices for

    • functions and macros

    • variables

    • types

    • other things

    • CONCEPTs and [GLOSSARY-TERM][8251]s.

    They are all grouped under an "Indices" section, that has :DOCUMENT-REFERRER-GROUPS.

  • [variable] *DOCUMENT-INDEX-REFERENT-LOCATIVE-TYPE-ABBREVS* ((GENERIC-FUNCTION "_(gf)_") #'"_(fn)_" (VARIABLE "_(var)_") (ASDF/SYSTEM:SYSTEM "_(asdf:system)_"))

    A list of (LOCATIVE-TYPE ABBREV) elements, where ABBREV (a Markdown string) is printed instead of the [referent][ad8e]'s locative when its DREF-LOCATIVE-TYPE is LOCATIVE-TYPE.

    The default behaviour is to bind [*PACKAGE*][5ed1] to the package of the symbol naming the definition (if it's a symbol) and print the locative with *PRINT-CASE* :DOWNCASE, omitting the package of the locative type.

  • [variable] *DOCUMENT-INDEX-REFERRER-GROUPS* "<value omitted>"

    A list of (DTYPE ABBREV &KEY DOCUMENTATION) elements that defines the grouping of [referrer][e7e0]s of similar types in referrer lists. For example, the default value of this variable has this element:

      ((or function macro compiler-macro method) "↩ _f_:"
     :documentation "_f_: for definitions in the function namespace ...")

    Thus, in the following example, the definitions of BAR and BAZ are grouped together under ↩ f: because they are both DTYPEP (OR FUNCTION MACRO COMPILER-MACRO METHOD):

      foo [my-lib] (fn)
    ↩ d: My Lib Utilities
    ↩ f: bar, baz

    • In general, the first matching DTYPE wins. See DTYPEs.

    • ABBREV may be a Markdown string or a cons of strings to control sort order in the manner of [concept subkey][5920]s.

    • DOCUMENTATION is a Markdown string for :DOCUMENT-REFERRER-GROUPS in [*DOCUMENT-INDICES*][8894].

8.9 Output Formats

  • [variable] *DOCUMENT-MARK-UP-SIGNATURES* T

    When true, some things such as function names and arglists are rendered as bold and italic. In HTML Output and PDF Output, locative types become links to sources (if :SOURCE-URI-FN is provided, see [PAGES][9c7d]), and the symbol becomes a self-link for your permalinking pleasure.

    For example, a reference is rendered in Markdown roughly as:

      - [function] foo x y

    With this option on, the above becomes:

      - [function] **foo** *x y*

    Also, in HTML **foo** will be a link to that very entry and [function] may turn into a link to sources.

8.9.1 Plain Output

This is the default :FORMAT of DOCUMENT, intended to be a replacement for [CL:DOCUMENTATION][c5ae]. :PLAIN (short for plain text) is very similar to :MARKDOWN, but most of the markup that would make reading in, say, the REPL unpleasant is removed.

  • Markup for [Markdown emphasis][c624], [Markdown inline code][68c1], [Markdown reference link][8c00]s and fenced code blocks is stripped from the output.

  • No link anchors are emitted.

  • No [section numbering][f12d].

  • No [table of contents][d4e7].

8.9.2 Markdown Output

By default, DREFs are documented in the following format.

- [<locative-type>] <name> <arglist>

    <docstring>

The line with the bullet is printed with [DOCUMENTING-DEFINITION][a412]. The docstring is processed with [DOCUMENT-DOCSTRING][7f1f] while [Local Definition][9db9]s established with WITH-DISLOCATED-NAMES are in effect for all variables locally bound in a definition with ARGLIST, and [*PACKAGE*][5ed1] is bound to the second return value of DOCSTRING.

With this default format, PAX supports all locative types, but for some Basic Locative Types defined in DRef and the PAX Locatives, special provisions have been made.

  • For definitions with a VARIABLE or CONSTANT locative, their initform is printed as their arglist. The initform is the INITFORM argument of the locative if provided, or the global symbol value of their name. If no INITFORM is provided, and the symbol is globally unbound, then no arglist is printed.

When the printed initform is too long, it is truncated.

  • Depending on what the SETF locative refers to, the ARGLIST of the setf expander, [setf function][99b0], or the method signature is printed as with the METHOD locative.

  • For definitions with a METHOD locative, the arglist printed is the method signature, which consists of the locative's QUALIFIERS and SPECIALIZERS appended.

  • For definitions with an ACCESSOR, READER or WRITER locative, the class on which they are specialized is printed as their arglist.

  • For definitions with a STRUCTURE-ACCESSOR locative, the arglist printed is the locative's CLASS-NAME argument if provided.

  • For definitions with a CLASS locative, the arglist printed is the list of [public superclasses][ff58] with [STANDARD-OBJECT][a843] and [CONDITION][83e1] omitted.

  • For definitions with a STRUCTURE locative, the arglist printed is the list of [public superclasses][ff58] with STRUCTURE-OBJECT omitted.

  • For definitions with a CONDITION locative, the arglist printed is the list of [public superclasses][ff58] with STANDARD-OBJECT and CONDITION omitted.

  • For definitions with a ASDF:SYSTEM locative, their most important slots are printed as an unnumbered list.

  • For definitions with the LOCATIVE locative type, their LOCATIVE-TYPE-DIRECT-SUPERS and LOCATIVE-TYPE-DIRECT-SUBS are printed.

  • When documentation is being generated for a definition with the [SECTION][672f] locative, a new (sub)section is opened (see [WITH-HEADING][80e8]), within which documentation for its [SECTION-ENTRIES][d850] is generated, separated from each other by [ENSURE-MD-PARAGRAPH][a021].

  • For definitions with a GLOSSARY-TERM locative, no arglist is printed, and if non-NIL, [GLOSSARY-TERM-TITLE][af78] is printed as name.

  • For definitions with a [CONCEPT][56b9] locative, no documentation is generated.

  • For definitions with a [GO][58f6] locative, its LOCATIVE-ARGS are printed as its arglist, along with a redirection message.

  • See the [INCLUDE][5cd7] locative.

  • For definitions with a [CLHS][ed5f] locative, the LOCATIVE-ARGS are printed as the arglist. For CLHS [SECTION][7a4e]s, the title is included in the arglist.

  • For definitions with an UNKNOWN locative, the LOCATIVE-ARGS are printed as the arglist. There is no docstring.

  • [glossary-term] public superclasses

    The public superclasses of a class are the tightest envelope of superclasses with names exported from some package. This envelope is constructed by recursing depth-first into the superclass hierarchy. If the name of the superclass currently processed is exported from any package, then it is collected as a public superclass, and we do not recurse into its superclasses.

8.9.3 PDF Output

When invoked with :FORMAT :PDF, DOCUMENT generates [Markdown Output][dd29] and converts it to PDF with [Pandoc][59d9], which in turn uses LaTeX. Make sure that they are installed.

(with-open-file (s "x.pdf" :direction :output :if-exists :supersede
                           :if-does-not-exist :create
                           :element-type '(unsigned-byte 8))
  (pax:document "Hello, World!" :stream s :format :pdf))

To see how the output looks like, visit pax-manual-v0.4.1.pdf and dref-manual-v0.4.1.pdf.

PDF output is similar to [*DOCUMENT-HTML-DEFAULT-STYLE*][90fa] :CHARTER without the off-white tint and with coloured instead of underlined links. The latter is because underlining interferes with hyphenation in LaTeX. As in HTML output, locative types link to the respective definition in the sources on GitHub (see [MAKE-GIT-SOURCE-URI-FN][587f]).

Note that linking from one PDF to another is currently not supported due to the lack of consistent support in PDF viewers. Therefore, such links are replaced by their label or the title if any (e.g. of a [SECTION][5fac] or [GLOSSARY-TERM][8251]).

The generation of Markdown is subject to the standard variables (again see DOCUMENT). The Markdown to PDF conversion can be customized with the following variables.

  • [variable] *DOCUMENT-PANDOC-PROGRAM* "pandoc"

    The name of the Pandoc binary. It need not be an absolute pathname as PATH is searched.

  • [variable] *DOCUMENT-PANDOC-PDF-OPTIONS* (("-V" "papersize=a4") ("-V" "margin-left=1.03in") ("-V" "margin-right=1.03in") ("-V" "margin-top=1.435in") ("-V" "margin-bottom=1.435in") ("-V" "fontfamily=XCharter") ("-V" "fontsize=11pt") ("-V" "colorlinks=true") ("-V" "linkcolor=blue") ("-V" "urlcolor=Maroon") ("-V" "toccolor=blue") "--verbose")

    The command-line options to invoke [*DOCUMENT-PANDOC-PROGRAM*][dd37] with. For ease of manipulation, related options are grouped into sublists, but the entire nested list is flattened to get the list of options to pass to Pandoc. If --verbose is specified, then in addition to Pandoc logging LaTeX sources, PAX will log to [*ERROR-OUTPUT*][66c6] the Markdown that it converts to PDF via LaTeX. The Markdown includes *DOCUMENT-PANDOC-PDF-HEADER-INCLUDES* and [*DOCUMENT-PANDOC-PDF-METADATA-BLOCK*][ee8b].

  • [variable] *DOCUMENT-PANDOC-PDF-HEADER-INCLUDES* "<too messy to include>"

    LaTeX code (a string) to include in the preamble via header-includes.

    The default includes have no configuration knobs. Look at the value to see how to customize it.

  • [variable] *DOCUMENT-PANDOC-PDF-METADATA-BLOCK* ""

    A [Pandoc YAML metadata block][84f2] as a string.

    Concatenate to this string to customize it.

8.9.4 Dummy Output

When the FORMAT argument of DOCUMENT is NIL, no output is generated, but Transcript Consistency Checking is still performed. Use this feature to quickly test documentation examples.

For example, in Try the test would look like this:

(try:signals-not (transcription-consistency-error)
  (document ... :format nil))

8.10 Documentation Generation Implementation Notes

Documentation Generation is supported on ABCL, AllegroCL, CLISP, CCL, CMUCL, ECL and SBCL, but their outputs may differ due to the lack of some introspective capability. SBCL generates complete output. see ARGLIST, DOCSTRING and SOURCE-LOCATION for implementation notes.

In addition, CLISP does not support the ambiguous case of [Browsing Live Documentation][a595] because the current implementation relies on Swank to list definitions of symbols (as VARIABLE, FUNCTION, etc), and that simply doesn't work.

8.11 Utilities for Generating Documentation

Two convenience functions are provided to serve the common case of having an ASDF system with some readmes and a directory with the HTML documentation and the default CSS stylesheet.

  • [function] UPDATE-ASDF-SYSTEM-READMES OBJECT ASDF-SYSTEM &KEY (URL-VERSIONS '(1)) (FORMATS '(:MARKDOWN))

    Convenience function to generate up to two readme files in the directory holding the ASDF-SYSTEM definition. OBJECT is passed on to DOCUMENT.

    If :MARKDOWN is in FORMATS, then README.md is generated with anchors, links, inline code, and other markup added. Not necessarily the easiest on the eye in an editor but looks good on GitHub.

    If :PLAIN is in FORMATS, then README is generated, which is optimized for reading in text format. It has less cluttery markup and no [Autolink][ec7a]ing.

    Example usage:

    (update-asdf-system-readmes @pax-manual :mgl-pax
                            :formats '(:markdown :plain))

    Note that *DOCUMENT-URL-VERSIONS* is bound to URL-VERSIONS, which defaults to using the uglier, version 1 style of URL for the sake of GitHub.

8.11.1 HTML Output

  • [function] UPDATE-ASDF-SYSTEM-HTML-DOCS SECTIONS ASDF-SYSTEM &KEY PAGES (TARGET-DIR (ASDF/SYSTEM:SYSTEM-RELATIVE-PATHNAME ASDF-SYSTEM "doc/")) (UPDATE-CSS-P T) (STYLE *DOCUMENT-HTML-DEFAULT-STYLE*)

    Generate pretty HTML documentation for a single ASDF system, possibly linking to GitHub. If UPDATE-CSS-P, copy the STYLE files to TARGET-DIR (see [*DOCUMENT-HTML-DEFAULT-STYLE*][90fa]).

    Example usage:

    (update-asdf-system-html-docs @pax-manual :mgl-pax)

    The same, linking to the sources on GitHub:

    (update-asdf-system-html-docs
  @pax-manual :mgl-pax
  :pages
  `((:objects (,mgl-pax::@pax-manual)
     :source-uri-fn ,(make-git-source-uri-fn
                      :mgl-pax
                      "https://github.com/melisgl/mgl-pax"))))

See the following variables, which control HTML generation.

  • [variable] *DOCUMENT-HTML-DEFAULT-STYLE* :DEFAULT

    The HTML style to use. It's either :DEFAULT or :CHARTER. The :DEFAULT CSS stylesheet relies on the default fonts (sans-serif, serif, monospace), while :CHARTER bundles some fonts for a more controlled look.

    The value of this variable affects the default style of [UPDATE-ASDF-SYSTEM-HTML-DOCS][bb12].

  • [variable] *DOCUMENT-HTML-MAX-NAVIGATION-TABLE-OF-CONTENTS-LEVEL* NIL

    NIL or a non-negative integer. If non-NIL, it overrides [*DOCUMENT-MAX-NUMBERING-LEVEL*][f12d] in the dynamic HTML table of contents on the left of the page.

  • [variable] *DOCUMENT-HTML-LANG* "en"

    The value for the html element's xml:lang and lang attributes in the generated HTML.

  • [variable] *DOCUMENT-HTML-CHARSET* "UTF-8"

    The value for charset attribute of the <meta http-equiv='Content-Type' content='text/html'> element in the generated HTML.

  • [variable] *DOCUMENT-HTML-HEAD* NIL

    Stuff to be included in the <head> of the generated HTML.

    • If NIL, nothing is included.

    • If a [STRING][68cc], then it is written to the HTML output as is without any escaping.

    • If a function designator, then it is called with a single argument, the HTML stream, where it must write the output.

  • [variable] *DOCUMENT-HTML-SIDEBAR* NIL

    Stuff to be included in the HTML sidebar.

    • If NIL, a default sidebar is generated, with [*DOCUMENT-HTML-TOP-BLOCKS-OF-LINKS*][e216], followed by the dynamic table of contents, and *DOCUMENT-HTML-BOTTOM-BLOCKS-OF-LINKS*.

    • If a [STRING][68cc], then it is written to the HTML output as is without any escaping.

    • If a function designator, then it is called with a single argument, the HTML stream, where it must write the output.

  • [variable] *DOCUMENT-HTML-TOP-BLOCKS-OF-LINKS* NIL

    A list of blocks of links to be displayed on the sidebar on the left, above the table of contents. A block is of the form (&KEY TITLE ID LINKS), where TITLE will be displayed at the top of the block in a HTML DIV with ID followed by the links. LINKS is a list of (URI LABEL) elements, where URI may be a string or an object being DOCUMENTed or a REFERENCE thereof.

  • [variable] *DOCUMENT-HTML-BOTTOM-BLOCKS-OF-LINKS* NIL

    Like [*DOCUMENT-HTML-TOP-BLOCKS-OF-LINKS*][e216], only it is displayed below the table of contents.

8.11.2 GitHub Workflow

It is generally recommended to commit generated readmes (see UPDATE-ASDF-SYSTEM-READMES) so that users have something to read without reading the code and sites like GitHub can display them.

Also, by registering in PAX World, you get documentation generated in various formats periodically. If you want to manage this yourself, there are a couple of things to keep in mind.

First, HTML documentation can also be committed, but there is an issue with that: when linking to the sources (see [MAKE-GIT-SOURCE-URI-FN][587f]), the commit id is in the link. This means that code changes need to be committed first, and only then can HTML documentation be regenerated and committed in a followup commit.

The second issue is that GitHub is not very good at serving HTML files from the repository itself (and http://htmlpreview.github.io chokes on links to the sources).

The recommended workflow is to use gh-pages, which can be made relatively painless with the git worktree command. The gist of it is to make the doc/ directory a checkout of the branch named gh-pages. There is a good description of this general process. Two commits are needed still, but it is somewhat less painful.

This way the HTML documentation will be available at

http://<username>.github.io/<repo-name>

It is probably a good idea to add sections like the [Links and Systems][ba90] section to allow jumping between the repository and the gh-pages site.

  • [function] MAKE-GITHUB-SOURCE-URI-FN ASDF-SYSTEM GITHUB-URI &KEY GIT-VERSION

    This function is a backward-compatibility wrapper around [MAKE-GIT-SOURCE-URI-FN][587f], which supersedes MAKE-GITHUB-SOURCE-URI-FN. All arguments are passed on to MAKE-GIT-SOURCE-URI-FN, leaving URI-FORMAT-STRING at its default, which is suitable for GitHub.

  • [function] MAKE-GIT-SOURCE-URI-FN ASDF-SYSTEM GIT-FORGE-URI &KEY GIT-VERSION GIT-ROOT (URI-FORMAT-STRING "~A/blob/~A/A#LS")

    Return an object suitable as :SOURCE-URI-FN of a page spec (see the PAGES argument of DOCUMENT). The function looks at the source location of the object passed to it, and if the location is found, the path is made relative to the GIT-ROOT (the top-level directory of the git checkout), and finally an URI pointing to your git forge (such as GitHub) is returned.

    • If ASDF-SYSTEM is non-NIL, then GIT-ROOT defaults to the directory of the .asd file of the system. Similarly, GIT-VERSION defaults current commit id in the checkout at GIT-ROOT.

    • If both GIT-ROOT and GIT-VERSION are explicitly provided, ASDF-SYSTEM has no effect.

    If GIT-ROOT is not specified and cannot be determined from ASDF-SYSTEM, then no links to the git forge will be generated.

    A warning is signalled whenever the source location lookup fails or if the source location points to a directory not below GIT-ROOT.

    If GIT-FORGE-URI is "https://github.com/melisgl/mgl-pax/" and GIT-VERSION is "master", then the returned URI may look like this:

      https://github.com/melisgl/mgl-pax/blob/master/src/pax-early.lisp#L12

    URI-FORMAT-STRING is a [CL:FORMAT][ad78] control string for four arguments:

    • GIT-FORGE-URI,

    • GIT-VERSION,

    • the relative path to the file of the source location of the reference,

    • and the line number.

    The default value of URI-FORMAT-STRING is for GitHub. If using a non-standard git forge, such as Sourcehut or GitLab, simply pass a suitable URI-FORMAT-STRING matching the URI scheme of your forge.

8.11.3 PAX World

PAX World is a registry of documents, which can generate cross-linked HTML documentation pages for all the registered documents. There is an official PAX World at https://fixnum.com/, which is updated every few hours. To have your stuff there, [REGISTER-DOC-IN-PAX-WORLD][f4fd] in your ASDF system and file an issue at https://github.com/melisgl/mgl-pax-world/issues to make PAX World actually load it.

  • [function] REGISTER-DOC-IN-PAX-WORLD ID SECTIONS PAGE-SPECS

    Register SECTIONS and PAGE-SPECS under ID (an arbitrary symbol) in PAX World. By default, [UPDATE-PAX-WORLD][ee51] generates documentation for all of these. SECTIONS and PAGE-SPECS must be lists of [SECTION][5fac]s and PAGE-SPECs (see [PAGES][9c7d]) or designators of functions with no arguments that return such lists.

    Using an existing ID overwrites the previous registration.

For example, this is how PAX registers itself:

(defun pax-sections ()
  (list @pax-manual))

(defun pax-pages ()
  `((:objects ,(pax-sections)
     :source-uri-fn ,(make-git-source-uri-fn
                      "mgl-pax" "https://github.com/melisgl/mgl-pax"))))

(register-doc-in-pax-world :pax 'pax-sections 'pax-pages)

  • [function] UPDATE-PAX-WORLD &KEY (DOCS *REGISTERED-PAX-WORLD-DOCS*) (FORMATS '(:HTML)) DIR UPDATE-CSS-P (STYLE *DOCUMENT-HTML-DEFAULT-STYLE*)

    Generate HTML documentation for all DOCS. Files are created in DIR ((asdf:system-relative-pathname :mgl-pax "world/") by default if DIR is NIL). DOCS is a list of entries of the form (NAME [SECTIONS][5fac] PAGE-SPECS). The default for DOCS is all the sections and pages registered with [REGISTER-DOC-IN-PAX-WORLD][f4fd].

    In the absence of :HEADER-FN :FOOTER-FN, :OUTPUT, every spec in PAGE-SPECS is augmented with HTML headers, footers and output location specifications (based on the name of the section).

    If necessary, a default page spec is created for every section.

9 Transcripts

What are transcripts for? When writing a tutorial, one often wants to include a REPL session with maybe a few defuns and a couple of forms whose output or return values are shown. Also, in a function's docstring an example call with concrete arguments and return values speaks volumes. A transcript is a text that looks like a REPL session, but which has a light markup for printed output and return values, while no markup (i.e. prompt) for Lisp forms. PAX transcripts may include output and return values of all forms, or only selected ones. In either case, the transcript itself can be easily generated from the source code. The main worry associated with including examples in the documentation is that they tend to get out-of-sync with the code. This is solved by being able to parse back and update transcripts.

9.1 Transcribing in Documentation

Consider this function:

(defun foo (x)
  "Return X + 1 and log what happened."
  (format t "Adding 1 to ~S~%" x)
  (1+ x))

Let's add an example to the docstring:

(defun foo (x)
  "Return X + 1 and log what happened.
  For example,

  ```cl
  (foo 7)
  .. Adding 1 to 7
  ..
  => 8
  ```"
  (format t "Adding 1 to ~S~%" x)
  (1+ x))

In this transcript above, output lines are prefixed with ".. " and return values with "=> ". The transcript could have been generated with (TRANSCRIBE "(foo 7)" *STANDARD-OUTPUT*) or interactively by invoking mgl-pax-transcribe-last-expression in Emacs with the point right after (FOO 7) (see [Transcript API][9dbc] and [Transcribing with Emacs][f5bd]).

If we want PAX to check that the transcript is consistent with the code when Generating Documentation, we add the cl-transcript tag to the code block:

(defun foo (x)
  "Return X + 1 and log what happened.
  For example,

  ```cl-transcript
  (foo 7)
  .. Adding 1 to 7
  ..
  => 8
  ```"
  (format t "Adding 1 to ~S~%" x)
  (+ x 2))

Since the code was also changed to (+ X 2), documenting this function causes the following error:

(document #'foo)
.. debugger invoked on TRANSCRIPTION-VALUES-CONSISTENCY-ERROR:
..   Transcription error: Readable value "8" in source is not EQUAL to "9".
..     [While documenting (MGL-PAX::FOO COMMON-LISP:FUNCTION)]
..
..   Form:
..   "(foo 7)"

When [Browsing Live Documentation][a595], any errors signalled during transcription are downgraded to warnings.

All in all, transcripts are a handy tool especially when combined with the Emacs support to update them and with PYTHONIC-STRING-READER's triple-quoted strings, that allow one to work with nested strings with less noise. The triple-quote syntax can be enabled with:

(in-readtable pythonic-string-syntax)

9.2 Transcribing with Emacs

With [Emacs set up][8541], we can have two Elisp functions at our disposal for working with transcripts.

  • mgl-pax-transcribe-last-expression: Evaluates a single form, and insert its output and return values into the current buffer.

  • mgl-pax-retranscribe-region: Retranscribes a region as with [TRANSCRIBE][f1f0] :UPDATE-ONLY T. The region defaults to the innermost enclosing cl-transcript block if any.

These functions work in any major and minor mode where the basic sexp movements are sufficiently Lisp-like. They also work anywhere in the buffer, including docstrings and comments at any indentation level. Like DOCUMENT, they understand the :DYNENV argument of cl-transcript and pass on the other arguments to TRANSCRIBE.

For example, consider this buffer content, which may be part of a docstring. Move the cursor right after the end of the form as if you were to evaluate it with C-x C-e. The cursor is marked by ^:

(values (princ :hello) (list 1 2))^

Now invoke the Elisp function mgl-pax-transcribe where the cursor is to insert the transcribed output and return values:

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)
^

Then, change the printed message and add comments to the second return value:

(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> ;; This is a list.
   (1
    ;; This value is arbitrary.
    2)

Obviously, the transcript is now out-of-date, and if it is in a cl-transcript code block in a docstring, then DOCUMENT will signal a [TRANSCRIPTION-CONSISTENCY-ERROR][a249]. So, let's update the transcript by marking the region bounded by | and the cursor at ^ in this example:

|(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> ;; This is a list.
   (1
    ;; This value is arbitrary.
    2)
^

Then, invoke the Elisp function mgl-pax-retranscribe-region to get

(values (princ :hello-world) (list 1 2))
.. HELLO-WORLD
=> :HELLO-WORLD
=> ;; This is a list.
   (1
    ;; This value is arbitrary.
    2)
^

Note how the indentation and comments of (1 2) were left alone, but the output and the first return value got updated.

Also note that when the cursor is in cl-transcript code block, mgl-pax-retranscribe-region defaults to the whole block when there is no active region.

Alternatively, C-u 1 mgl-pax-transcribe will emit commented markup:

(values (princ :hello) (list 1 2))
;.. HELLO
;=> :HELLO
;=> (1 2)

C-u 0 mgl-pax-retranscribe-region will turn commented into non-commented markup. In general, the numeric prefix argument is the index of the syntax to be used in [*TRANSCRIBE-SYNTAXES*][ebd3]. Without a prefix argument, mgl-pax-retranscribe-region will not change the markup style.

Finally, not only do both functions work at any indentation level but in comments too:

;;;; (values (princ :hello) (list 1 2))
;;;; .. HELLO
;;;; => :HELLO
;;;; => (1 2)

With mgl-pax-transcribe-last-expression, we strip the longest run of leading spaces and semicolons common to all lines of the expression in the buffer.

For mgl-pax-retranscribe-region, the longest run is truncated so that it does not extend beyond the column of the first form to be transcribed. Without this rule, the syntax used

;;(list 1 2)
;;;=> (1
;;;->  2)

would be ambiguous, as the ;=> could refer to => in the :DEFAULT syntax or to ;=> in :COMMENTED-1.

The dynamic environment of the transcription is determined by the :DYNENV argument of the enclosing cl-transcript code block (see [Controlling the Dynamic Environment][6b59]).

9.3 Controlling the Dynamic Environment

When the [TRANSCRIBE][f1f0] function is called directly, the forms in the transcript are evaluated in the current dynamic environment with the following exceptions.

  • The variables [*][9590], [**][78d1], [***][ea37], [/][9f2f], [//][e433], ///, -, [+][72a7], [++][fbb1], +++ are locally bound and updated as in a new [REPL][f83b] session.

  • [*STANDARD-OUTPUT*][e7ee], [*ERROR-OUTPUT*][66c6], *TRACE-OUTPUT*, *DEBUG-IO*, [*QUERY-IO*][f4bf] and [*TERMINAL-IO*][633c] are redirected to capture the output.

If TRANSCRIBE is invoked [by Emacs][f5bd] or [by the DOCUMENT function][9bd5], a new dynamic environment is established by [STANDARD-TRANSCRIBE-DYNENV][588c] (similar in spirit to WITH-STANDARD-IO-SYNTAX), which binds printer and reader variables to fixed values to make IO more predictable. This default can be overridden with the :DYNENV argument of cl-transcript:

```cl-transcript (:dynenv my-transcript)
...
```

In this case, instead of calling TRANSCRIBE directly, MY-TRANSCRIPT is called with a thunk (a function of no arguments) that wraps a call to TRANSCRIBE. Once MY-TRANSCRIPT establishes the desired dynamic environment, it calls its argument. The following definition of MY-TRANSCRIPT simply packages up oft-used settings to TRANSCRIBE.

(defun my-transcript (fn)
  (standard-transcribe-dynenv
    (let ((*transcribe-check-consistency*
            '((:output my-transcript-output=)
              (:readable equal)
              (:unreadable nil))))
      (funcall fn))))

(defun my-transcript-output= (string1 string2)
  (string= (my-transcript-normalize-output string1)
           (my-transcript-normalize-output string2)))

(defun my-transcript-normalize-output (string)
  (squeeze-whitespace (delete-trailing-whitespace (delete-comments string))))

The default for :DYNENV is STANDARD-TRANSCRIBE-DYNENV, and it is generally a good idea to use it as the above example does, but this is not required. Specify :DYNENV NIL explicitly if you want to use the current dynamic environment without any changes.

A more involved solution could establish bindings for global variables set in transcripts, unintern symbols created or even create a temporary package for evaluation.

  • [function] STANDARD-TRANSCRIBE-DYNENV FN

    Bind printer and reader variables to standard values and call FN.

    The bindings are the same as with WITH-STANDARD-IO-SYNTAX, but

    • [*PACKAGE*][5ed1] and [*READTABLE*][b79a] are unaffected (because both DOCUMENT and Emacs set these up);

    • [*PRINT-READABLY*][8aca] is NIL, [*PRINT-PRETTY*][782a] is T, [*PRINT-CIRCLE*][c8cb] is T and [*PRINT-RIGHT-MARGIN*][5ab6] is 72.

    This function is the default for the :DYNENV argument of cl-transcript. A function that overrides the default may want to call STANDARD-TRANSCRIBE-DYNENV with a lambda that establishes more bindings.

9.4 Transcript API

  • [function] TRANSCRIBE INPUT OUTPUT &KEY UPDATE-ONLY (INCLUDE-NO-OUTPUT UPDATE-ONLY) (INCLUDE-NO-VALUE UPDATE-ONLY) (ECHO T) (CHECK-CONSISTENCY *TRANSCRIBE-CHECK-CONSISTENCY*) DEFAULT-SYNTAX (INPUT-SYNTAXES *TRANSCRIBE-SYNTAXES*) (OUTPUT-SYNTAXES *TRANSCRIBE-SYNTAXES*)

    Read forms from INPUT and write them (iff ECHO) to OUTPUT followed by any output and return values produced by calling EVAL on the form.

    INPUT can be a stream or a string, while OUTPUT can be a stream or NIL, in which case output goes into a string. The return value is the OUTPUT stream or the string that was constructed. As the second value, a generalized boolean indicating whether any form was transcribed is returned.

    Go up to [Transcribing with Emacs][f5bd] for nice examples. A more mind-bending one is this:

    (transcribe "(princ 42) " nil)
=> "(princ 42)
.. 42
=> 42
"
=> T

    However, the above may be a bit confusing since this documentation uses TRANSCRIBE markup syntax in this very example, so let's do it differently. If we have a file with these contents:

    (values (princ 42) (list 1 2))

    it is transcribed to:

    (values (princ 42) (list 1 2))
.. 42
=> 42
=> (1 2)

    Output to all standard streams is captured and printed with the :OUTPUT prefix (".."). The return values above are printed with the :READABLE prefix ("=>"). Note how these prefixes are always printed on a new line to facilitate parsing.

    Updating

    TRANSCRIBE is able to parse its own output. If we transcribe the previous output above, we get it back exactly. However, if we remove all output markers, leave only a placeholder value marker and pass :UPDATE-ONLY T with source:

    (values (princ 42) (list 1 2))
=>

    we get this:

    (values (princ 42) (list 1 2))
=> 42
=> (1 2)

    With UPDATE-ONLY, the printed output of a form is transcribed only if there were output markers in the source. Similarly, with UPDATE-ONLY, return values are transcribed only if there were value markers in the source.

    No Output/Values

    If the form produces no output or returns no values, then whether or not output and values are transcribed is controlled by INCLUDE-NO-OUTPUT and INCLUDE-NO-VALUE, respectively. By default, neither is on so:

    (values)
..
=>

    is transcribed to

    (values)

    With UPDATE-ONLY true, we probably wouldn't like to lose those markers since they were put there for a reason. Hence, with UPDATE-ONLY, INCLUDE-NO-OUTPUT and INCLUDE-NO-VALUE default to true. So, with UPDATE-ONLY the above example is transcribed to:

    (values)
..
=> ; No value

    where the last line is the :NO-VALUE prefix.

    Consistency Checks

    If CHECK-CONSISTENCY is true, then TRANSCRIBE signals a continuable [TRANSCRIPTION-OUTPUT-CONSISTENCY-ERROR][8492] whenever a form's output as a string is different from what was in INPUT, provided that INPUT contained the output. Similarly, for values, a continuable TRANSCRIPTION-VALUES-CONSISTENCY-ERROR is signalled if a value read from the source does not print as the value returned by EVAL. This allows readable values to be hand-indented without failing consistency checks:

    (list 1 2)
=> ;; This is commented, too.
   (1
      ;; Funny indent.
      2)

    See Transcript Consistency Checking for the full picture.

    Unreadable Values

    The above scheme involves [READ][fe58], so consistency of unreadable values cannot be treated the same. In fact, unreadable values must even be printed differently for transcribe to be able to read them back:

    (defclass some-class () ())

(defmethod print-object ((obj some-class) stream)
  (print-unreadable-object (obj stream :type t)
    (format stream "~%~%end")))

(make-instance 'some-class)
==> #<SOME-CLASS 
-->
--> end>

    where "==>" is the :UNREADABLE prefix and "-->" is the :UNREADABLE-CONTINUATION prefix. As with outputs, a consistency check between an unreadable value from the source and the value from EVAL is performed with STRING= by default. That is, the value from EVAL is printed to a string and compared to the source value. Hence, any change to unreadable values will break consistency checks. This is most troublesome with instances of classes with the default PRINT-OBJECT method printing the memory address. See [Finer-Grained Consistency Checks][6e18].

    Errors

    If an [ERROR][d162] condition is signalled, the error is printed to the output and no values are returned.

    (progn
  (print "hello")
  (error "no greeting"))
..
.. "hello" 
.. debugger invoked on SIMPLE-ERROR:
..   no greeting

    To keep the textual representation somewhat likely to be portable, the printing is done with (FORMAT T "#<~S ~S>" (TYPE-OF ERROR) (PRINC-TO-STRING ERROR)). [SIMPLE-CONDITION][f2f5]s are formatted to strings with SIMPLE-CONDITION-FORMAT-CONTROL and [SIMPLE-CONDITION-FORMAT-ARGUMENTS][da14].

    Syntaxes

    Finally, a transcript may employ different syntaxes for the output and values of different forms. When INPUT is read, the syntax for each form is determined by trying to match all prefixes from all syntaxes in INPUT-SYNTAXES against a line. If there are no output or values for a form in INPUT, then the syntax remains undetermined.

    When OUTPUT is written, the prefixes to be used are looked up in DEFAULT-SYNTAX of OUTPUT-SYNTAXES if DEFAULT-SYNTAX is not NIL. If DEFAULT-SYNTAX is NIL, then the syntax used by the same form in the INPUT is used or (if that could not be determined) the syntax of the previous form. If there was no previous form, then the first syntax if OUTPUT-SYNTAXES is used.

    To produce a transcript that's executable Lisp code, use :DEFAULT-SYNTAX :COMMENTED-1:

    (make-instance 'some-class)
;==> #<SOME-CLASS
;-->
;--> end>

(list 1 2)
;=> (1
;->    2)

    To translate the above to uncommented syntax, use :DEFAULT-SYNTAX :DEFAULT. If DEFAULT-SYNTAX is NIL (the default), the same syntax will be used in the output as in the input as much as possible.

  • [variable] *TRANSCRIBE-CHECK-CONSISTENCY* NIL

    The default value of [TRANSCRIBE][f1f0]'s CHECK-CONSISTENCY argument.

  • [variable] *TRANSCRIBE-SYNTAXES* ((:DEFAULT (:OUTPUT "..") (:NO-VALUE "=> ; No value") (:READABLE "=>") (:UNREADABLE "==>") (:UNREADABLE-CONTINUATION "-->")) (:COMMENTED-1 (:OUTPUT ";..") (:NO-VALUE ";=> ; No value") (:READABLE ";=>") (:READABLE-CONTINUATION ";->") (:UNREADABLE ";==>") (:UNREADABLE-CONTINUATION ";-->")) (:COMMENTED-2 (:OUTPUT ";;..") (:NO-VALUE ";;=> ; No value") (:READABLE ";;=>") (:READABLE-CONTINUATION ";;->") (:UNREADABLE ";;==>") (:UNREADABLE-CONTINUATION ";;-->")))

    The default syntaxes used by [TRANSCRIBE][f1f0] for reading and writing lines containing output and values of an evaluated form.

    A syntax is a list of of the form (SYNTAX-ID &REST PREFIXES) where PREFIXES is a list of (PREFIX-ID PREFIX-STRING) elements. For example the syntax :COMMENTED-1 looks like this:

    (:commented-1
 (:output ";..")
 (:no-value ";=>  No value")
 (:readable ";=>")
 (:readable-continuation ";->")
 (:unreadable ";==>")
 (:unreadable-continuation ";-->"))

    All of the above prefixes must be defined for every syntax except for :READABLE-CONTINUATION. If that's missing (as in the :DEFAULT syntax), then the following value is read with [READ][fe58] and printed with [PRIN1][6384] (hence no need to mark up the following lines).

    When writing, an extra space is added automatically if the line to be prefixed is not empty. Similarly, the first space following the prefix is discarded when reading.

    See TRANSCRIBE for how the actual syntax to be used is selected.

  • [condition] TRANSCRIPTION-ERROR [ERROR][d162] CONDITION-CONTEXT-MIXIN

    Represents syntactic errors in the SOURCE argument of [TRANSCRIBE][f1f0] and also serves as the superclass of [TRANSCRIPTION-CONSISTENCY-ERROR][a249].

  • [condition] TRANSCRIPTION-OUTPUT-CONSISTENCY-ERROR [TRANSCRIPTION-CONSISTENCY-ERROR][a249]

    Signalled (with CERROR) by [TRANSCRIBE][f1f0] when invoked with :CHECK-CONSISTENCY and the output of a form is not the same as what was parsed.

  • [condition] TRANSCRIPTION-VALUES-CONSISTENCY-ERROR [TRANSCRIPTION-CONSISTENCY-ERROR][a249]

    Signalled (with CERROR) by [TRANSCRIBE][f1f0] when invoked with :CHECK-CONSISTENCY and the values of a form are inconsistent with their parsed representation.

9.5 Transcript Consistency Checking

The main use case for consistency checking is detecting out-of-date examples in documentation, although using it for writing tests is also a possibility. Here, we focus on the former.

When a Markdown code block tagged cl-transcript is processed during Generating Documentation, the code in it is replaced with the output of (TRANSCRIBE <CODE> NIL :UPDATE-ONLY T :CHECK-CONSISTENCY T). Suppose we have the following example of the function GREET, that prints hello and returns 7.

```cl-transcript
(greet)
.. hello
=> 7
```

Now, if we change GREET to print or return something else, a [TRANSCRIPTION-CONSISTENCY-ERROR][a249] will be signalled during documentation generation. Then we may fix the documentation or CONTINUE from the error.

By default, comparisons of previous to current output, readable and unreadable return values are performed with STRING=, EQUAL, and STRING=, respectively, which is great in the simple case. Non-determinism aside, exact matching becomes brittle as soon as the notoriously unportable pretty printer is used or when unreadable objects are printed with their #<> syntax, especially when [PRINT-UNREADABLE-OBJECT][9439] is used with :IDENTITY T.

9.5.1 Finer-Grained Consistency Checks

To get around this problem, consistency checking of output, readable and unreadable values can be customized individually by supplying [TRANSCRIBE][f1f0] with a CHECK-CONSISTENCY argument like ((:OUTPUT <OUTPUT-CHECK>) (:READABLE <READABLE-CHECK>) (:UNREADABLE <UNREADABLE-CHECK>)). In this case, <OUTPUT-CHECK> may be NIL, T, or a function designator.

  • If it's NIL or there is no :OUTPUT entry in the list, then the output is not checked for consistency.

  • If it's T, then the outputs are compared with the default, STRING=.

  • If it's a function designator, then it's called with two strings and must return whether they are consistent with each other.

The case of <READABLE-CHECK> and <UNREADABLE-CHECK> is similar.

Code blocks tagged cl-transcript can take arguments, which they pass on to TRANSCRIBE. The following shows how to check only the output.

```cl-transcript (:check-consistency ((:output t)))
(error "Oh, no.")
.. debugger invoked on SIMPLE-ERROR:
..   Oh, no.

(make-condition 'simple-error)
==> #<SIMPLE-ERROR {1008A81533}>
```

It is often a good idea to package up these settings in the :DYNENV argument of cl-transcript (see [Controlling the Dynamic Environment][6b59]).

9.5.2 Utilities for Consistency Checking

  • [function] SQUEEZE-WHITESPACE STRING

    Replace consecutive whitespace characters with a single space in STRING and trim whitespace from the right. This is useful to undo the effects of pretty printing when building comparison functions for [TRANSCRIBE][f1f0].

  • [function] DELETE-TRAILING-WHITESPACE STRING

    Delete whitespace characters after the last non-whitespace character in each line in STRING.

  • [function] DELETE-COMMENTS STRING &KEY (PATTERN ";")

    For each line in STRING delete the rest of the line after and including the first occurrence of PATTERN. On changed lines, delete trailing whitespace too. This function does not parse STRING as Lisp forms, hence all occurrences of PATTERN (even those seemingly in string literals) are recognized as comments.

    Let's define a comparison function:

    (defun string=/no-comments (string1 string2)
  (string= (delete-comments string1) (delete-comments string2)))

    And use it to check consistency of output:

      ```cl-transcript (:check-consistency ((:output string=/no-comments)))
  (format t "hello~%world")
  .. hello     ; This is the first line.
  .. world     ; This is the second line.
  ```

    Just to make sure the above example works, here it is without being quoted.

    (format t "hello~%world")
.. hello     ; This is the first line.
.. world     ; This is the second line.

10 Parsing

10.1 Parsing Names

When encountering a [word][d7b0] such as CLASSes in a docstring, PAX needs to find the [name][88cf], and how that's done varies slightly. [Codification][f1ab], for example, looks for [interesting][7445] names, Navigating Sources in Emacs for names with Lisp DEFINITIONS, and Linking for names with any kind of definition.

This is not as straightforward as it sounds because it needs to handle cases like un[READ][fe58]able, [PRINT][d451]ed, and all the various forms of Linking in docstrings as well as in comments, and the (NAME LOCATIVE) syntax in [DEFSECTION][72b4].

  • [glossary-term] word

    A word is a string from which we want to extract a [name][88cf]. When Navigating, the word is slime-sexp-at-point or the label of a [Markdown reference link][8c00] if point is over one. Similarly, when Generating Documentation, it is a non-empty string between whitespace characters in a docstring or the label of a [Markdown reference link][8c00].

  • [glossary-term] raw name

    A raw name is a string from which a [name][88cf] may be read. Raw names correspond to an intermediate parsing step between [word][d7b0]s and [name][88cf]s. See Names in Raw Names.

  • [glossary-term] name

    A name is a DRef name. That is, a symbol, a string or a nested list of the previous associated with a definition, whose kind is given by a locative.

Depending on the context, trimming and depluralization may be enabled (see [Raw Names in Words][f0d5]), while the possible names may be restricted to symbols (see Names in Raw Names).

  • Trimming: Enabled for M-. Defaulting and [Codification][f1ab].

  • Depluralization: Enabled when the [word][d7b0] is part of the normal flow of text (i.e. not for [Specific Reflink with Text][fb17], Unspecific Reflink with Text and various Elisp functions such as mgl-pax-apropos unless they determine their argument from buffer contents).

  • Symbols only: This is the case for [Codification][f1ab] and [Unspecific Autolink][e2a4] to prevent string-based definitions from littering the documentation with links without the control provided by explicitly [IMPORT][8f46]ing symbols.

For a word, a number of [raw name][f5af]s are generated by trimming delimiter characters and plural markers, and for each raw name a number of names are considered until one is found suitable in the context. The following subsections describe the details of the parsing algorithm.

10.1.1 Raw Names in Words

From [word][d7b0]s, [raw name][f5af]s are parsed by trimming some prefixes and suffixes. For a given word, multiple raw names are considered in the following order.

  1. The entire word.

  2. Trimming the following characters from the left of the word:

     #<{;"'`

  3. Trimming the following characters from the right of the word:

     ,;:.>}"'`

  4. Trimming both of the previous two at the same time.

  5. From the result of 4., If a [word][d7b0] ends with what looks like a plural marker (case-insensitive), then a [name][88cf] is created by removing it. For example, from the word BUSES the plural marker ES is removed to produce the name BUS. The list of plural markers considered is SES (e.g. GASSES), ES (e.g. BUSES), S (e.g. CARS), ZES (e.g. FEZZES), and REN (e.g. CHILDREN).

  6. From the result of 4., removing the prefix before the first, and the suffix after the last uppercase character if it contains at least one lowercase character.

10.1.2 Names in Raw Names

For each [raw name][f5af] from [Raw Names in Words][f0d5], various [name][88cf]s may be considered until one is found that's suitable in the context.

The following examples list the names considered for a given raw name, assuming that READTABLE-CASE is :UPCASE as well as that FOO and |Foo| are interned.

  • "foo": FOO, "foo", "FOO" (rules 1, 2, 3)

  • "FOO": FOO, "FOO" (rules 1, 2)

  • "Foo": "Foo", "FOO" (rules 2, 3)

  • "|Foo|": |Foo| (rule 4)

  • "\"foo\"": "foo" (rule 5)

The rules are:

  1. If the raw name is not mixed case (i.e. it doesn't have both upper- and lowercase characters) and it names an interned symbol (subject to the current [Package and Readtable][ab7e]), then that symbol is considered as a name.

  2. The raw name itself (a string) is considered a name.

  3. The raw name upcased or downcased according to READTABLE-CASE (subject to the [current readtable][ab7e]) but still as a string. This is to allow [dref][package] to refer to the "DREF" package regardless of whether the symbol DREF is interned in the current package.

  4. If the raw name is explicitly a symbol (it starts with #\|), and it names an interned symbol (subject to the current [Package and Readtable][ab7e]), then that symbol is considered as a name and nothing else.

  5. If the raw name has an embedded string (it starts with #\") and [READ-FROM-STRING][d813] can read the embedded string from it, then that string is considered as a name and nothing else.

10.1.3 Name Parsing Example

For example, when M-. is pressed while point is over unREADable., the last word of the sentence It may be unREADable., the following [raw name][f5af]s are considered until one is found with a definition:

  1. The entire word, "unREADable.".

  2. Trimming left does not produce a new raw name.

  3. Trimming right removes the dot and gives "unREADable".

  4. Trimming both is the same as trimming right.

  5. No plural markers are found.

  6. The lowercase prefix and suffix is removed around the uppercase core, giving "READ". This names an interned symbol which has a definition, so M-. will visit it.

When Generating Documentation, [Autolink][ec7a]ing behaves similarly.

10.2 Parsing Locatives

Locatives are parsed almost as if by [READ][fe58]. They are found in buffer contents around a [word][d7b0] when M-. Defaulting or Generating Documentation, and in the string entered when [M-. Prompting][ed46], with a similar distinction when [Browsing Live Documentation][a595].

Parsing deviates from READ in the following ways.

  • No new symbols are [INTERN][b4f0]ed during parsing. If an expression contains uninterned symbols, then it is not parsable as a locative.

  • Read-time evaluation ([#.][ffd7]) follows normal READ semantics. Thus, (METHOD ((EQL #.(FIND-PACKAGE 'CL)))) may INTERN the symbol CL.

  • A locative that involves unreadable objects that print using the #< syntax (e.g. (METHOD ((EQL #<PACKAGE DREF>)))) is parsable in the context of a [name][88cf] if each unreadable object in the locative occurs in one of the DEFINITIONS of that name and it prints to an equivalent string (e.g. #<PACKAGE DREF> above).

  • [glossary-term] prints to an equivalent string

    An object with an unreadable representation is said to print to some string S if its [PRIN1][6384] representation (under WITH-STANDARD-IO-SYNTAX but in the current package and with [*PRINT-READABLY*][8aca] NIL) is the same as S, where consecutive whitepace characters are replaced with a single space in both strings, and the comparison is case-insensitive.

    See the related concept of stable printed locative, that requires the printed representation of entire locatives to be unique and non-changing to support Linking.

11 Writing Extensions

11.1 Adding New Locatives

Once everything in Extending DRef has been done, there are only a couple of PAX generic functions left to extend.

  • [generic-function] DOCUMENT-OBJECT* OBJECT STREAM

    Write the documentation of OBJECT in Markdown to STREAM. The output may depend on *FORMAT*. Specialize this on a subclass of DREF if that subclass is not RESOLVEable, else on the type of object it resolves to. This function is for extension only. Don't call it directly.

  • [generic-function] EXPORTABLE-REFERENCE-P PACKAGE SYMBOL LOCATIVE-TYPE LOCATIVE-ARGS

    Return true if SYMBOL is to be exported from PACKAGE when it occurs in a [DEFSECTION][72b4] in a reference with LOCATIVE-TYPE and LOCATIVE-ARGS. SYMBOL is [accessible][9b15] in PACKAGE.

    The default method calls [EXPORTABLE-LOCATIVE-TYPE-P][c930] with LOCATIVE-TYPE and ignores the other arguments.

    By default, [SECTION][5fac]s, [GLOSSARY-TERM][8251]s, CONCEPTs and [NOTE][e2ae]s are not exported although they are EXPORTABLE-LOCATIVE-TYPE-P. To export symbols naming sections from MGL-PAX, the following method could be added:

    (defmethod exportable-reference-p ((package (eql (find-package 'mgl-pax)))
                                   symbol (locative-type (eql 'section))
                                   locative-args)
  t)

  • [generic-function] EXPORTABLE-LOCATIVE-TYPE-P LOCATIVE-TYPE

    Return true if symbols in references with LOCATIVE-TYPE are to be exported by default when they occur in a [DEFSECTION][72b4]. The default method returns T, while the methods for locative types PACKAGE, METHOD, ASDF:SYSTEM, and [INCLUDE][5cd7] return NIL.

    This function is called by the default method of [EXPORTABLE-REFERENCE-P][e51f] to decide what symbols DEFSECTION shall export when its EXPORT argument is true.

Also note that due to the [HOME-SECTION][fda4] logic, especially for locative types with string names, DREF-EXT:DOCSTRING* should probably return a non-NIL package.

11.2 Locative Aliases

DEFINE-LOCATIVE-ALIAS can be used to help M-. and Specific Autolinks disambiguate references based on the context of a [name][88cf] as described on Parsing.

The following example shows how to make docstrings read more naturally by defining an alias.

(defclass my-string ()
  ())

(defgeneric my-string (obj)
  (:documentation "Convert OBJ to MY-STRING."))

;;; This version of FOO has a harder to read docstring because
;;; it needs to disambiguate the MY-STRING reference.
(defun foo (x)
  "FOO takes an argument X, a [MY-STRING][class] object.")

;;; Define OBJECT as an alias for the CLASS locative.
(define-locative-alias object class)

;;; Note how no explicit link is needed anymore.
(defun foo (x)
  "FOO takes an argument X, a MY-STRING object.")

Similarly, defining the indefinite articles as aliases of the CLASS locative can reduce the need for explicit linking.

(define-locative-alias a class)
(define-locative-alias an class)

Since these are unlikely to be universally helpful, make sure not to export the symbols A and AN.

11.3 Extending DOCUMENT

For all definitions that it encounters, DOCUMENT calls [DOCUMENT-OBJECT*][8269] to generate documentation. The following utilities are for writing new DOCUMENT-OBJECT* methods. They emit Markdown encoded in an unspecified format internal to DOCUMENT.

  • [variable] *FORMAT*

    Bound by DOCUMENT to its FORMAT argument, this allows Markdown output to depend on the output format.

  • [macro] WITH-HEADING (STREAM &KEY DREF LINK-TITLE-TO (NUMBERED T)) &BODY BODY

    Write a Markdown heading with the [DOCTITLE][e619] of DREF to STREAM.

    • DREF defaults to the definition for which documentation is currently being generated.

    • Nested WITH-HEADINGs produce nested headings.

    • If *DOCUMENT-LINK-SECTIONS*, generate anchors based on DREF.

    • LINK-TITLE-TO behaves like the LINK-TITLE-TO argument of [DEFSECTION][72b4].

    • NUMBERED controls whether a heading number is assigned to the heading. The heading number reflects the nesting structure of WITH-HEADINGS (e.g. 3.1.2). Descendants of unnumbered headings are unnumbered regardless of NUMBERED.

  • [generic-function] DOCTITLE* OBJECT

    DOCTITLE* extends [DOCTITLE][e619] in the same way as DOCSTRING* extends DOCSTRING.

    The default method returns NIL.

    This function is for extension only. Do not call it directly.

  • [macro] DOCUMENTING-DEFINITION (STREAM &KEY DREF PACKAGE READTABLE ARGLIST) &BODY BODY

    Write DREF to STREAM as described in [*DOCUMENT-MARK-UP-SIGNATURES*][8fb6], and establish DREF as a [Local Definition][9db9] for the processing of BODY.

    • DREF defaults to the definition for which documentation is currently being generated.

    • If DREF has a [DOCTITLE][e619], then it is [PRINC][676d]ed after the LOCATIVE-TYPE (see Markdown in Titles). Else, (DREF-NAME DREF) is printed subject to [*DOCUMENT-DOWNCASE-UPPERCASE-CODE*][a5ee] but with all Markdown and [MathJax][a17d] markup escaped.

    • [*PACKAGE*][5ed1] and [*READTABLE*][b79a] are bound to PACKAGE and READTABLE for the duration of printing the ARGLIST and the processing of BODY. If either is NIL, then a default value is computed as described in [Package and Readtable][ab7e].

    • ARGLIST:

      • If it is not provided, then it defaults to (ARGLIST DREF).

      • If NIL, then it is not printed.

      • If it is a list, then it is must be a [lambda list][98ff] and is printed without the outermost parens and with the package names removed from the argument names. [Keywords][88a0] are printed with a leading colon.

        &AUX variables and SUPPLIED-P-PARAMETERs of &OPTIONAL and &KEY arguments are considered internal and are not printed.

      • If it is a string, then it must be valid Markdown.

    • It is not allowed to have [WITH-HEADING][80e8] within the dynamic extent of BODY.

  • [macro] WITH-DISLOCATED-NAMES NAMES &BODY BODY

    For each name in NAMES, establish a [Local Definition][9db9].

  • [function] DOCUMENT-DOCSTRING DOCSTRING STREAM &KEY (INDENTATION " ") EXCLUDE-FIRST-LINE-P (PARAGRAPHP T)

    Write DOCSTRING to STREAM, [sanitizing the Markdown][7bf5] in it, performing [Codification][f1ab] and Linking, finally prefixing each line with INDENTATION. The prefix is not added to the first line if EXCLUDE-FIRST-LINE-P. If PARAGRAPHP, then add a newline before and after the output.

  • [function] ESCAPE-MARKDOWN STRING &KEY (ESCAPE-INLINE T) (ESCAPE-MATHJAX T) (ESCAPE-HTML T) (ESCAPE-BLOCK T)

    Backslash escape Markdown constructs in STRING.

    • If ESCAPE-INLINE, then escape the following characters:

        *_`[]\

    • If ESCAPE-MATHJAX, then escape $ characters.

    • If ESCAPE-HTML, then escape the following characters:

        <&

    • If ESCAPE-BLOCK, then escape whatever is necessary to avoid starting a new Markdown block (e.g. a paragraph, heading, etc).

  • [function] ENSURE-MD-PARAGRAPH STREAM

    Ensure that output previously written to STREAM is in a separate Markdown paragraph than the output that follows. Calling this function multiple times without writing anything else to STREAM in between is equivalent to calling it once.

11.4 Sections

[SECTION][5fac] objects rarely need to be dissected since [DEFSECTION][72b4] and DOCUMENT cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

  • [class] SECTION

    [DEFSECTION][72b4] stores its NAME, TITLE, PACKAGE, [READTABLE][d646] and ENTRIES arguments in [SECTION][5fac] objects.

  • [reader] SECTION-NAME [SECTION][5fac] (:NAME)

    The name of the global variable whose value is this [SECTION][5fac] object.

  • [reader] SECTION-PACKAGE [SECTION][5fac] (:PACKAGE)

    [*PACKAGE*][5ed1] will be bound to this package when generating documentation for this section.

  • [reader] SECTION-READTABLE [SECTION][5fac] (:READTABLE)

    [*READTABLE*][b79a] will be bound to this when generating documentation for this section.

  • [reader] SECTION-TITLE [SECTION][5fac] (:TITLE)

    A title or NIL. Used in generated documentation (see [Markdown Output][dd29]) and is returned by [DOCTITLE][e619] for [SECTION][5fac] objects and SECTION definitions.

  • [function] SECTION-LINK-TITLE-TO SECTION

  • [function] SECTION-ENTRIES SECTION

    A list of Markdown docstrings and XREFs in the order they occurred in [DEFSECTION][72b4].

11.5 Glossary Terms

[GLOSSARY-TERM][8251] objects rarely need to be dissected since [DEFINE-GLOSSARY-TERM][8ece] and DOCUMENT cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

  • [class] GLOSSARY-TERM

    See [DEFINE-GLOSSARY-TERM][8ece].

  • [reader] GLOSSARY-TERM-NAME [GLOSSARY-TERM][8251] (:NAME)

    The name of the global variable whose value is this [GLOSSARY-TERM][8251] object.

  • [reader] GLOSSARY-TERM-TITLE [GLOSSARY-TERM][8251] (:TITLE)

    A title or NIL. Used in generated documentation (see [Markdown Output][dd29]) and is returned by [DOCTITLE][e619] for [GLOSSARY-TERM][8251] objects and GLOSSARY-TERM definitions.

  • [reader] GLOSSARY-TERM-URL [GLOSSARY-TERM][8251] (:URL)

    A string or NIL.

Truncated — view the full README on GitHub.

common-lisp
documentation-generator
documentation-tools
illiterate-programming
rubber-doc-development
untangled-lp

Contributors

melisgl

982 commits

svetlyak40wt

2 commits

fare

1 commits

kingcons

1 commits

Similar repos

elixir-lang/ex_doc

ExDoc produces HTML, EPUB, and Markdown documentation for Erlang and Elixir projects

1,615

sphinx-doc/sphinx

The Sphinx documentation generator

8,023

atgreen/icl

Interactive Common Lisp: an enhanced REPL

201

machow/quartodoc

Generate API documentation with quarto

265

susam/emacs4cl

A tiny DIY kit to set up vanilla Emacs for Common Lisp programming

394

iamgio/quarkdown

🪐 Markdown with superpowers: from ideas to papers, presentations, websites, books, and knowledge…

16,210

jupyter/papyri

87

tefkah/emacs-docs

The documentation website Emacs deserves

356

Languages

Common Lisp

87.1%

Emacs Lisp

10.3%

CSS

2.0%

melisgl/mgl-pax

PAX, the Common Lisp documentation system

Common Lisp

94

988 commits

updated Aug 3, 2026

fixnum.com

See the code

README

PAX Manual

Table of Contents

  • [1 Introduction][685e]

  • [2 Links and Systems][ba90]

  • [3 Emacs Setup][8541]

    • [3.1 Functionality Provided][d4a9]

    • [3.2 Installing from Quicklisp][f3f4]

    • [3.3 Loading PAX][d3fc]

    • [3.4 Setting up Keys][cfab]

  • [4 Background][f74b]

  • [5 Basics][94c7]

  • 6 PAX Locatives

  • 7 Navigating Sources in Emacs

    • 7.1 M-. Defaulting

    • [7.2 M-. Prompting][ed46]

      • [7.2.1 M-. Minibuffer Syntax][8106]

      • [7.2.2 M-. Completion][e444]

  • 8 Generating Documentation

    • [8.1 The DOCUMENT Function][dc0a]

      • 8.1.1 DOCUMENTABLE

      • [8.1.2 Return Values][7dc7]

      • [8.1.3 PAGES][9c7d]

      • [8.1.4 Package and Readtable][ab7e]

    • [8.2 Browsing Live Documentation][a595]

      • [8.2.1 Browsing with w3m][83d5]

      • [8.2.2 Browsing with Other Browsers][c434]

      • [8.2.3 Apropos][b7fc]

      • [8.2.4 PAX Live Home Page][9d50]

    • [8.3 Markdown Support][c2d3]

    • [8.4 Codification][f1ab]

    • 8.5 Linking

      • [8.5.1 Reflink][cbc4]

      • [8.5.2 Autolink][ec7a]

      • [8.5.3 Linking to the HyperSpec][7cc3]

      • 8.5.4 Linking to Sections

      • [8.5.5 Filtering Links][b2e4]

      • [8.5.6 Link Format][c0d2]

    • [8.6 Local Definition][9db9]

    • 8.7 Overview of Escaping

    • [8.8 Indexing][adf8]

    • [8.9 Output Formats][8d9b]

      • [8.9.1 Plain Output][c879]

      • [8.9.2 Markdown Output][dd29]

      • 8.9.3 PDF Output

      • [8.9.4 Dummy Output][f7e6]

    • [8.10 Documentation Generation Implementation Notes][d1ca]

    • 8.11 Utilities for Generating Documentation

  • [9 Transcripts][6300]

    • [9.1 Transcribing in Documentation][9bd5]

    • [9.2 Transcribing with Emacs][f5bd]

    • [9.3 Controlling the Dynamic Environment][6b59]

    • [9.4 Transcript API][9dbc]

    • 9.5 Transcript Consistency Checking

      • [9.5.1 Finer-Grained Consistency Checks][6e18]

      • [9.5.2 Utilities for Consistency Checking][8423]

  • 10 Parsing

    • [10.1 Parsing Names][e65d]

    • [10.2 Parsing Locatives][ab38]

  • [11 Writing Extensions][c4ce]

[in package MGL-PAX with nicknames PAX]

1 Introduction

What if documentation were code?

If some narrative glued docstrings together, we'd be able to develop and explore the code alongside its documentation. The main tool that PAX provides for this is [DEFSECTION][72b4]:

(defsection @foo-random-manual (:title "Foo Random manual")
  "Foo Random is a random number generator library."
  (foo-random-state class)
  (uniform-random function)
  (@foo-random-examples section))

Like the one above, sections can have docstrings and references to definitions (e.g. (UNIFORM-RANDOM FUNCTION)). These are the glue. To support interactive development, PAX

  • makes [SLIME][6be7]'s [M-.][cb15] work with references and

  • adds a documentation browser.

See [Emacs Setup][8541].

Beyond interactive workflows, Generating Documentation from sections and all the referenced items in Markdown or HTML format is also implemented.

PAX [emphasizes the narrative][bbc6] much like Literate Programming, but it is an untangled LP system.

Docstrings

PAX automatically recognizes and [marks up code][f1ab] with backticks and links names in code to their definitions. Take, for instance, SBCL's ABORT function, whose docstring is written in the usual style, uppercasing names of symbols:

(docstring #'abort)
=> "Transfer control to a restart named ABORT, signalling
a CONTROL-ERROR if none exists."

Note how in the generated documentation, ABORT is set with a monospace font, while CONTROL-ERROR is [Autolink][ec7a]ed:

  • [function] ABORT &OPTIONAL CONDITION

    Transfer control to a restart named ABORT, signalling a CONTROL-ERROR if none exists.

The following [transcript][6300] shows the raw Markdown for the previous example.

(document #'abort :format :markdown)
.. - [function] **ABORT** *&OPTIONAL CONDITION*
..
..     Transfer control to a restart named `ABORT`, signalling
..     a [`CONTROL-ERROR`][7c2c] if none exists.
..
..   [7c2c]: http://www.lispworks.com/documentation/HyperSpec/Body/e_contro.htm "CONTROL-ERROR (MGL-PAX:CLHS CONDITION)"
..
A Complete Example

Here is an example of how it all works together:

(mgl-pax:define-package :foo-random
  (:documentation "This package provides various utilities for random.
  See FOO-RANDOM:@FOO-RANDOM-MANUAL.")
  (:use #:common-lisp #:mgl-pax))

(in-package :foo-random)

(defsection @foo-random-manual (:title "Foo Random manual")
  "FOO-RANDOM is a random number generator library inspired by CL:RANDOM.
  Functions such as UNIFORM-RANDOM use *FOO-STATE* and have a
  :RANDOM-STATE keyword arg."
  (foo-random-state class)
  (state (reader foo-random-state))
  "Hey we can also print states!"
  (print-object (method (foo-random-state t)))
  (*foo-state* variable)
  (gaussian-random function)
  (uniform-random function)
  ;; This is a subsection.
  (@foo-random-examples section))

(defclass foo-random-state ()
  ((state :reader state)))

(defmethod print-object ((object foo-random-state) stream)
  (print-unreadable-object (object stream :type t)))

(defvar *foo-state* (make-instance 'foo-random-state)
  "Much like *RANDOM-STATE* but uses the FOO algorithm.")

(defun uniform-random (limit &key (random-state *foo-state*))
  "Return a random number from the uniform distribution between 0 and
  LIMIT (exclusive)."
  nil)

(defun gaussian-random (stddev &key (random-state *foo-state*))
  "Return a random number from a zero mean normal distribution with
  STDDEV."
  nil)

(defsection @foo-random-examples (:title "Examples")
  "Let's see the transcript of a real session of someone working
  with FOO:

  ```cl-transcript
  (values (princ :hello) (list 1 2))
  .. HELLO
  => :HELLO
  => (1 2)

  (make-instance 'foo-random-state)
  ==> #<FOO-RANDOM-STATE >
  ```")

Note how (VARIABLE *FOO-STATE*) in the [DEFSECTION][72b4] form both exports *FOO-STATE* and includes its documentation in @FOO-RANDOM-MANUAL. The symbols VARIABLE and FUNCTION are just two instances of locatives, used in DEFSECTION to refer to definitions tied to symbols.

(DOCUMENT @FOO-RANDOM-MANUAL) generates fancy Markdown or HTML output with [automatic markup][f25f] and [Autolink][ec7a]s uppercase [word][d7b0]s found in docstrings, numbers sections, and creates a table of contents.

One can even generate documentation for different but related libraries at the same time with the output going to different files but with cross-page links being automatically added for symbols mentioned in docstrings. In fact, this is what PAX World does. See Generating Documentation for some convenience functions to cover the most common cases.

The [transcript][6300] in the code block tagged with cl-transcript is automatically checked for up-to-dateness when documentation is generated.

The official repository is https://github.com/melisgl/mgl-pax, and this document is available in various formats on https://fixnum.com for the latest version. There is also a PAX channel on YouTube with a couple of videos.

PAX is built on top of the DRef library. See https://github.com/melisgl/dref.

  • Installation for deployment

    The base system is [mgl-pax][6fdb]. It has very few dependencies and is sufficient as a dependency for systems using the [Basics][94c7] to add documentation. This is to keep deployed code small. To install only the bare minimum, with no intention of using Navigating Sources in Emacs, Generating Documentation, [Browsing Live Documentation][a595] or using [Transcripts][6300], under Quicklisp for example, PAX could be installed as:

      (ql:quickload "mgl-pax")

  • Installation for development

    The heavier dependencies are on the other systems, which correspond to the main functionalities provided, intended to be used primarily during development. To install the dependencies for all features under Quicklisp, do

      (ql:quickload "mgl-pax/full")

    Having thus installed the dependencies, it is enough to load the base system, which will autoload the other systems as necessary.

  • [system] "mgl-pax"

  • [system] "mgl-pax/navigate"

    • Description: Support for Navigating Sources in Emacs via Slime's [M-.][cb15] in MGL-PAX.

    • Depends on: alexandria, autoload-doc, dref/full, [mgl-pax][6fdb], swank(?)

    • Defsystem depends on: autoload

  • [system] "mgl-pax/document"

    • Description: Support for Generating Documentation in MGL-PAX.

    • Depends on: 3bmd, 3bmd-ext-code-blocks, 3bmd-ext-math, alexandria, autoload-doc, colorize, md5, [mgl-pax/navigate][f155], [mgl-pax/transcribe][5825], trivial-utf-8

    • Defsystem depends on: autoload

  • [system] "mgl-pax/web"

    • Description: Web server for [Browsing Live Documentation][a595] in MGL-PAX.

    • Depends on: hunchentoot, mgl-pax/document

    • Defsystem depends on: autoload

  • [system] "mgl-pax/transcribe"

    • Description: Support for [Transcripts][6300] in MGL-PAX.

    • Depends on: alexandria, [mgl-pax/navigate][f155]

    • Defsystem depends on: autoload

  • [system] "mgl-pax/full"

    • Description: The [mgl-pax][6fdb] system with all features preloaded.

    • Depends on: mgl-pax/document, [mgl-pax/navigate][f155], [mgl-pax/transcribe][5825], [mgl-pax/web][a8c5]

    • Defsystem depends on: autoload

3 Emacs Setup

Here is a quick recipe for setting up PAX for use via [SLIME][6be7] to take advantage of the [conveniences on offer][d4a9]. Conversely, there is no need to do any of this just to use [DEFSECTION][72b4], write docstrings and for Generating Documentation.

If PAX was installed from Quicklisp, then evaluate this in CL to install the Elisp code in a stable location:

(mgl-pax:install-pax-elisp "~/quicklisp/")

Assuming the Elisp file is in the ~/quicklisp/ directory, add something like this to your .emacs:

(add-to-list 'load-path "~/quicklisp/")
(require 'mgl-pax)
(global-set-key (kbd "C-.") 'mgl-pax-document)
(global-set-key (kbd "s-x t") 'mgl-pax-transcribe-last-expression)
(global-set-key (kbd "s-x r") 'mgl-pax-retranscribe-region)

Alternatively, with use-package:

(use-package mgl-pax
  :load-path "~/quicklisp"
  :after slime
  :demand t
  :bind (("C-." . mgl-pax-document)
         ("s-x t" . mgl-pax-transcribe-last-expression)
         ("s-x r" . mgl-pax-retranscribe-region))
  :config
  (mgl-pax-hijack-slime-doc-keys))

When [Browsing with Other Browsers][c434], for clicking on the locative next to a definition to visit the corresponding source location in Emacs, permission needs to be given:

(setq slime-enable-evaluate-in-emacs t)

3.1 Functionality Provided

  • For Navigating Sources in Emacs, loading mgl-pax extends slime-edit-definitions ([M-.][cb15]) by adding mgl-pax-edit-definitions to slime-edit-definition-hooks. There are no related variables to customize.

  • For [Browsing Live Documentation][a595], mgl-pax-browser-function and mgl-pax-web-server-port can be customized in Elisp. To browse within Emacs, choose w3m-browse-url (see [w3m][7439]), and make sure both the w3m binary and the w3m Emacs package are installed. On Debian, simply install the w3m-el package. With other browser functions, a HUNCHENTOOT web server is started.

  • See [Transcribing with Emacs][f5bd] for how to use the transcription features. There are no related variables to customize.

Independently from the Common Lisp side, the Elisp functions mgl-pax-hideshow-documentation and mgl-pax-hideshow-comments help focus on the code only by folding or unfolding [MGL-PAX:DEFSECTION][72b4], [MGL-PAX:DEFINE-GLOSSARY-TERM][8ece] forms and long strings, or comments.

3.2 Installing from Quicklisp

If you installed PAX with Quicklisp, the location of mgl-pax.el may change with updates, and you may want to copy the current version of mgl-pax.el to a stable location by evaluating this in CL:

(mgl-pax:install-pax-elisp "~/quicklisp/")

If working from, say, a git checkout, there is no need for this step.

  • [function] INSTALL-PAX-ELISP TARGET-DIR

    Install mgl-pax.el distributed with this package in TARGET-DIR.

3.3 Loading PAX

Assuming the Elisp file is in the ~/quicklisp/ directory, add something like this to your .emacs:

(add-to-list 'load-path "~/quicklisp/")
(require 'mgl-pax)

If the Elisp variable mgl-pax-autoload is true (the default), then PAX will be loaded in the connected Lisp on-demand via [SLIME][6be7].

If loading fails, mgl-pax will be unloaded from Emacs and any [overridden Slime key bindings][cfab] restored.

3.4 Setting up Keys

The recommended key bindings are

(global-set-key (kbd "C-.") 'mgl-pax-document)
(global-set-key (kbd "s-x t") 'mgl-pax-transcribe-last-expression)
(global-set-key (kbd "s-x r") 'mgl-pax-retranscribe-region)

The global key bindings above are global because their commands work in any mode. If that's not desired, one may bind C-. locally in all Slime related modes like this:

(slime-bind-keys slime-parent-map nil '(("C-." mgl-pax-document)))

If the customizable variable mgl-pax-hijack-slime-doc-keys is true, then upon loading mgl-pax, the following changes are made to slime-doc-map (assuming it's bound to C-c C-d):

  • C-c C-d a: replaces slime-apropos with mgl-pax-apropos

  • C-c C-d z: replaces slime-apropos-all with mgl-pax-apropos-all

  • C-c C-d p: replaces slime-apropos-package with mgl-pax-apropos-package

  • C-c C-d f: replaces slime-describe-function with mgl-pax-document

  • C-c C-d d: replaces slime-describe-symbol with mgl-pax-hideshow-documentation

  • C-c C-d c: installs mgl-pax-hideshow-comments

  • C-c C-d u: installs mgl-pax-edit-parent-section

Calling mgl-pax-unhijack-slime-doc-keys reverts these changes.

4 Background

As a user, I frequently run into documentation that's incomplete and out of date, so I tend to stay in the editor and explore the code by jumping around with [SLIME][6be7]'s [M-.][cb15] (slime-edit-definition). As a library author, I spend a great deal of time polishing code but precious little writing documentation.

In fact, I rarely write anything more comprehensive than docstrings for exported stuff. Writing docstrings feels easier than writing a separate user manual, and they are always close at hand during development. The drawback of this style is that users of the library have to piece the big picture together themselves.

That's easy to solve, I thought, let's just put all the narrative that holds docstrings together in the code and be a bit like a Literate Programmer turned inside out. The original prototype, which did almost everything I wanted, was this:

(defmacro defsection (name docstring)
  `(defun ,name () ,docstring))

Armed with this DEFSECTION, I soon found myself organizing code following the flow of user-level documentation and relegated comments to implementation details entirely. However, some parts of DEFSECTION docstrings were just listings of all the functions, macros and variables related to the narrative, and this list was repeated in the [DEFPACKAGE][9b43] form complete with little comments that were like section names. A clear violation of [OAOO][7d18], one of them had to go, so DEFSECTION got a list of symbols to export.

That was great, but soon I found that the listing of symbols is ambiguous if, for example, a function, a compiler macro and a class were named by the same symbol. This did not concern exporting, of course, but it didn't help readability. Distractingly, on such symbols, M-. was popping up selection dialogs. There were two birds to kill, and the symbol got accompanied by a type, which was later generalized into the concept of locatives:

(defsection @introduction ()
  "A single line for one man ..."
  (foo class)
  (bar function))

After a bit of elisp hacking, M-. was smart enough to disambiguate based on the locative found in the vicinity of the symbol, and everything was good for a while.

Then, I realized that sections could refer to other sections if there were a [SECTION][672f] locative. Going down that path, I soon began to feel the urge to generate pretty documentation as all the necessary information was available in the DEFSECTION forms. The design constraint imposed on documentation generation was that following the typical style of upcasing symbols in docstrings, there should be no need to explicitly mark up links: if M-. works, then the documentation generator shall also be able to figure out what's being referred to.

I settled on [Markdown][a317] as a reasonably non-intrusive format, and a few thousand lines later PAX was born. Since then, locatives and references were factored out into the DRef library to let PAX focus on M-. and documentation.

5 Basics

Now let's examine the most important pieces.

  • [macro] DEFSECTION NAME (&KEY (PACKAGE '*PACKAGE*) (READTABLE '*READTABLE*) (EXPORT T) TITLE LINK-TITLE-TO (DISCARD-DOCUMENTATION-P *DISCARD-DOCUMENTATION-P*) CONCEPTS) &BODY ENTRIES

    Define a documentation section and maybe export referenced symbols. A bit behind the scenes, a global variable with NAME is defined and is bound to a [SECTION][5fac] object. By convention, section names start with the character @. See [Introduction][685e] for an example.

    Entries

    ENTRIES consists of docstrings and references in any order. Docstrings are arbitrary strings in Markdown format.

    References are XREFs given in the form (NAME LOCATIVE). For example, (FOO FUNCTION) refers to the function FOO, (@BAR SECTION) says that @BAR is a subsection of this one. (BAZ (METHOD (T T T))) refers to the default method of the three argument generic function BAZ. (FOO FUNCTION) is equivalent to (FOO (FUNCTION)). See the DRef Introduction for more.

    The same name may occur in multiple references, typically with different locatives, but this is not required.

    The references are not LOCATEd until documentation is generated, so they may refer to things yet to be defined.

    Exporting

    If EXPORT is true (the default), NAME and the [name][88cf]s of references among ENTRIES which are [SYMBOL][e5af]s are candidates for exporting. A candidate symbol is exported if

    • it is accessible in PACKAGE, and

    • there is a reference to it in the section being defined which is approved by [EXPORTABLE-REFERENCE-P][e51f].

    See [DEFINE-PACKAGE][63f3] if you use the export feature. The idea with conflating documentation and exporting is to force documentation of all exported symbols and to reduce duplication.

    Misc

    • TITLE is a string containing Markdown or NIL. If non-NIL, it determines the text of the heading in the generated output. LINK-TITLE-TO is a reference given as an (NAME LOCATIVE) pair or NIL, to which the heading will link when generating HTML. If not specified, the heading will link to its own anchor.

    • When DISCARD-DOCUMENTATION-P (defaults to [*DISCARD-DOCUMENTATION-P*][730f]) is true, ENTRIES will not be recorded to save memory.

    • CONCEPTS is a list of concept names and [concept key][b19d]s, for which this section is a primary source. See [Indexing Concepts][c001].

  • [variable] *DISCARD-DOCUMENTATION-P* NIL

    The default value of [DEFSECTION][72b4]'s DISCARD-DOCUMENTATION-P argument. One may want to set *DISCARD-DOCUMENTATION-P* to true before building a binary application.

  • [macro] DEFINE-PACKAGE PACKAGE &BODY OPTIONS

    This is like [CL:DEFPACKAGE][9b43] but silences warnings and errors signalled when the redefined package is at variance with the current state of the package. Typically this situation occurs when symbols are exported by calling EXPORT (as is the case with [DEFSECTION][72b4]) as opposed to adding :EXPORT forms to the DEFPACKAGE form and the package definition is subsequently reevaluated. See the section on package variance in the SBCL manual.

    The bottom line is that if you rely on DEFSECTION to do the exporting, then you'd better use DEFINE-PACKAGE.

  • [macro] DEFINE-GLOSSARY-TERM NAME (&KEY TITLE URL (DISCARD-DOCUMENTATION-P *DISCARD-DOCUMENTATION-P*) CONCEPTS) &BODY DOCSTRING

    Define a global variable with NAME, and set it to a [GLOSSARY-TERM][8251] object. TITLE, URL and DOCSTRING are Markdown strings or NIL. Glossary terms are DOCUMENTed in the lightweight bullet + locative + name/title style. See the glossary entry [name][88cf] for an example.

    • When a glossary term is linked to in documentation, its TITLE will be the link text instead of the name of the symbol (as with [SECTION][5fac]s).

    • Glossary terms with non-NIL URLs are like external links: they are linked to their URL in the generated documentation. These offer a more reliable alternative to using Markdown reference links and are usually not included in SECTIONs.

    • When DISCARD-DOCUMENTATION-P (defaults to [*DISCARD-DOCUMENTATION-P*][730f]) is true, DOCSTRING will not be recorded to save memory.

    • CONCEPTS is a list of concept names and [concept key][b19d]s, for which this term is a primary source. See [Indexing Concepts][c001].

    By [default][8894], glossary terms are indexed alongside concepts.

  • [macro] NOTE &BODY ARGS

    Define a note with an optional NAME and an optional DOCSTRING. The DOCSTRING of the note is its own docstring concatenated with docstrings of other notes in the lexical scope of BODY.

    ARGS has the form [NAME] [DOCSTRING] BODY, where the square brackets indicate optional arguments. See below for the details of parsing ARGS.

    Note: NOTE is experimental and, as such, subject to change.

    NOTE can occur in any evaluated position without changing its BODY's run-time behaviour or introducing any run-time overhead. Top level forms remain top level when wrapped in NOTE. The names of notes live in the same global namespace regardless of nesting or whether they are top level forms. These properties come at the price of NOTE being weird: it defines named notes at macro-expansion time (or load time). But the definitions are idempotent, so it's fine to macroexpand NOTE any number of times.

    Notes are similar to Lisp comments, but they can be included in the documentation. In fact, notes are auto-included: a Specific Link to a note is equivalent to including it with the [DOCSTRING][ce75] locative. Thus, NOTE can double as a tool to avoid boilerplate. An [Unspecific Link][8e71] to a note has no effect.

    Notes are intended to help reduce the distance between code and its documentation when there is no convenient definition docstring to use nearby.

    (note @xxx "We change the package."
  (in-package :mgl-pax))
==> #<PACKAGE "MGL-PAX">
(values (docstring (dref '@xxx 'note)))
=> "We change the package."

    Here is an example of how to overdo things:

    (note @1+*
  "This is a seriously overdone example."
  (defun 1+* (x)
    "[@1+* note][docstring]"
    (if (stringp x)
        (note (@1+*/1 :join #\Newline)
          "- If X is a STRING, then it is parsed as a REAL number."
          (let ((obj (read-from-string x)))
            (note "It is an error if X does not contain a REAL."
              (unless (realp obj)
                (assert nil)))
            (1+ obj)))
        (note "- Else, X is assumed to be a REAL number, and we simply
                 add 1 to it."
          (1+ x)))))

(1+* "7")
=> 8

(values (docstring (dref '@1+* 'note)))
=> "This is a seriously overdone example.

- If X is a STRING, then it is parsed as a REAL number.
It is an error if X does not contain a REAL.

- Else, X is assumed to be a REAL number, and we simply
add 1 to it."

    The parsing of ARGS:

    • If the first element of ARGS is not a string, then it is a NAME (a non-NIL [SYMBOL][e5af]) or name with options, currently destructured as (NAME &KEY JOIN). As in [DEFSECTION][72b4] and [DEFINE-GLOSSARY-TERM][8ece], the convention is that NAME starts with a @ character.

      JOIN is [PRINC][676d]ed before the docstring of a child note is output. Its default value is a string of two newline characters.

    • The next element of ARGS is a Markdown docstring. See [Markdown in Docstrings][7bf5].

    • The rest of ARGS is the BODY. If BODY is empty, then NIL is returned.

    Note that named and nameless notes can contain other named or nameless notes without restriction, but nameless notes without a lexically enclosing named note are just an [implicit progn][c2fd] with BODY, and their docstring is discarded.

    If NOTE occurs as a top level form, then its SOURCE-LOCATION is reliably recorded. Else, the quality of the source location varies, but it is at least within the right top level form on all implementations. On SBCL, exact source location is supported.

6 PAX Locatives

In addition to DRef's own, PAX defines a few locative types using the facilities described in Adding New Locatives. Locatives allow referenceing definitions, which is used in [DEFSECTION][72b4], Navigating Sources in Emacs and docstrings (see [Codification][f1ab] and Linking in the context of Generating Documentation).

  • [locative] SECTION

    • Direct locative supertypes: VARIABLE

    • Direct locative subtypes: DYNAMIC-SECTION

    Refers to a [SECTION][5fac] defined by [DEFSECTION][72b4].

    SECTION is [EXPORTABLE-LOCATIVE-TYPE-P][c930] but not exported by default (see [EXPORTABLE-REFERENCE-P][e51f]).

  • [locative] GLOSSARY-TERM

    • Direct locative supertypes: VARIABLE

    Refers to a [GLOSSARY-TERM][8251] defined by [DEFINE-GLOSSARY-TERM][8ece].

    GLOSSARY-TERM is [EXPORTABLE-LOCATIVE-TYPE-P][c930] but not exported by default (see [EXPORTABLE-REFERENCE-P][e51f]).

  • [locative] CONCEPT

    • Direct locative supertypes: VARIABLE

    Refers to a named group of [concept key][b19d]s defined by [DEFINE-CONCEPT][b80d] for multiplexing its keys to definitions that link to it in their docstrings. See [Indexing Concepts][c001] for more.

    CONCEPT is [EXPORTABLE-LOCATIVE-TYPE-P][c930] but not exported by default (see [EXPORTABLE-REFERENCE-P][e51f]).

    CONCEPTs have no ARGLIST or DOCSTRING.

  • [locative] NOTE

    Refers to named notes defined by the [NOTE][e2ae] macro.

    When Generating Documentation, if a single link would be made to a NOTE (be it either a Specific Link or an unambiguous [Unspecific Link][8e71]), then the NOTE's DOCSTRING is included as if with the [DOCSTRING][ce75] locative.

    NOTE is [EXPORTABLE-LOCATIVE-TYPE-P][c930] but not exported by default (see [EXPORTABLE-REFERENCE-P][e51f]).

  • [locative] DISLOCATED

    Refers to a symbol in a non-specific context. Useful for suppressing [Unspecific Autolink][e2a4]ing. For example, if there is a function called FOO then

      `FOO`

    will be linked (if [*DOCUMENT-LINK-CODE*][d9ee]) to its definition. However,

      [`FOO`][dislocated]

    will not be. With a dislocated locative, LOCATE always fails with a LOCATE-ERROR condition. Also see [Escaping Autolinking][a270].

    DISLOCATED references do not RESOLVE.

  • [locative] ARGUMENT

    An alias for [DISLOCATED][e391], allowing one to refer to an argument of a macro without accidentally linking to a class that has the same name as that argument. In the following example, FORMAT may link to CL:FORMAT (if we generated documentation for it):

    "See FORMAT in DOCUMENT."

    Since ARGUMENT is a locative, we can prevent that linking by writing:

    "See the FORMAT argument of DOCUMENT."

    ARGUMENT references do not RESOLVE.

  • [locative] INCLUDE SOURCE &KEY LINE-PREFIX HEADER FOOTER HEADER-NL FOOTER-NL

    This PSEUDO locative refers to a region of a file. SOURCE can be a [STRING][b93c] or a PATHNAME, in which case the whole file is being pointed to, or it can explicitly supply START, END locatives. INCLUDE is typically used to include non-lisp files in the documentation (say Markdown or Elisp as in the next example) or regions of Lisp source files. This can reduce clutter and duplication.

    (defsection @example-section ()
  (mgl-pax.el (include #.(asdf:system-relative-pathname
                          :mgl-pax "src/mgl-pax.el")
                       :header-nl "```elisp" :footer-nl "```"))
  (foo-example (include (:start (dref-ext:make-source-location function)
                         :end (dref-ext:source-location-p function))
                        :header-nl "```"
                        :footer-nl "```")))

    In the above example, when documentation is generated, the entire src/mgl-pax.el file is included in the Markdown output surrounded by the strings given as HEADER-NL and FOOTER-NL. The documentation of FOO-EXAMPLE will be the region of the file from the SOURCE-LOCATION of the START reference (inclusive) to the SOURCE-LOCATION of the END reference (exclusive). If only one of START and END is specified, then they default to the beginning and end of the file, respectively.

    Since START and END are literal references, pressing M-. on PAX.EL will open the src/mgl-pax.el file and put the cursor on its first character. M-. on FOO-EXAMPLE will go to the source location of the FOO function.

    With the LAMBDA locative, one can specify positions in arbitrary files.

    • SOURCE is either an absolute pathname designator or a list matching the [destructuring lambda list][6067] (&KEY START END), where START and END must be NIL or (<NAME> <LOCATIVE>) lists (not evaluated) like a [DEFSECTION][72b4] entry. Their SOURCE-LOCATIONs constitute the bounds of the region of the file to be included. Note that the file of the source location of START and END must be the same. If SOURCE is a pathname designator, then it must be absolute so that the locative is context independent.

    • If specified, LINE-PREFIX is a string that's prepended to each line included in the documentation. For example, a string of four spaces makes Markdown think it's a code block.

    • HEADER and FOOTER, if non-NIL, are printed before the included string.

    • HEADER-NL and FOOTER-NL, if non-NIL, are printed between two FRESH-LINE calls.

    INCLUDE is not [EXPORTABLE-LOCATIVE-TYPE-P][c930], and INCLUDE references do not RESOLVE.

  • [locative] DOCSTRING

    DOCSTRING is a PSEUDO locative for including the parse tree of the Markdown DOCSTRING of a definition in the parse tree of a docstring when generating documentation. It has no source location information and only works as an explicit link. This construct is intended to allow docstrings to live closer to their implementation, which typically involves a non-exported definition.

    (defun div2 (x)
  "X must be [even* type][docstring]."
  (/ x 2))

(deftype even* ()
  "an even integer"
  '(satisfies evenp))

(document #'div2)
.. - [function] DIV2 X
..
..     X must be an even integer.
..

    There is no way to LOCATE DOCSTRINGs, so nothing to RESOLVE either.

  • [locative] GO (NAME LOCATIVE)

    Redirect to a definition in the context of the reference designated by NAME and LOCATIVE. This PSEUDO locative is intended for things that have no explicit global definition.

    As an example, consider this part of a hypothetical documentation of CLOS:

      (defsection @clos ()
    (defmethod macro)
    (call-next-method (go (defmethod macro))))

    The GO reference exports the symbol [CALL-NEXT-METHOD][6832] and also produces a terse redirection message in the documentation.

    GO behaves as described below.

    • A GO reference RESOLVEs to what NAME with LOCATIVE resolves to:

      (resolve (dref 'xxx '(go (print function))))
==> #<FUNCTION PRINT>
=> T

    • The DOCSTRING of a GO reference is NIL.

    • SOURCE-LOCATION (thus M-.) returns the source location of the embedded reference:

      (equal (source-location (dref 'xxx '(go (print function))))
       (source-location (dref 'print 'function)))
=> T

  • [locative] CLHS &OPTIONAL NESTED-LOCATIVE

    Refers to definitions, glossary entries, sections, issues and issue summaries in the Common Lisp HyperSpec. These have no source location so [M-.][cb15] will not work. What works is linking in documentation, including [Browsing Live Documentation][a595]. The generated links are relative to [*DOCUMENT-HYPERSPEC-ROOT*][f585] and work even if [*DOCUMENT-LINK-TO-HYPERSPEC*][875e] is NIL. All matching is case-insensitive.

    • definitions: These are typically unnecessary as DOCUMENT will produce the same link for e.g. PPRINT, [PPRINT][function], or [PPRINT][] if *DOCUMENT-LINK-TO-HYPERSPEC* is non-NIL and the [PPRINT][6af6] function in the running Lisp is not [linkable][7eb5]. When [Browsing Live Documentation][a595], a slight difference is that everything is linkable, so using the CLHS link bypasses the page with the definition in the running Lisp.

      • unambiguous definition: [pprint][clhs] ([pprint][6af6])

      • disambiguation page: [function][clhs] ([function][aeb6])

      • specific: [function][(clhs class)] (function)

    • glossary terms:

      • [lambda list][(clhs glossary-term)] ([lambda list][98ff])

    • issues:

      • [ISSUE:AREF-1D][clhs] ([ISSUE:AREF-1D][63ef])

      • [ISSUE:AREF-1D][(clhs section)] ([ISSUE:AREF-1D][63ef])

    • issue summaries: These render as ([SUMMARY:CHARACTER-PROPOSAL:2-6-5][935f]):

      • [SUMMARY:CHARACTER-PROPOSAL:2-6-5][clhs]

      • [SUMMARY:CHARACTER-PROPOSAL:2-6-5][(clhs section)]

      Since these summary ids are not particularly reader friendly, the anchor text a [Specific Reflink with Text][fb17] may be used:

      • [see this][SUMMARY:CHARACTER-PROPOSAL:2-6-5 (clhs section)] ([see this][935f]).

    • sections:

      • by section number: [3.4][clhs] or [3.4][(clhs section)] ([3.4][e442])

      • by section title: With the locative (CLHS SECTION), substring matching against the title starting at word boundaries is performed. With the locative [CLHS][ed5f] (where SECTION is not specified explicitly), the name must match the title exactly. For example, [lambda list][(clhs section)] refers to the same definition as [lambda lists][clhs] ([Lambda Lists][e442]).

      • by filename: [03_d][clhs] or [03_d][(clhs section)] ([03_d][e442])

      • by alias

        • [Format directives][d273] are aliases of the sections describing them. Thus, [~c][clhs] is equivalent to [22.3.1.1][clhs] and [Tilde C: Character][clhs]. The full list is ~C [%][7bd6] ~& ~| [~~][65bc] [R][9927] ~D [B][6897] [O][76df] [X][f9fa] [F][cae2] ~E [G][76ab] [$][5b4d] [A][f275] [S][b39f] [W][e6d3] ~_ [<][ed9f] [:>][ed9f] ~I [/][db38] ~T ~< Justification [>][bf38] [*][bdd6] [[][76ea] []][945b] ~{ [}][886c] ~? [(][ac30] ~) [P][60e4] [;][d296] ~^ ~Newline.

        • Similarly, reader macro characters are aliases of the sections describing them. The full list is ( [)][e43c] ['][8e92] [;][56ba] ["][cd66] ` [,][826b] [#][698d] [#\][7b6f] [#'][8a5e] [#(][7a7f] [#*][adf2] [#:][ac5e] [#.][ffd7] [#B][c93e] [#O][68d2] #X #R [#C][bfaa] [#A][7163] [#S][dded] #P [#=][fa43] [##][dd03] [#+][d5e1] [#-][81b3] #| #< [#)][9478].

        • Finally, loop keywords have aliases to the sections describing them. For example, the strings loop:for, for and :for are aliases of CLHS 6.1.2.1. The loop:* aliases are convenient for completion at the prompt when [Browsing Live Documentation][a595], while the other aliases are for defaulting to buffer contents.

    As the above examples show, the NESTED-LOCATIVE argument of the [CLHS][ed5f] locative may be omitted. In that case, definitions, glossary terms, issues, issue summaries, and sections are considered in that order. Sections are considered last because a substring of a section title can be matched by chance easily.

    All examples so far used [Reflink][cbc4]s. [Autolink][ec7a]ing also works if the [name][88cf] is marked up as code or is [codified][f1ab] (e.g. in COS clhs ([COS][c4a3] clhs).

    As mentioned above, M-. does not do anything over CLHS references. Slightly more usefully, the [live documentation browser][a595] understands CLHS links so one can enter inputs like 3.4 clhs, "lambda list" clhs or error (clhs function).

    CLHS references do not RESOLVE.

7 Navigating Sources in Emacs

Integration into [SLIME][6be7]'s [M-.][cb15] (slime-edit-definition) allows one to visit the SOURCE-LOCATION of a definition. PAX extends standard Slime functionality by

  • adding support for all kinds of definitions (see e.g. ASDF:SYSTEM, READTABLE in Basic Locative Types), not just the ones Slime knows about,

  • providing a portable way to refer to even standard definitions,

  • disambiguating the definition based on buffer content, and

  • adding more powerful completions.

The definition is either determined from the buffer content at point or is prompted for. At the prompt, TAB-completion is available for both names and locatives. With a prefix argument (C-u M-.), the buffer contents are not consulted, and M-. always prompts.

The M-. extensions can be enabled by loading src/mgl-pax.el. See [Emacs Setup][8541]. In addition, the Elisp command mgl-pax-edit-parent-section visits the source location of the section containing the definition with point in it.

A close relative of M-. is C-. for [Browsing Live Documentation][a595].

7.1 M-. Defaulting

When [M-.][cb15] is invoked, it first tries to find a [name][88cf] in the current buffer at point. If no name is found, then it [prompts][ed46].

First, (slime-sexp-at-point) is taken as a [word][d7b0], from which the [name][88cf] will be parsed. Then, candidate locatives are looked for before and after the [word][d7b0]. Thus, if a locative is the previous or the next expression, then M-. will go straight to the definition which corresponds to the locative. If that fails, M-. will try to find the definitions in the normal way, which may involve popping up an xref buffer and letting the user interactively select one of possible definitions.

M-. works on parenthesized references, such as those in [DEFSECTION][72b4]:

(defsection @foo ()
  (cos function))

Here, when the cursor is on one of the characters of COS or just after COS, pressing M-. will visit the definition of the function [COS][c4a3].

To play nice with Generating Documentation, forms suitable for [Autolink][ec7a]ing are recognized:

function cos
cos function

... as well as [Reflink][cbc4]s:

[cos][function]
[see this][cos function]

... and [Markdown inline code][68c1]:

cos `function`
`cos` function
`cos` `function`

Everything works the same way in comments and docstrings as in code. In the next example, pressing M-. on RESOLVE* will visit its denoted method:

;;; See RESOLVE* (method (function-dref)) for how this all works.

7.2 M-. Prompting

7.2.1 M-. Minibuffer Syntax

At the minibuffer prompt, the definitions to edit can be specified as follows.

  • NAME: Refers to all DREF:DEFINITIONS of NAME with a Lisp locative type. See these NAME -> DEFINITIONS examples:

      print    ->  PRINT FUNCTION
  PRINT    ->  PRINT FUNCTION
  MGL-PAX  ->  "mgl-pax" ASDF:SYSTEM, "MGL-PAX" package
  pax      ->  "PAX" PACKAGE
  "PAX"    ->  "PAX" PACKAGE

    Note that depending on the Lisp implementation there may be more definitions. For example, SBCL has an UNKNOWN :DEFOPTIMIZER definition for PRINT.

  • NAME LOCATIVE: Refers to a single definition (as in (DREF:DREF NAME LOCATIVE)). Example inputs of this form:

      print function
  dref-ext:docstring* (method (t))

  • LOCATIVE NAME: This has the same form as the previous: two sexps, but here the first one is the locative. If ambiguous, this is considered in addition to the previous one. Example inputs:

      function print
  (method (t)) dref-ext:docstring*

In all of the above NAME is a [raw name][f5af], meaning that print will be recognized as PRINT and pax as "PAX".

The package in which symbols are read is the Elisp slime-current-package. In Lisp buffers, this is the buffer's package, else it's the package of the Slime repl buffer.

7.2.2 M-. Completion

When M-. prompts for the definition to edit, TAB-completion is available in the minibuffer for both names and locatives. To reduce clutter, string names are completed only if they are typed explicitly with an opening quotation mark, and they are case-sensitive. Examples:

  • pri<TAB> invokes the usual Slime completion.

  • print <TAB> (note the space) lists [FUNCTION][a51f] and (PAX:CLHS [FUNCTION][aeb6]) as locatives.

  • class dref:<TAB> lists DREF:XREF and DREF:DREF (all the classes in the package DREF).

  • pax:locative <TAB> lists all locative types (see the CL function DREF:LOCATIVE-TYPES).

  • package "MGL<TAB> lists the names of packages that start with "MGL".

  • package <TAB> lists the names of all packages as strings and also CLASS, MGL-PAX:LOCATIVE because PACKAGE denotes a class and also a locative.

For more powerful search, see [Apropos][b7fc].

8 Generating Documentation

8.1 The DOCUMENT Function

  • [function] DOCUMENT DOCUMENTABLE &KEY (STREAM T) PAGES (FORMAT :PLAIN)

    Write DOCUMENTABLE in FORMAT to STREAM diverting some output to PAGES. FORMAT is one of [:PLAIN][c879], [:MARKDOWN][dd29], :HTML, :PDF or [NIL][f7e6]. STREAM may be a [STREAM][d5a9] object, T or NIL as with [CL:FORMAT][ad78].

    To look up the documentation of the DOCUMENT function itself:

      (document #'document)

    The same with fancy markup:

      (document #'document :format :markdown)

    To document a [SECTION][5fac]:

      (document pax::@pax-manual)

    To generate the documentation for separate libraries with automatic cross-links:

      (document (list pax::@pax-manual dref::@dref-manual) :format :markdown)

    See Utilities for Generating Documentation for more.

    Definitions that do not define a first-class object are supported via DRef:

      (document (dref:locate 'foo 'type))

    There are quite a few special variables that affect how output is generated. See [Codification][f1ab], [Linking to the HyperSpec][7cc3], Linking to Sections, [Link Format][c0d2] and [Output Formats][8d9b].

    For the details, see the following sections, starting with DOCUMENTABLE. Also see [Writing Extensions][c4ce] and [DOCUMENT-OBJECT*][8269].

8.1.1 DOCUMENTABLE

The DOCUMENTABLE argument of DOCUMENT may be a single object (e.g. #'PRINT), a definition such as (DREF 'PRINT 'FUNCTION), a string, or a nested list of these. More precisely, DOCUMENTABLE is one of the following:

  • single definition designator: A DREF or anything else that is LOCATEable. This includes non-DREF XREFs and first-class objects such as FUNCTIONs. The generated documentation typically includes the definition's DOCSTRING. See [Markdown Output][dd29] for more.

  • docstring: A string, in which case it is processed like a docstring in [DEFSECTION][72b4]. That is, with [docstring sanitization][7bf5], [Codification][f1ab], and Linking.

  • list of documentables: A nested list of LOCATEable objects and docstrings. The objects in it are documented in depth-first order. The structure of the list is otherwise unimportant.

  • [variable] *DOCUMENT-TIGHT* NIL

    Normally, DOCUMENT calls [ENSURE-MD-PARAGRAPH][a021] before and after documenting each atomic DOCUMENTABLE. If *DOCUMENT-TIGHT* is true, then this doesn't happen for top-level string documentables.

8.1.2 Return Values

If PAGES are NIL, then DOCUMENT - like [CL:FORMAT][ad78] - returns a string (when [STREAM][d5a9] is NIL) else NIL.

If PAGES, then a list of output designators are returned, one for each non-empty page (to which some output has been written), which are determined as follows.

  • The string itself if the output was to a string.

  • The stream if the output was to a stream.

  • The pathname of the file if the output was to a file.

If the default page given by the STREAM argument of DOCUMENT was written to, then its output designator is the first element of the returned list. The rest of the designators correspond to the non-empty pages in the PAGES argument of DOCUMENT in that order.

8.1.3 PAGES

The PAGES argument of DOCUMENT is to create multi-page documents by routing some of the generated output to files, strings or streams. PAGES is a list of page specs. A page spec is a [property list][b18e] with keys :OBJECTS, :OUTPUT, :URI-FRAGMENT, :SOURCE-URI-FN, :HEADER-FN and :FOOTER-FN.

PAGES may look something like this:

`((;; The section about SECTIONs and everything below it ...
   :objects (, @sections)
   ;; ... is so boring that it's not worth the disk space, so
   ;; send it to a string.
   :output (nil)
   ;; Explicitly tell other pages not to link to these guys.
   :uri-fragment nil)
  ;; Send the @EXTENSION-API section and everything reachable
  ;; from it ...
  (:objects (, @extension-api)
   ;; ... to build/tmp/pax-extension-api.html.
   :output "build/tmp/pax-extension-api.html"
   ;; However, on the web server html files will be at this
   ;; location relative to some common root, so override the
   ;; default:
   :uri-fragment "doc/dev/pax-extension-api.html"
   ;; Set html page title, stylesheet, charset.
   :header-fn 'write-html-header
   ;; Just close the body.
   :footer-fn 'write-html-footer)
  ;; Catch references that were not reachable from the above. It
  ;; is important for this page spec to be last.
  (:objects (, @pax-manual)
   :output "build/tmp/manual.html"
   ;; Links from the extension api page to the manual page will
   ;; be to ../user/pax-manual#<anchor>, while links going to
   ;; the opposite direction will be to
   ;; ../dev/pax-extension-api.html#<anchor>.
   :uri-fragment "doc/user/pax-manual.html"
   :header-fn 'write-html-header
   :footer-fn 'write-html-footer))

Documentation is initially sent to a default stream (the STREAM argument of DOCUMENT), but output is temporary redirected to :OUTPUT for the duration of generating documentation for a docstring or definition that matches :OBJECTS, shadowing any existing redirection if any. This nesting typically happens when a [SECTION][5fac] references another definition and they both have their own page specs.

:OBJECTS is a list of objects whose definitions are LOCATEable. In addition, docstrings can be included. The latter can be useful if DOCUMENTABLE includes a docstring.

  • A docstring matches :OBJECTS if it is [EQ][5a82] to one of its elements.

  • A definition matches :OBJECTS if it is XREF= to one of its DREF elements.

If multiple page specs match, then the first one has precedence.

  • :OUTPUT can be a number of things:

    • If it's NIL, then output will be collected in a string.

    • If it's T, then output will be sent to [*STANDARD-OUTPUT*][e7ee].

    • If it's a stream, then output will be sent to that stream.

    • If it's a list whose first element is a string or a pathname, then output will be sent to the file denoted by that and the rest of the elements of the list are passed on to [CL:OPEN][6547]. One extra keyword argument is :ENSURE-DIRECTORIES-EXIST. If it's true, [ENSURE-DIRECTORIES-EXIST][876d] will be called on the pathname before it's opened.

    Note that even if PAGES is specified, STREAM acts as a catch all, absorbing the generated documentation for references not claimed by any pages.

  • :HEADER-FN, if not NIL, is a function of a single stream argument, which is called just before the first write to the page. Since :FORMAT :HTML only generates HTML fragments, this makes it possible to print arbitrary headers, typically setting the title, CSS stylesheet, or charset.

  • :FOOTER-FN is similar to :HEADER-FN, but it's called after the last write to the page. For HTML, it typically just closes the body.

  • :URI-FRAGMENT is a string such as "doc/manual.html" that specifies where the page will be deployed on a webserver. It defines how links between pages will look. If it's not specified and :OUTPUT refers to a file, then it defaults to the name of the file. If :URI-FRAGMENT is NIL, then no links will be made to or from that page.

  • :SOURCE-URI-FN is a function of a single, DREF argument. If it returns a value other than NIL, then it must be a string representing an URI. This affects [*DOCUMENT-MARK-UP-SIGNATURES*][8fb6] and [*DOCUMENT-FANCY-HTML-NAVIGATION*][6ab0]. Also see [MAKE-GIT-SOURCE-URI-FN][587f].

8.1.4 Package and Readtable

While generating documentation, symbols may be read from docstrings and printed. Our goal in general is to use the [*PACKAGE*][5ed1] and [*READTABLE*][b79a] in effect at the time the docstring was [READ][fe58]. This keeps the correspondence between

  • [M-.][cb15] and Linking, and

  • interned symbols and [Codification][f1ab].

What values of *PACKAGE* and *READTABLE* are used is determined separately for each definition being documented. For a [SECTION][5fac], its [SECTION-PACKAGE][a5b1] and [SECTION-READTABLE][88a7] are in effect.

For non-SECTION definitions, the package and the readtable are given by the following rules. If multiple rules provide a non-NIL package, then the first such rule takes precedence and similarly for the readtable.

  • DOCSTRING: The second value returned by DOCSTRING for the definition provides the package.

    • If the value of *PACKAGE* in effect at the time of definition was captured (e.g. by DEFINE-LOCATIVE-TYPE), then DOCSTRING returns that.

    • The user may provide defaults for the package. See DREF-EXT:DOCSTRING* (especially the package-wide default).

  • Home section: If the definition has a [HOME-SECTION][fda4], then the home section's SECTION-PACKAGE and SECTION-READTABLE are used.

  • Arglist heuristic: If the definition has an argument list, then the package of the first argument that's not external in any package is used.

  • Name heuristic: If the definition is named by a symbol, then its [SYMBOL-PACKAGE][e5ab] is used, and *READTABLE* is set to the standard readtable (NAMED-READTABLES:FIND-READTABLE :COMMON-LISP).

  • Default: *PACKAGE* is set to the CL-USER package and *READTABLE* to the standard readtable.

The values thus determined come into effect after the name itself is printed, for printing of the arglist and the docstring.

CL-USER> (pax:document #'foo)
- [function] FOO <!> X Y &KEY (ERRORP T)

    Do something with X and Y.

In the above, the <!> marks the place where *PACKAGE* and *READTABLE* are bound.

  • [function] HOME-SECTION OBJECT

    The home section of an object is a [SECTION][5fac] that contains the object's definition in its [SECTION-ENTRIES][d850] or NIL. In the overwhelming majority of cases there should be at most one containing section.

    If there are multiple containing sections, the following apply.

    • If the name of the definition is a non-keyword symbol, only those containing sections are considered whose package is closest to the [SYMBOL-PACKAGE][e5ab] of the name, where closest is defined as having the longest common prefix between the two [PACKAGE-NAME][db68]s.

    • If there are multiple sections with equally long matches or the name is not a non-keyword symbol, then it's undefined which one is the home section.

    For example, (MGL-PAX:DOCUMENT FUNCTION) is an entry in the MGL-PAX::@BASICS section. Unless another section that contains it is defined in the MGL-PAX package, the home section is guaranteed to be MGL-PAX::@BASICS because the SYMBOL-PACKAGEs of MGL-PAX:DOCUMENT and MGL-PAX::@BASICS are the same (hence their common prefix is maximally long).

    This scheme would also work, for example, if the home package of DOCUMENT were MGL-PAX/IMPL, and it were reexported from MGL-PAX because the only way to externally change the home package would be to define a containing section in a package like MGL-PAX/IMP.

    Thus, relying on the package system makes it possible to find the intended home section of a definition among multiple containing sections with high probability. However, for names which are not symbols, there is no package system to take advantage of.

  • [variable] *DOCUMENT-NORMALIZE-PACKAGES* T

    Whether to print [in package <package-name>] in the documentation when the package changes.

8.2 Browsing Live Documentation

Documentation for definitions in the running Lisp can be browsed directly without generating documentation in the offline manner. HTML documentation, complete with [Codification][f1ab] and Linking, is generated from docstrings of all kinds of definitions and PAX [SECTION][5fac]s in the running Lisp on the fly. This allows ad-hoc exploration of the Lisp, much like describe-function, apropos-command and other online help commands in Emacs, for which direct parallels are provided.

Still, even without Emacs and [SLIME][6be7], limited functionality is available through the [PAX Live Home Page][9d50] by starting the live documentation web server [manually][72cc].

If [Emacs Setup][8541] has been done, the Elisp function mgl-pax-document (maybe bound to C-.) generates and displays documentation as a single HTML page. If necessary, a disambiguation page is generated with the documentation of all matching definitions. For example, to view the documentation of this very SECTION, one can do:

M-x mgl-pax-document
View Documentation of: pax::@browsing-live-documentation

Alternatively, pressing C-. with point over the text pax::@browsing-live-documentation in a buffer achieves the same effect.

In interactive use, mgl-pax-document behaves similarly to M-. except:

  • It shows the DOCUMENTation of some definition and does not visit its SOURCE-LOCATION.

  • It considers definitions with all LOCATIVE-TYPES not just LISP-LOCATIVE-TYPES because it doesn't need SOURCE-LOCATION.

    This also means that completion works for [CLHS][ed5f] definitions:

    • "lambda list<TAB> lists "lambda list" and "lambda list keywords", both HyperSpec glossary entries. This is similar to common-lisp-hyperspec-glossary-term in Elisp but also works for HyperSpec section titles.

    • "#<TAB> lists all sharpsign reader macros (similar to common-lisp-hyperspec-lookup-reader-macro in Elisp).

    • "~<TAB> lists all [CL:FORMAT][ad78] directives (similar to common-lisp-hyperspec-format in Elisp).

    • "loop:~<TAB> lists all [loop keywords][e016].

  • It works in non-lisp-mode buffers by reinterpreting a few lines of text surrounding point as lisp code (hence the suggested global binding) in package MGL-PAX.

  • It supports fragment syntax at the prompt:

      <name> <locative> <fragment-name> <fragment-locative>

    This is like <NAME> <LOCATIVE>, but the browser scrolls to the definition of <FRAGMENT-NAME> <FRAGMENT-LOCATIVE> within that page.

    For example, entering this at the prompt will generate the entire PAX manual as a single page and scroll to the very section you are reading within it:

      pax::@pax-manual pax:section pax::@browsing-live-documentation pax:section

  • If the empty string is entered at the prompt, and there is no existing w3m buffer or w3m is not used, then [PAX Live Home Page][9d50] is visited. If there is a w3m buffer, then entering the empty string displays that buffer.

The convenience function mgl-pax-current-definition-toggle-view (C-c C-d c) documents the definition with point in it.

  • [variable] *BROWSE-CONTEXT*

    When set to a DOCUMENTABLE, live browsing will link only to definitions reachable from it as opposed to linking to all definitions in the running Lisp when it is unbound. This makes it easier to see how offline generated documentation will turn out.

    For convenience, setting *BROWSE-CONTEXT* to :PAX-WORLD is equivalent to listing all [registered][f4fd] sections.

8.2.1 Browsing with w3m

When the value of the Elisp variable mgl-pax-browser-function is w3m-browse-url (see [Emacs Setup][8541]), the Emacs w3m browser is used without the need for a web server, and also offering somewhat tighter integration than [Browsing with Other Browsers][c434].

With w3m's default key bindings, moving the cursor between links involves TAB and S-TAB (or <up> and <down>). RET and <right> follow a link, while B and <left> go back in history.

In addition, the following PAX-specific key bindings are available:

  • M-. visits the source location of the definition corresponding to the link under the point.

  • Invoking mgl-pax-document on a section title link will show the documentation of that section on its own page.

  • n moves to the next PAX definition on the page.

  • p moves to the previous PAX definition on the page.

  • u follows the first Up: link (to the first containing [SECTION][5fac]) if any.

  • U is like u but positions the cursor at the top of the page.

  • v visits the source location of the current definition (the one under the cursor or the first one above it).

  • V visits the source location of the first definition on the page.

8.2.2 Browsing with Other Browsers

When the value of the Elisp variable mgl-pax-browser-function is not w3m-browse-url (see [Emacs Setup][8541]), requests are served via a web server started in the running Lisp, and documentation is most likely displayed in a separate browser window.

By default, mgl-pax-browser-function is nil, which makes PAX use browse-url-browser-function. You may want to customize the related browse-url-new-window-flag or, for Chrome, set browse-url-chrome-arguments to ("--new-window").

By default, mgl-pax-web-server-port is nil, and PAX will pick a free port automatically.

In the browser, clicking on the locative on the left of the name (e.g. in - [function] PRINT) will raise and focus the Emacs window (if Emacs is not in text mode, and also subject to window manager focus stealing settings), then go to the corresponding source location. For sections, clicking on the lambda link will do the same (see [*DOCUMENT-FANCY-HTML-NAVIGATION*][6ab0]).

Finally, note that the URLs exposed by the web server are subject to change, and even the port used may vary by session if the Elisp variable mgl-pax-web-server-port is nil.

  • [variable] *BROWSE-HTML-STYLE* :CHARTER

    The HTML style to use for browsing live documentation. Affects only non-w3m browsers. See [*DOCUMENT-HTML-DEFAULT-STYLE*][90fa] for the possible values.

    If you change this variable, you may need to do a hard refresh in the browser (often C-<f5>).

8.2.3 Apropos

The Elisp functions mgl-pax-apropos, mgl-pax-apropos-all, and mgl-pax-apropos-package can display the results of DREF-APROPOS in the [live documentation browser][a595]. These extend the functionality of slime-apropos, slime-apropos-all and slime-apropos-package to support more kinds of definitions in an extensible way. The correspondence is so close that the PAX versions might [take over the Slime key bindings][8541].

Note that apropos functionality is also exposed via the [PAX Live Home Page][9d50].

More concretely, the PAX versions supports the following extensions:

  • Definitions with string names. One can search for ASDF:SYSTEMs, PACKAGEs and [CLHS][ed5f] sections, glossary entries, format directives, reader macro characters, loop keywords.

  • Exact or substring matching of the name and the package.

  • Matching only symbol or string names.

On the [PAX Live Home Page][9d50], one may [Browse by Locative Types][659d], which gives access to some of the apropos functionality via the browser without involving Emacs.

On the result page:

  • A DREF-APROPOS form to reproduce the results at the REPL is shown.

  • One may toggle the EXTERNAL-ONLY and CASE-SENSITIVE boolean arguments.

  • One may switch between list, and detailed view. The list view only shows the first, [bulleted line][dd29] for each definition, while the detailed view includes the full documentation of definitions with the exception of [SECTION][5fac]s.

  • The returned references are presented in two groups: those with non-symbol and those with symbol [name][88cf]s. The non-symbol group is sorted by locative type then by name. The symbol group is sorted by name then by locative type.

With mgl-pax-apropos-all and mgl-pax-apropos-package being simple convenience functions on top of mgl-pax-apropos, we only discuss the latter in detail here. For the others, see the Elisp docstrings.

The STRING Argument of mgl-pax-apropos

The STRING argument consists of a name pattern and a DTYPE.

The name pattern has the following forms.

  • :print matches definitions whose names are the string print or a symbol with SYMBOL-NAME print. Vertical bar form as in :|prInt| is also also supported and is useful in when CASE-SENSITIVE is true.

  • "print" matches definitions whose names contain print as a substring.

  • print is like the previous, substring matching case. Use this form to save typing if the pattern does not contain spaces and does not start with a colon.

  • The empty string matches everything.

After the name pattern, STRING may contain a DTYPE that the definitions must match.

  • print t matches definitions with LISP-LOCATIVE-TYPES, which is the default (equivalent to print).

  • print function matches functions whose names contain print (e.g. [CL:PRINT][d451] and [CL:PPRINT][6af6]).

  • :print function is like the previous example but with exact name match (so it matches CL:PRINT but not CL:PPRINT).

  • print variable matches for example [*PRINT-ESCAPE*][ff76].

  • print (or variable function) matches all variables and functions with print in their names.

  • array (or type (not class)) matches [DEFTYPE][7f9a]s and but not CLASSes with the string array in their names.

  •  pax:section (note the leading space) matches all PAX sections (EXTERNAL-ONLY NIL is necessary to see many of them).

  • print dref:pseudo matches definitions with PSEUDO-LOCATIVE-TYPES such as MGL-PAX:CLHS.

  • print dref:top matches definitions with all locative types (LOCATIVE-TYPES).

The PACKAGE Argument of mgl-pax-apropos

When mgl-pax-apropos is invoked with a prefix argument, it prompts for a package pattern among other things. The pattern may be like the following examples.

  • :none restricts matches to non-symbol names.

  • :any restricts matches to symbol names.

  • :cl restricts matches to symbols in the CL package.

  • :|X Y| is similar to the previous, but the vertical bar syntax allows for spaces in names.

  • mgl restricts matches to packages whose name contains mgl as a substring.

  • "x y" is the same as the previous, but the explicit quotes allow for spaces in names.

The above examples assume case-insensitive matching.

8.2.4 PAX Live Home Page

When [Browsing Live Documentation][a595], the home page provides quick access to documentation of the definitions in the system. In Emacs, when mgl-pax-document is invoked with the empty string, it visits the home page.

The home page may also be accessed directly by going to the root page of the web server (if one is started). Here, unless the home page is viewed [with w3m][83d5], one may directly look up documentation and access [Apropos][b7fc] via the input boxes provided.

  • [function] ENSURE-WEB-SERVER &KEY PORT HYPERSPEC-ROOT

    Start or update a web server on PORT of localhost for [Browsing Live Documentation][a595]. Returns the base URL of the server (e.g. http://localhost:32790), which goes to the [PAX Live Home Page][9d50]. If the web server is running already (ENSURE-WEB-SERVER) simply returns its base URL.

    Note that even when using Emacs but [Browsing with Other Browsers][c434], the web server is started automatically. When [Browsing with w3m][83d5], no web server is involved at all. Calling this function explicitly is only needed if the Emacs integration is not used, or to override PORT and HYPERSPEC-ROOT.

    • If PORT is NIL or 0, then the server will use any free port.

    • If there is a server already running and PORT is not NIL or 0, then the server is restarted on PORT.

    • If HYPERSPEC-ROOT is NIL, the HyperSpec pages will be served from any previously provided HYPERSPEC-ROOT or, failing that, from [*DOCUMENT-HYPERSPEC-ROOT*][f585].

    • If HYPERSPEC-ROOT is non-NIL, then pages in the HyperSpec will be served from HYPERSPEC-ROOT. The following command changes the root without affecting the server in any other way:

        (ensure-web-server :hyperspec-root "/usr/share/doc/hyperspec/")

Top-level PAX Sections

The [PAX Live Home Page][9d50] lists the top-level PAX sections: those that have no other [SECTION][5fac]s referencing them (see [HOME-SECTION][fda4]) and those registered in PAX World.

The [PAX Live Home Page][9d50] lists all ASDF:SYSTEMs and PACKAGEs in the Lisp. For easier overview, they are grouped based on their SOURCE-LOCATIONs. Two systems are in the same group if the directory of one (i.e. the directory of the .asd file in which it was defined) is the same or is below the other's.

A PACKAGE is presented under a group of systems if the SOURCE-LOCATION of the package is below the top-most directory among the systems in the group.

Systemless Packages

The [PAX Live Home Page][9d50] lists PACKAGEs [unrelated][b033] to any ASDF:SYSTEM as systemless.

Browse by Locative Types

The [PAX Live Home Page][9d50] provides quick links to [Apropos][b7fc] result pages for all Basic Locative Types which may have definitions.

  • [glossary-term] related

    Two definitions are related if the directory of one's SOURCE-LOCATIONs contains the directory of the other's.

8.3 Markdown Support

[Markdown][a317] in docstrings and titles is processed with the 3BMD library.

8.3.1 Markdown in Docstrings

  • Docstrings can be indented in any of the usual styles. PAX normalizes indentation by stripping the longest run of leading spaces common to all non-blank lines except the first. Thus, the following two docstrings are equivalent:

      (defun foo ()
    "This is
    indented
    differently")
  
  (defun foo ()
    "This is
  indented
  differently")

  • When [Browsing Live Documentation][a595], the page displayed can be of, say, a single function within what would constitute the offline documentation of a library. Because Markdown reference link definitions, for example

      [Daring Fireball]: http://daringfireball.net/

    can be defined anywhere, they wouldn't be resolvable in that case, their use is discouraged. Currently, only reflink definitions within the documentation of the same definition are guaranteed to be resolvable. This is left intentionally vague because the specifics are subject to change.

    See [DEFINE-GLOSSARY-TERM][8ece] for a better alternative to Markdown reference links.

Docstrings of definitions that do not have a [HOME-SECTION][fda4] and are not PAX constructs themselves (e.g [SECTION][5fac], [GLOSSARY-TERM][8251], [NOTE][e2ae]) are assumed to have been written with no knowledge of PAX and to conform to Markdown only by accident. These docstrings are thus sanitized more aggressively.

  • Indentation of what looks like blocks of Lisp code is rounded up to a multiple of 4. More precisely, non-zero indented lines between blank lines or the docstring boundaries are reindented if the first non-space character of the first line is an ( or a ; character.

  • Special HTML characters <& are escaped.

  • Furthermore, to reduce the chance of inadvertently introducing a Markdown heading, if a line starts with a string of # characters, then the first one is automatically escaped. Thus, the following two docstrings are equivalent:

      The characters #\Space, #\Tab and
  #\Return are in the whitespace group.

  The characters #\Space, #\Tab and
  \#\Return are in the whitespace group.

8.3.2 Markdown in Titles

  • [glossary-term] title

    A title is a [STRING][68cc] associated with a definition (e.g. with the TITLE argument of [DEFSECTION][72b4] or [DEFINE-GLOSSARY-TERM][8ece]). Titles are accessible via [DOCTITLE][e619] and processed according to Markdown in Titles.

Titles undergo [Codification][f1ab] and may be a single paragraph containing explicit [Markdown inline code][68c1], [Markdown emphasis][c624], [Markdown image][d534]s, inline [MathJax][a17d] and HTML entities (e.g. &quot;). Other kinds of Markdown markup and block elements are not allowed.

  • [function] DOCTITLE OBJECT

    Return the title of OBJECT if it has one or NIL. For [Codification][f1ab], the title is interpreted in the package returned by DOCSTRING. DOCTITLE can be extended via [DOCTITLE*][c34e].

8.3.3 Syntax Highlighting

For syntax highlighting, GitHub's fenced code blocks Markdown extension to mark up code blocks with triple backticks is enabled so all you need to do is write:

```elisp
(defun foo ())
```

to get syntactically marked up HTML output. Copy src/style.css from PAX and you are set. The language tag, elisp in this example, is optional and defaults to common-lisp.

See the documentation of 3BMD and Colorize for the details.

8.3.4 MathJax

Displaying pretty mathematics in TeX format is supported via MathJax.

  • Inline

    It can be done inline (within a paragraph):

      Pretty, eh? $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$ Yes.

    which is displayed as

    Pretty, eh? $\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$ Yes.

    To avoid rendering between $5 and $6 with inline math, both the opening and the closing $ character must be followed / preceded by a non-space character. This agrees with Pandoc.

    Alternatively, the $`x_0`$ syntax may be used (renders as $x_0$), which has no restriction on spacing.

  • Block

    The $$ is supported as a block element:

      Pretty, eh?
  
  $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$
  
  Yes.

    which will be rendered in its own paragraph:

    Pretty, eh?

    $$\int_0^\infty e^{-x^2} dx=\frac{\sqrt{\pi}}{2}$$

    Yes.

    $$ is also legal to use inline, but it's not recommended as it gets rendered inline on GitHub but as display math in PDF Output.

MathJax will leave inline code (e.g. those between single backticks) and code blocks (triple backticks) alone. Outside code, use <span>$</span> to scare MathJax off.

Escaping all those backslashes in TeX fragments embedded in Lisp strings can be a pain. [Pythonic String Reader][d3fc5] can help with that.

8.4 Codification

  • [variable] *DOCUMENT-UPPERCASE-IS-CODE* T

    When true, [interesting][7445] [name][88cf]s extracted from [codifiable][b89a] [word][d7b0]s are marked up as code with backticks. For example, this docstring

      "T PRINT CLASSes SECTION *PACKAGE* MGL-PAX ASDF
  CaMeL Capital"

    is equivalent to this:

      "`T` `PRINT` `CLASS`es `SECTION` `*PACKAGE*` `MGL-PAX` `ASDF`
  CaMel Capital"

    and renders as

    T [PRINT][d451] CLASSes [SECTION][5fac] MGL-PAX ASDF CaMel Capital

    where the links are added due to [*DOCUMENT-LINK-CODE*][d9ee].

    To suppress codification, add a backslash to the beginning of the a [codifiable][b89a] word or right after the leading * if it would otherwise be parsed as Markdown emphasis:

      "\\SECTION *\\PACKAGE*"

    The number of backslashes is doubled above because that's how the example looks in a docstring. Note that the backslash is discarded even if *DOCUMENT-UPPERCASE-IS-CODE* is false.

  • [glossary-term] codifiable

    A [word][d7b0] is codifiable if

    • it has a single uppercase character (e.g. it's T) and no lowercase characters at all, or

    • there is more than one uppercase character and no lowercase characters between them (e.g. CLASSes, unREADable, CLASS-NAMEs but not Classes or aTe).

  • [glossary-term] interesting

    A [name][88cf] is interesting if

    • it names a symbol external to its package, or

    • it is at least 3 characters long and names an interned symbol, or

    • it names a [Local Definition][9db9].

    See [Package and Readtable][ab7e].

  • [variable] *DOCUMENT-DOWNCASE-UPPERCASE-CODE* NIL

    If true, then all [Markdown inline code][68c1] (e.g. `code`, which renders as code) – including [Codification][f1ab] – which has no lowercase characters is downcased in the output. Characters of literal strings in the code may be of any case. If this variable is :ONLY-IN-MARKUP and the output format does not support markup (e.g. it's :PLAIN), then no downcasing is performed. For example,

      `(PRINT "Hello")`

    is downcased to

      `(print "Hello")`

    because it only contains uppercase characters outside the string. However,

      `MiXed "RESULTS"`

    is not altered because it has lowercase characters.

    If the first two characters are backslashes, then no downcasing is performed, in addition to [Escaping Autolinking][a270]. Use this to mark inline code that's not Lisp.

      Press `\\M-.` in Emacs.

8.5 Linking

PAX supports linking to definitions either with explicit [Reflink][cbc4]s or with [Autolink][ec7a]s.

When generating offline documentation, only the definitions in DOCUMENTABLE may be [linkable][7eb5], but when [Browsing Live Documentation][a595], everything is linkable as documentation is generated on-demand.

Many examples in this section link to standard Common Lisp definitions. In the offline case, these will link to [external URLs][f585], while in the live case to disambiguation pages that list the definition in the running Lisp and in the HyperSpec.

Invoking M-. on WORD or NAME in any of the following examples will disambiguate based on the textual context, determining the locative. This is because navigation and linking use the same Parsing algorithm, although linking is a bit more strict about trimming, depluralization, and it performs [Filtering Links][b2e4]. On the other hand, M-. cannot visit the [CLHS][ed5f] references because there are no associated source locations.

  • [glossary-term] stable printed locative

    The [Link Format][c0d2] relies on definitions having a unique textual representation that doesn't change. More concretely, if [PRIN1][6384] under WITH-STANDARD-IO-SYNTAX but with [*PRINT-READABLY*][8aca] NIL produces the same unique string deterministically, then linking to definitions works even with non-readable locatives. The uniqueness condition requires that if two definitions are different under XREF=, then their textual representations are also different.

    On the other hand, for example, a method involving an EQL specializer with an object printed with [PRINT-UNREADABLE-OBJECT][9439] :IDENTITY T does not produce a stable string and links will break.

The [Markdown reference link][8c00] syntax [label][id] is repurposed for linking to definitions. In the following, we discuss the various forms of reflinks.

Format: [ [WORD][d7b0] ][ LOCATIVE ]

The first [name][88cf] in WORD (with depluralization) that forms a valid DREF with LOCATIVE is determined, and that definition is linked to. If there is no such DREF, then an [UNRESOLVABLE-REFLINK][64be] warning is signalled.

Examples:

  • [`EQL`][type] renders as [EQL][5fd4].

  • [EQL][type] renders as [EQL][5fd4].

The Markdown link definition (i.e. [type][7c9f] above) needs no backticks to mark it as code, but here and below, the second example relies on [*DOCUMENT-UPPERCASE-IS-CODE*][f25f] being true.

Specific Reflink with Text

Format: [LINK TEXT][ [NAME][88cf] LOCATIVE ]

If NAME and LOCATIVE form a valid DREF, then that definition is linked to with link text LINK TEXT. If there is no such DREF, then an [UNRESOLVABLE-REFLINK][64be] warning is signalled.

In this form, if NAME starts with #\", then it's read as a string, else as a symbol.

Examples:

  • [see this][output-label restart] renders as [see this][65e5].

  • [see this]["MGL-PAX" package] renders as see this.

Format: [ [WORD][d7b0] ][]

The first [name][88cf] in WORD (with depluralization, symbols only) that has some DEFINITIONS is determined, and those definitions are linked to. If no [name][88cf] with any definition is found, then an [UNRESOLVABLE-REFLINK][64be] warning is signalled.

Examples:

  • single link: [PRINT][] renders as [PRINT][d451].

  • multiple links: [OUTPUT-LABEL][] renders as OUTPUT-LABEL([0][c818] [1][65e5]).

  • no definitions: [BAD-NAME][] renders as BAD-NAME.

Unspecific Reflink with Text

Format: [LINK TEXT][ [NAME][88cf] ]

The DEFINITIONS of NAME are determined, and those definitions are linked to. If NAME has no definitions, then an [UNRESOLVABLE-REFLINK][64be] warning is signalled.

Examples:

  • [see this][print] renders as [see this][d451].

  • [see this][output-label] renders as see this([0][c818] [1][65e5]).

Format: [label][id]

This is a normal [Markdown reference link][8c00] if id is not a valid locative.

  • [see this][user-defined] renders unchanged.

    (dref:dref 'user-defined 'locative)
.. debugger invoked on LOCATE-ERROR:
..   Could not locate USER-DEFINED LOCATIVE.
..   USER-DEFINED is not a valid locative type or locative alias.

    (document "[see this][user-defined]" :format :markdown)
.. [see this][user-defined]

Use URLs with [DEFINE-GLOSSARY-TERM][8ece] as a better alternative to Markdown reference links (see [Markdown in Docstrings][7bf5]).

  • [condition] UNRESOLVABLE-REFLINK [WARNING][bcb6] CONDITION-CONTEXT-MIXIN

    When DOCUMENT encounters a [Reflink][cbc4] that looks like a PAX construct but has no matching definition, it signals an UNRESOLVABLE-REFLINK warning with the [OUTPUT-REFLINK][ff9c] and OUTPUT-LABEL([0][c818] [1][65e5]) restarts available.

    If the warning is not handled, then it is printed to [*ERROR-OUTPUT*][66c6], and it behaves as if [OUTPUT-LABEL][65e5] was invoked.

  • [restart] OUTPUT-REFLINK

    In the context of an [UNRESOLVABLE-REFLINK][64be], print no warning, and leave the Markdown link unchanged. [MUFFLE-WARNING][9dac] is equivalent to OUTPUT-REFLINK.

  • [restart] OUTPUT-LABEL

    In the context of an [UNRESOLVABLE-REFLINK][64be], print no warning, and replace the Markdown link by its label. For example, [NONEXISTENT][function] becomes NONEXISTENT.

  • [function] OUTPUT-REFLINK &OPTIONAL CONDITION

    Invoke the [OUTPUT-REFLINK][ff9c] restart.

  • [function] OUTPUT-LABEL &OPTIONAL CONDITION

    Invoke the [OUTPUT-LABEL][65e5] restart.

[Markdown inline code][68c1] automatically links to the corresponding definitions without having to use [Reflink][cbc4]s. This works especially well in conjunction with [Codification][f1ab]. The following examples assume that [*DOCUMENT-UPPERCASE-IS-CODE*][f25f] is true. If that's not the case, explicit backticks are required on [WORD][d7b0] (but not on LOCATIVE).

Format: [WORD][d7b0] LOCATIVE or LOCATIVE [WORD][d7b0]

The first [name][88cf] in WORD (with depluralization) that forms a valid DREF with LOCATIVE is determined, and that definition is linked to. If no such name is found, then [Unspecific Autolink][e2a4] is attempted.

Examples:

  • PRINT function renders as [PRINT][d451] function.

  • type EQL renders as type [EQL][5fd4].

  • type EQL function renders as type [EQL][db03] function.

If LOCATIVE has spaces, then it needs to be marked up as code, too. For example,

DREF-NAME `(reader dref)`

renders as DREF-NAME (reader dref).

Format: [WORD][d7b0]

The first [name][88cf] in WORD (with depluralization, symbols only) that has some DEFINITIONS is determined, and those definitions are linked to. If no such name is found or the autolink to this name is suppressed (see below), then WORD is left unchanged. If a locative is found before or after WORD, then Specific Autolink is tried first.

Examples:

  • PRINT renders as [PRINT][d451].

  • EQL renders as EQL.

[Unspecific Autolink][e2a4]ing is suppressed if the name found has a [Local Definition][9db9] or was linked to before in the same docstring:

  • "My other CAR is also a CAR" renders as "My other [CAR][d5a2] is also a CAR".

  • "[COS][] and COS" renders as "[COS][c4a3] and COS".

  • "[EQL][type] and EQL" renders as "[EQL][5fd4] and EQL".

  • "EQ and the EQ function" renders as "[EQ][5a82] and the [EQ][5a82] function".

[Unspecific Autolink][e2a4]ing to T and NIL is also suppressed (see [*DOCUMENT-LINK-TO-HYPERSPEC*][875e]):

  • "T and NIL" renders as "T and NIL".

As an exception, a single link (be it either a Specific Link or an unambiguous [Unspecific Link][8e71]) to a [SECTION][5fac], [GLOSSARY-TERM][8251] or [NOTE][e2ae] is not suppressed to allow their titles to be displayed or their docstring to be included properly:

  • "@NAME and @NAME" renders as "[name][88cf] and [name][88cf]".

Escaping Autolinking

In the common case, when [*DOCUMENT-UPPERCASE-IS-CODE*][f25f] is true, prefixing an uppercase [word][d7b0] with a backslash prevents it from being codified and thus also prevents [Autolink][ec7a]ing from kicking in. For example,

\DOCUMENT

renders as DOCUMENT. If it should be marked up as code but not autolinked, the backslash must be within backticks like this:

`\DOCUMENT`

This renders as DOCUMENT. Alternatively, the [DISLOCATED][e391] or the [ARGUMENT][8710] locative may be used as in [DOCUMENT][dislocated].

8.5.3 Linking to the HyperSpec

  • [variable] *DOCUMENT-LINK-TO-HYPERSPEC* T

    If true, consider definitions found in the Common Lisp HyperSpec for linking. For example,

    • PRINT renders as [PRINT][d451].

    In offline documentation, this would be a link to the hyperspec unless #'PRINT in the running Lisp is DOCUMENTABLE.

    When [Browsing Live Documentation][a595], everything is [linkable][7eb5], so the generated link will go to a disambiguation page that lists the definition in the Lisp and in the HyperSpec.

    Locatives work as expected (see [*DOCUMENT-LINK-CODE*][d9ee]): FIND-IF links to [FIND-IF][5884], FUNCTION links to [FUNCTION][a51f], and [FUNCTION][type] links to FUNCTION.

    [Unspecific Autolink][e2a4]ing to T and NIL is suppressed. If desired, use [Reflink][cbc4]s such as [T][] (that links to T) or [T][constant] (that links to [T][fe21]).

    Note that linking explicitly with the [CLHS][ed5f] locative is not subject to the value of this variable, so

    • [PRINT][clhs] always renders as [PRINT][d451], and

    • PRINT clhs always renders as [PRINT][d451] clhs.

    See also the filtering of [Unspecific Link][8e71]s.

  • [variable] *DOCUMENT-HYPERSPEC-ROOT* "http://www.lispworks.com/documentation/HyperSpec/"

    A URL to the Common Lisp HyperSpec. The URL may not have a scheme, in which case file is assumed.

    The default value is the canonical location. When [invoked from Emacs][a595], the Elisp variable common-lisp-hyperspec-root is in effect.

8.5.4 Linking to Sections

The following variables control how to generate section numbering, table of contents and navigation links.

  • [variable] *DOCUMENT-LINK-SECTIONS* T

    When true, HTML anchors and PDF destinations are generated before the headings (e.g. of sections), which allows the table of contents to contain links and also code-like references to sections (like @FOO-MANUAL) to be translated to links with the [TITLE][72b4] being the link text.

  • [variable] *DOCUMENT-MAX-NUMBERING-LEVEL* 3

    A non-negative integer. In their hierarchy, sections on levels less than this value get numbered in the format of 3.1.2. Setting it to 0 turns numbering off.

  • [variable] *DOCUMENT-MAX-TABLE-OF-CONTENTS-LEVEL* 3

    An integer that determines the depth of the table of contents.

    • If negative, then no table of contents is generated.

    • If non-negative, and there are multiple top-level sections on a page, then they are listed at the top of the page.

    • If positive, then for each top-level section a table of contents is printed after its heading, which includes a nested tree of section titles whose depth is limited by this value.

    If *DOCUMENT-LINK-SECTIONS* is true, then the tables will link to the sections.

  • [variable] *DOCUMENT-TEXT-NAVIGATION* NIL

    If true, then before each heading a line is printed with links to the previous, parent and next section. Needs *DOCUMENT-LINK-SECTIONS* to be on to work.

  • [variable] *DOCUMENT-FANCY-HTML-NAVIGATION* T

    If true and the output format is HTML, then headings get a navigation component that consists of links to the previous, parent, next section, a self-link, and a link to the definition in the source code if available (see :SOURCE-URI-FN in DOCUMENT). This component is normally hidden, it is visible only when the mouse is over the heading. Has no effect if *DOCUMENT-LINK-SECTIONS* is false.

  • [variable] *DOCUMENT-LINK-CODE* T

    Whether definitions of things other than [SECTION][5fac]s are allowed to be [linkable][7eb5].

  • [glossary-term] linkable

    When a reference is encountered to definition D while processing documentation for some page C, we say that definition D is linkable (from C) if

    • D denotes a [SECTION][5fac] and *DOCUMENT-LINK-SECTIONS* is true, or

    • D does not denote a SECTION and [*DOCUMENT-LINK-CODE*][d9ee] is true

    ... and

    • We are [Browsing Live Documentation][a595], or

    • D is an external definition ([CLHS][ed5f] or denotes a [GLOSSARY-TERM][8251] with a [URL][8ece]), or

    • D's page is C, or

    • D's page is relativizable to C.

    In the above, D's page is the last of the pages in the DOCUMENTABLE to which D's documentation is written (see :OBJECTS in [PAGES][9c7d]), and we say that a page is relativizable to another if it is possible to construct a relative link between their :URI-FRAGMENTs.

Specific links are those [Reflink][cbc4]s and [Autolink][ec7a]s that have a single locative and therefore at most a single matching definition. These are [Specific Reflink][f7a5], [Specific Reflink with Text][fb17] and Specific Autolink.

A specific link to a [linkable][7eb5] definition produces a link in the output. If the definition is not linkable, then the output will contain only what would otherwise be the link text.

Unspecific links are those [Reflink][cbc4]s and [Autolink][ec7a]s that do not specify a locative and match all definitions with a name. These are Unspecific Reflink, Unspecific Reflink with Text and [Unspecific Autolink][e2a4].

To make the links predictable and manageable in number, the following steps are taken.

  1. Definitions that are not symbol-based (i.e. whose DREF-NAME is not a symbol) are filtered out to prevent unrelated PACKAGEs, ASDF:SYSTEMs and [CLHS][ed5f] sections from cluttering the documentation without the control provided by importing symbols.

  2. Links to the CLHS are filtered out when the corresponding non-CLHS definition is being documented (and thus among the links).

  3. All references with LOCATIVE-TYPE LOCATIVE are filtered out.

  4. Non-[linkable][7eb5] definitions are removed.

  5. If the definitions include a GENERIC-FUNCTION, then all definitions with LOCATIVE-TYPE METHOD, ACCESSOR, READER and WRITER are removed to avoid linking to a possibly large number of methods.

If at most a single definition remains, then the output is the same as with a Specific Link.

If multiple definitions remain, then the link text is output followed by a number of numbered links, one to each definition. When [Browsing Live Documentation][a595], ambiguities can be much more frequent; and thus a single disambiguation page is linked to instead (but see [*BROWSE-CONTEXT*][805d]).

The following variables control various aspects of links and URLs.

  • [variable] *DOCUMENT-URL-VERSIONS* (2 1)

    A list of versions of PAX URL formats to support in the generated documentation. The first in the list is used to generate links.

    PAX emits HTML anchors before the documentation of [SECTION][5fac]s (see Linking to Sections) and other things (see Linking). For the function FOO, in the current version (version 2), the anchor is <a id="MGL-PAX:FOO%20FUNCTION">, and its URL will end with #MGL-PAX:FOO%20FUNCTION.

    Note: To make the URL independent of whether a symbol is internal or external to their [SYMBOL-PACKAGE][e5ab], single colon is printed where a double colon would be expected. Package and symbol names are both printed verbatim except for escaping colons and spaces with a backslash. For exported symbols with no funny characters, this coincides with how [PRIN1][6384] would print the symbol, while having the benefit of making the URL independent of the Lisp printer's escaping strategy and producing human-readable output for mixed-case symbols. No such promises are made for non-ASCII characters, and their URLs may change in future versions. Locatives are printed with PRIN1.

    Version 1 is based on the more strict HTML4 standard and the id of FOO is "x-28MGL-PAX-3A-3AFOO-20FUNCTION-29". This is supported by GitHub-flavoured Markdown. Version 2 has minimal clutter and is obviously preferred. However, in order not to break external links, by default, both anchors are generated.

    Let's understand the generated Markdown.

    (defun foo (x))

(document #'foo :format :markdown)
=> ("<a id=\"x-28MGL-PAX-3AFOO-20FUNCTION-29\"></a>
<a id=\"MGL-PAX:FOO%20FUNCTION\"></a>

- [function] **FOO** *X*
")

(let ((*document-url-versions* '(1)))
  (document #'foo :format :markdown))
=> ("<a id=\"x-28MGL-PAX-3AFOO-20FUNCTION-29\"></a>

- [function] **FOO** *X*
")

  • [variable] *DOCUMENT-MIN-LINK-HASH-LENGTH* 4

    Recall that [Markdown reference link][8c00]s (like [label][id]) are used for Linking. It is desirable to have ids that are short to maintain legibility of the generated Markdown, but also stable to reduce the spurious diffs in the generated documentation, which can be a pain in a version control system.

    Clearly, there is a tradeoff here. This variable controls how many characters of the MD5 sum of the full link id (the reference as a string) are retained. If collisions are found due to the low number of characters, then the length of the hash of the colliding reference is increased.

    This variable has no effect on the HTML generated from Markdown, but it can make Markdown output more readable.

  • [variable] *DOCUMENT-BASE-URL* NIL

    When *DOCUMENT-BASE-URL* is non-NIL, this is prepended to all Markdown relative URLs. It must be a valid URL without query or fragment parts. For example,

      http://lisp.org/doc/

    but not either of the following:

      http://lisp.org/doc?a=1
  http://lisp.org/doc#fragment.

    Note that intra-page links using only URL fragments (e.g. and explicit HTML links (e.g. <a href="...">) in Markdown are not affected.

8.6 Local Definition

While documentation is generated for a definition, that definition is considered local. Other local definitions may also be established. Local definitions inform [Codification][f1ab] through [interesting][7445] [name][88cf]s and affect [Unspecific Autolink][e2a4]ing.

(defun foo (x)
  "FOO adds one to X."
  (1+ x)

In this example, while the docstring of FOO is being processed, the global definition (DREF 'FOO 'FUNCTION) is also considered local, which suppresses linking FOO in the FOO's docstring back to its definition. If FOO has other definitions, [Unspecific Autolink][e2a4]ing to those is also suppressed.

Furthermore, the purely local definition (DREF 'X 'ARGUMENT) is established, causing the argument name X to be [codified][f1ab] because X is now [interesting][7445].

See [DOCUMENTING-DEFINITION][a412] and WITH-DISLOCATED-NAMES in [Extending DOCUMENT][574a].

8.7 Overview of Escaping

Let's recap how escaping [Codification][f1ab], [downcasing][a5ee], and Linking works.

  • One backslash in front of a [word][d7b0] turns codification off. Use this to prevent codification words such as DOCUMENT, which is all uppercase hence [codifiable][b89a], and it names an exported symbol hence it is [interesting][7445].

  • One backslash right after an opening backtick turns autolinking off.

  • Two backslashes right after an opening backtick turns autolinking and downcasing off. Use this for things that are not Lisp code but which need to be in a monospace font.

In the following examples, capital C/D/A letters mark the presence, and a/b/c the absence of codification, downcasing, and autolinking assuming all these features are enabled by [*DOCUMENT-UPPERCASE-IS-CODE*][f25f], [*DOCUMENT-DOWNCASE-UPPERCASE-CODE*][a5ee], and [*DOCUMENT-LINK-CODE*][d9ee].

DOCUMENT                => [`document`][1234]    (CDA)
\DOCUMENT               => DOCUMENT              (cda)
`\DOCUMENT`             => `document`            (CDa)
`\\DOCUMENT`            => `DOCUMENT`            (Cda)
[DOCUMENT][]            => [`document`][1234]    (CDA)
[\DOCUMENT][]           => [DOCUMENT][1234]      (cdA)
[`\DOCUMENT`][]         => [`document`][1234]    (CDA) *
[`\\DOCUMENT`][]        => [`DOCUMENT`][1234]    (CdA)
[DOCUMENT][dislocated]  => `document`            (CDa)

Note that in the example marked with *, the single backslash, that would normally turn autolinking off is ignored because it is in an explicit link.

8.8 Indexing

PAX can create indices that cross-reference definitions and [concept key][b19d]s with where they are mentioned in the documentation.

  • [glossary-term] referrer

    If the documentation of a definition links to another definition, the former is considered a referrer to the latter. See [referent][ad8e].

  • [glossary-term] referent

    A definition is considered a referent of another definition if the latter's documentation links to it. See [referrer][e7e0].

8.8.1 Indexing Definitions

PAX generates indices with entries like this:

foo [my-lib] (fn)
  ↩ d: My Lib Utilities
  ↩ f: bar, baz

MY-LIB:FOO here is a function whose documentation has been linked to by the My Lib Utilities [SECTION][5fac] as well as the BAR and BAZ functions (grouped together under f). For example, BAR could be defined like this:

(defun bar (x)
  "Extends FOO."
  (1+ (foo x)))

In this example, MY-LIB:FOO is the [referent][ad8e] and the others are the [referrer][e7e0]s. If there are no referrers or they are in the same group, then a single line is printed. Here is an example for the latter:

foo [my-lib] (fn) ↩ f: bar, baz

8.8.2 Indexing Concepts

Let's first [DEFINE-CONCEPT][b80d]s:

(define-concept @albert-einstein (:title "Albert Einstein"
                                  :keys ("Einstein, Albert")))

(define-concept @einstein-summation (:title "Einstein summation"
                                     :keys ("Einstein summation")))

CONCEPTs are definitions, but in the generated documentation, links to them are [hidden][56b9], and they are not meant to be documented themselves. They exist to endow their [referrer][e7e0]s with [concept key][b19d]s:

(defun einsum (x)
  "Implement @EINSTEIN-SUMMATION by @ALBERT-EINSTEIN."
  x)

This makes EINSUM a [referrer][e7e0] to the concept @EINSTEIN-SUMMATION, and thus will be indexed under the concept keys of @EINSTEIN-SUMMATION.

[DEFSECTION][72b4] and [DEFINE-GLOSSARY-TERM][8ece] can associate concepts with the section or glossary term being defined:

(defsection @famous-people (:title "Famous People"
                            :concepts (@albert-einstein))
  "Let's start with one of the most famous, Albert Einstein.")

This is like the mention in EINSUM, but it also makes the section a primary source of information. These are typeset more prominently in the generated documentation. If the section only mentions Einstein in passing, then don't use the :CONCEPTS argument, just mention @ALBERT-EINSTEIN in a section docstring as EINSUM does.

Now, let's define a section to hold them together, configure a minimal index, and see what comes out:

(defsection @test (:title "Test" :export nil)
  (@famous-people section)
  (einsum function))

(defun print-concept-index (documentable)
  (let* ((*document-index-formats* '(:plain))
         (*document-indices* '((:concepts t :title "Concept Index")))
         (output (document documentable :stream nil)))
    (princ (subseq output (search "## Concept Index" output)))))

(print-concept-index @test)
.. ## Concept Index
..
.. - Einstein summation ↩ f: EINSUM
..
.. - Einstein, Albert
..
..     - ↩ d: Famous People
..
..     - ↩ f: EINSUM
..

This is okay, but let's group summation and Albert under Einstein by breaking the string keys into lists of strings:

(define-concept @albert-einstein (:title "Albert Einstein"
                                  :keys (("Einstein" "Albert"))))

(define-concept @einstein-summation (:title "Einstein summation"
                                     :keys (("Einstein" "summation"))))

(print-concept-index @test)
.. ## Concept Index
..
.. - Einstein
..
..     - Albert
..
..         - ↩ d: Famous People
..
..         - ↩ f: EINSUM
..
..     - summation ↩ f: EINSUM
..

See [DEFINE-CONCEPT][b80d] about pure multiplexer concepts (those without :TITLEs) and how concepts can inherit the keys of other concepts.

Also, see [concept subkey][5920] on how to order concepts in the output.

  • [macro] DEFINE-CONCEPT NAME (&KEY TITLE KEYS)

    Define a CONCEPT for [Indexing Concepts][c001]. TITLE is a [STRING][68cc] or NIL, while KEYS is a list of [concept key][b19d]s and [SYMBOL][e5af]s (naming other concepts, see below).

    When Generating Documentation, if a link would be made to a CONCEPT, then:

    • The concept's :KEYS are recorded as [referent][ad8e]s with the definition being documented as the [referrer][e7e0].

      SYMBOLs in :KEYS are resolved to the concepts they name, and they are substituted with their keys recursively (circularities are not allowed, but there are no checks for them).

    • In the docstring being processed, the text denoting the name of the concept is replaced by the concept's :TITLE or the empty string if the title is NIL. In an ambiguous [Unspecific Link][8e71], the links to concepts are simply removed.

    By convention, names of TITLEd concepts start with a #@ character, to indicate that these names are part of the sentence flow, as their titles are substituted for their mentions. Names of unTITLEd concepts start with a #~ character to indicate that they are pure index key multiplexers, and do not appear in the generated documentation.

    See also [DEFINE-GLOSSARY-TERM][8ece].

  • [glossary-term] concept key

    A concept key identifies an entry in an index. It is a [concept subkey][5920] or a list of [concept subkey][5920]s. Concept keys occur in [DEFINE-CONCEPT][b80d], [DEFSECTION][72b4] and [DEFINE-GLOSSARY-TERM][8ece].

    • Single subkey: For example, the key "Einstein, Albert" consists of a single subkey. A single subkey is equivalent to a one-element list.

    • List of subkeys: When Einstein summation is also referenced, we can instead use the lists ("Einstein," "Albert") and ("Einstein," "summation") to ensure that their [referrer][e7e0]s are grouped together under "Einstein,".

      List concept keys can be arbitrarily long, although the PDF Output imposes a limit of 9.

  • [glossary-term] concept subkey

    A concept subkey (a part of a [concept key][b19d]) can be a string or a cons of strings. In the latter form, the [CAR][d5a2] is for printing, and the [CDR][e012] is for sorting. For example, use ("Éclair" . "Eclair") to have "Éclair" printed but sorted as if it were "Eclair". Sorting is done with plain STRING<, which is not Unicode-aware.

    Note that print names are ESCAPE-MARKDOWNed.

8.8.3 Configuring Indices

  • [variable] *DOCUMENT-INDEX-SECTIONS* :HOMELESS-DOCUMENTABLE

    Controls what sections can get an index (if [*DOCUMENT-INDEX-FORMATS*][ba76] also allows it and they are non-empty).

    • NIL: No indices are generated.

    • :DOCUMENTABLE: Sections that appear in DOCUMENTABLE (at any level) can get indices. Their subsections cannot.

    • :HOMELESS-DOCUMENTABLE: Sections that appear in DOCUMENTABLE and have no [HOME-SECTION][fda4] can get indices.

  • [variable] *DOCUMENT-INDEX-FORMATS* (:HTML :PDF)

    The list of [Output Formats][8d9b] for which index generation is allowed.

  • [variable] *DOCUMENT-INDICES* "<value omitted>"

    A nested list of index specifications. Each spec is of the form

      (&key title documentation document-referrer-groups
        dtype concepts children)

    Non-CONCEPT [referent][ad8e]s recorded during documentation generation are assigned to the first (in depth-first order) spec with a matching DTYPE. For [referent][ad8e]s that are concepts, their keys are assigned to the first spec with :CONCEPTS true.

    Specs are processed in the following way and order:

    • If a spec and all its :CHILDREN are empty (nothing was assigned to them above), then they are not output.

    • Else, a dynamically generated [SECTION][5fac] with TITLE (a Markdown string) is entered.

    • If DOCUMENTATION is non-NIL, then it is a Markdown docstring and is written as is to the output.

    • If DOCUMENT-REFERRER-GROUPS is true, then the ABBREVs and DOCUMENTATIONs in [*DOCUMENT-INDEX-REFERRER-GROUPS*][85c9] are listed in the output.

    • The [referent][ad8e]s assigned to this spec are output along with their [referrer][e7e0]s (see the examples in [Indexing Concepts][c001]).

    • The CHILDREN specs are processed.

    • The dynamically generated section is closed.

    The default value specifies separate indices for

    • functions and macros

    • variables

    • types

    • other things

    • CONCEPTs and [GLOSSARY-TERM][8251]s.

    They are all grouped under an "Indices" section, that has :DOCUMENT-REFERRER-GROUPS.

  • [variable] *DOCUMENT-INDEX-REFERENT-LOCATIVE-TYPE-ABBREVS* ((GENERIC-FUNCTION "_(gf)_") #'"_(fn)_" (VARIABLE "_(var)_") (ASDF/SYSTEM:SYSTEM "_(asdf:system)_"))

    A list of (LOCATIVE-TYPE ABBREV) elements, where ABBREV (a Markdown string) is printed instead of the [referent][ad8e]'s locative when its DREF-LOCATIVE-TYPE is LOCATIVE-TYPE.

    The default behaviour is to bind [*PACKAGE*][5ed1] to the package of the symbol naming the definition (if it's a symbol) and print the locative with *PRINT-CASE* :DOWNCASE, omitting the package of the locative type.

  • [variable] *DOCUMENT-INDEX-REFERRER-GROUPS* "<value omitted>"

    A list of (DTYPE ABBREV &KEY DOCUMENTATION) elements that defines the grouping of [referrer][e7e0]s of similar types in referrer lists. For example, the default value of this variable has this element:

      ((or function macro compiler-macro method) "↩ _f_:"
     :documentation "_f_: for definitions in the function namespace ...")

    Thus, in the following example, the definitions of BAR and BAZ are grouped together under ↩ f: because they are both DTYPEP (OR FUNCTION MACRO COMPILER-MACRO METHOD):

      foo [my-lib] (fn)
    ↩ d: My Lib Utilities
    ↩ f: bar, baz

    • In general, the first matching DTYPE wins. See DTYPEs.

    • ABBREV may be a Markdown string or a cons of strings to control sort order in the manner of [concept subkey][5920]s.

    • DOCUMENTATION is a Markdown string for :DOCUMENT-REFERRER-GROUPS in [*DOCUMENT-INDICES*][8894].

8.9 Output Formats

  • [variable] *DOCUMENT-MARK-UP-SIGNATURES* T

    When true, some things such as function names and arglists are rendered as bold and italic. In HTML Output and PDF Output, locative types become links to sources (if :SOURCE-URI-FN is provided, see [PAGES][9c7d]), and the symbol becomes a self-link for your permalinking pleasure.

    For example, a reference is rendered in Markdown roughly as:

      - [function] foo x y

    With this option on, the above becomes:

      - [function] **foo** *x y*

    Also, in HTML **foo** will be a link to that very entry and [function] may turn into a link to sources.

8.9.1 Plain Output

This is the default :FORMAT of DOCUMENT, intended to be a replacement for [CL:DOCUMENTATION][c5ae]. :PLAIN (short for plain text) is very similar to :MARKDOWN, but most of the markup that would make reading in, say, the REPL unpleasant is removed.

  • Markup for [Markdown emphasis][c624], [Markdown inline code][68c1], [Markdown reference link][8c00]s and fenced code blocks is stripped from the output.

  • No link anchors are emitted.

  • No [section numbering][f12d].

  • No [table of contents][d4e7].

8.9.2 Markdown Output

By default, DREFs are documented in the following format.

- [<locative-type>] <name> <arglist>

    <docstring>

The line with the bullet is printed with [DOCUMENTING-DEFINITION][a412]. The docstring is processed with [DOCUMENT-DOCSTRING][7f1f] while [Local Definition][9db9]s established with WITH-DISLOCATED-NAMES are in effect for all variables locally bound in a definition with ARGLIST, and [*PACKAGE*][5ed1] is bound to the second return value of DOCSTRING.

With this default format, PAX supports all locative types, but for some Basic Locative Types defined in DRef and the PAX Locatives, special provisions have been made.

  • For definitions with a VARIABLE or CONSTANT locative, their initform is printed as their arglist. The initform is the INITFORM argument of the locative if provided, or the global symbol value of their name. If no INITFORM is provided, and the symbol is globally unbound, then no arglist is printed.

When the printed initform is too long, it is truncated.

  • Depending on what the SETF locative refers to, the ARGLIST of the setf expander, [setf function][99b0], or the method signature is printed as with the METHOD locative.

  • For definitions with a METHOD locative, the arglist printed is the method signature, which consists of the locative's QUALIFIERS and SPECIALIZERS appended.

  • For definitions with an ACCESSOR, READER or WRITER locative, the class on which they are specialized is printed as their arglist.

  • For definitions with a STRUCTURE-ACCESSOR locative, the arglist printed is the locative's CLASS-NAME argument if provided.

  • For definitions with a CLASS locative, the arglist printed is the list of [public superclasses][ff58] with [STANDARD-OBJECT][a843] and [CONDITION][83e1] omitted.

  • For definitions with a STRUCTURE locative, the arglist printed is the list of [public superclasses][ff58] with STRUCTURE-OBJECT omitted.

  • For definitions with a CONDITION locative, the arglist printed is the list of [public superclasses][ff58] with STANDARD-OBJECT and CONDITION omitted.

  • For definitions with a ASDF:SYSTEM locative, their most important slots are printed as an unnumbered list.

  • For definitions with the LOCATIVE locative type, their LOCATIVE-TYPE-DIRECT-SUPERS and LOCATIVE-TYPE-DIRECT-SUBS are printed.

  • When documentation is being generated for a definition with the [SECTION][672f] locative, a new (sub)section is opened (see [WITH-HEADING][80e8]), within which documentation for its [SECTION-ENTRIES][d850] is generated, separated from each other by [ENSURE-MD-PARAGRAPH][a021].

  • For definitions with a GLOSSARY-TERM locative, no arglist is printed, and if non-NIL, [GLOSSARY-TERM-TITLE][af78] is printed as name.

  • For definitions with a [CONCEPT][56b9] locative, no documentation is generated.

  • For definitions with a [GO][58f6] locative, its LOCATIVE-ARGS are printed as its arglist, along with a redirection message.

  • See the [INCLUDE][5cd7] locative.

  • For definitions with a [CLHS][ed5f] locative, the LOCATIVE-ARGS are printed as the arglist. For CLHS [SECTION][7a4e]s, the title is included in the arglist.

  • For definitions with an UNKNOWN locative, the LOCATIVE-ARGS are printed as the arglist. There is no docstring.

  • [glossary-term] public superclasses

    The public superclasses of a class are the tightest envelope of superclasses with names exported from some package. This envelope is constructed by recursing depth-first into the superclass hierarchy. If the name of the superclass currently processed is exported from any package, then it is collected as a public superclass, and we do not recurse into its superclasses.

8.9.3 PDF Output

When invoked with :FORMAT :PDF, DOCUMENT generates [Markdown Output][dd29] and converts it to PDF with [Pandoc][59d9], which in turn uses LaTeX. Make sure that they are installed.

(with-open-file (s "x.pdf" :direction :output :if-exists :supersede
                           :if-does-not-exist :create
                           :element-type '(unsigned-byte 8))
  (pax:document "Hello, World!" :stream s :format :pdf))

To see how the output looks like, visit pax-manual-v0.4.1.pdf and dref-manual-v0.4.1.pdf.

PDF output is similar to [*DOCUMENT-HTML-DEFAULT-STYLE*][90fa] :CHARTER without the off-white tint and with coloured instead of underlined links. The latter is because underlining interferes with hyphenation in LaTeX. As in HTML output, locative types link to the respective definition in the sources on GitHub (see [MAKE-GIT-SOURCE-URI-FN][587f]).

Note that linking from one PDF to another is currently not supported due to the lack of consistent support in PDF viewers. Therefore, such links are replaced by their label or the title if any (e.g. of a [SECTION][5fac] or [GLOSSARY-TERM][8251]).

The generation of Markdown is subject to the standard variables (again see DOCUMENT). The Markdown to PDF conversion can be customized with the following variables.

  • [variable] *DOCUMENT-PANDOC-PROGRAM* "pandoc"

    The name of the Pandoc binary. It need not be an absolute pathname as PATH is searched.

  • [variable] *DOCUMENT-PANDOC-PDF-OPTIONS* (("-V" "papersize=a4") ("-V" "margin-left=1.03in") ("-V" "margin-right=1.03in") ("-V" "margin-top=1.435in") ("-V" "margin-bottom=1.435in") ("-V" "fontfamily=XCharter") ("-V" "fontsize=11pt") ("-V" "colorlinks=true") ("-V" "linkcolor=blue") ("-V" "urlcolor=Maroon") ("-V" "toccolor=blue") "--verbose")

    The command-line options to invoke [*DOCUMENT-PANDOC-PROGRAM*][dd37] with. For ease of manipulation, related options are grouped into sublists, but the entire nested list is flattened to get the list of options to pass to Pandoc. If --verbose is specified, then in addition to Pandoc logging LaTeX sources, PAX will log to [*ERROR-OUTPUT*][66c6] the Markdown that it converts to PDF via LaTeX. The Markdown includes *DOCUMENT-PANDOC-PDF-HEADER-INCLUDES* and [*DOCUMENT-PANDOC-PDF-METADATA-BLOCK*][ee8b].

  • [variable] *DOCUMENT-PANDOC-PDF-HEADER-INCLUDES* "<too messy to include>"

    LaTeX code (a string) to include in the preamble via header-includes.

    The default includes have no configuration knobs. Look at the value to see how to customize it.

  • [variable] *DOCUMENT-PANDOC-PDF-METADATA-BLOCK* ""

    A [Pandoc YAML metadata block][84f2] as a string.

    Concatenate to this string to customize it.

8.9.4 Dummy Output

When the FORMAT argument of DOCUMENT is NIL, no output is generated, but Transcript Consistency Checking is still performed. Use this feature to quickly test documentation examples.

For example, in Try the test would look like this:

(try:signals-not (transcription-consistency-error)
  (document ... :format nil))

8.10 Documentation Generation Implementation Notes

Documentation Generation is supported on ABCL, AllegroCL, CLISP, CCL, CMUCL, ECL and SBCL, but their outputs may differ due to the lack of some introspective capability. SBCL generates complete output. see ARGLIST, DOCSTRING and SOURCE-LOCATION for implementation notes.

In addition, CLISP does not support the ambiguous case of [Browsing Live Documentation][a595] because the current implementation relies on Swank to list definitions of symbols (as VARIABLE, FUNCTION, etc), and that simply doesn't work.

8.11 Utilities for Generating Documentation

Two convenience functions are provided to serve the common case of having an ASDF system with some readmes and a directory with the HTML documentation and the default CSS stylesheet.

  • [function] UPDATE-ASDF-SYSTEM-READMES OBJECT ASDF-SYSTEM &KEY (URL-VERSIONS '(1)) (FORMATS '(:MARKDOWN))

    Convenience function to generate up to two readme files in the directory holding the ASDF-SYSTEM definition. OBJECT is passed on to DOCUMENT.

    If :MARKDOWN is in FORMATS, then README.md is generated with anchors, links, inline code, and other markup added. Not necessarily the easiest on the eye in an editor but looks good on GitHub.

    If :PLAIN is in FORMATS, then README is generated, which is optimized for reading in text format. It has less cluttery markup and no [Autolink][ec7a]ing.

    Example usage:

    (update-asdf-system-readmes @pax-manual :mgl-pax
                            :formats '(:markdown :plain))

    Note that *DOCUMENT-URL-VERSIONS* is bound to URL-VERSIONS, which defaults to using the uglier, version 1 style of URL for the sake of GitHub.

8.11.1 HTML Output

  • [function] UPDATE-ASDF-SYSTEM-HTML-DOCS SECTIONS ASDF-SYSTEM &KEY PAGES (TARGET-DIR (ASDF/SYSTEM:SYSTEM-RELATIVE-PATHNAME ASDF-SYSTEM "doc/")) (UPDATE-CSS-P T) (STYLE *DOCUMENT-HTML-DEFAULT-STYLE*)

    Generate pretty HTML documentation for a single ASDF system, possibly linking to GitHub. If UPDATE-CSS-P, copy the STYLE files to TARGET-DIR (see [*DOCUMENT-HTML-DEFAULT-STYLE*][90fa]).

    Example usage:

    (update-asdf-system-html-docs @pax-manual :mgl-pax)

    The same, linking to the sources on GitHub:

    (update-asdf-system-html-docs
  @pax-manual :mgl-pax
  :pages
  `((:objects (,mgl-pax::@pax-manual)
     :source-uri-fn ,(make-git-source-uri-fn
                      :mgl-pax
                      "https://github.com/melisgl/mgl-pax"))))

See the following variables, which control HTML generation.

  • [variable] *DOCUMENT-HTML-DEFAULT-STYLE* :DEFAULT

    The HTML style to use. It's either :DEFAULT or :CHARTER. The :DEFAULT CSS stylesheet relies on the default fonts (sans-serif, serif, monospace), while :CHARTER bundles some fonts for a more controlled look.

    The value of this variable affects the default style of [UPDATE-ASDF-SYSTEM-HTML-DOCS][bb12].

  • [variable] *DOCUMENT-HTML-MAX-NAVIGATION-TABLE-OF-CONTENTS-LEVEL* NIL

    NIL or a non-negative integer. If non-NIL, it overrides [*DOCUMENT-MAX-NUMBERING-LEVEL*][f12d] in the dynamic HTML table of contents on the left of the page.

  • [variable] *DOCUMENT-HTML-LANG* "en"

    The value for the html element's xml:lang and lang attributes in the generated HTML.

  • [variable] *DOCUMENT-HTML-CHARSET* "UTF-8"

    The value for charset attribute of the <meta http-equiv='Content-Type' content='text/html'> element in the generated HTML.

  • [variable] *DOCUMENT-HTML-HEAD* NIL

    Stuff to be included in the <head> of the generated HTML.

    • If NIL, nothing is included.

    • If a [STRING][68cc], then it is written to the HTML output as is without any escaping.

    • If a function designator, then it is called with a single argument, the HTML stream, where it must write the output.

  • [variable] *DOCUMENT-HTML-SIDEBAR* NIL

    Stuff to be included in the HTML sidebar.

    • If NIL, a default sidebar is generated, with [*DOCUMENT-HTML-TOP-BLOCKS-OF-LINKS*][e216], followed by the dynamic table of contents, and *DOCUMENT-HTML-BOTTOM-BLOCKS-OF-LINKS*.

    • If a [STRING][68cc], then it is written to the HTML output as is without any escaping.

    • If a function designator, then it is called with a single argument, the HTML stream, where it must write the output.

  • [variable] *DOCUMENT-HTML-TOP-BLOCKS-OF-LINKS* NIL

    A list of blocks of links to be displayed on the sidebar on the left, above the table of contents. A block is of the form (&KEY TITLE ID LINKS), where TITLE will be displayed at the top of the block in a HTML DIV with ID followed by the links. LINKS is a list of (URI LABEL) elements, where URI may be a string or an object being DOCUMENTed or a REFERENCE thereof.

  • [variable] *DOCUMENT-HTML-BOTTOM-BLOCKS-OF-LINKS* NIL

    Like [*DOCUMENT-HTML-TOP-BLOCKS-OF-LINKS*][e216], only it is displayed below the table of contents.

8.11.2 GitHub Workflow

It is generally recommended to commit generated readmes (see UPDATE-ASDF-SYSTEM-READMES) so that users have something to read without reading the code and sites like GitHub can display them.

Also, by registering in PAX World, you get documentation generated in various formats periodically. If you want to manage this yourself, there are a couple of things to keep in mind.

First, HTML documentation can also be committed, but there is an issue with that: when linking to the sources (see [MAKE-GIT-SOURCE-URI-FN][587f]), the commit id is in the link. This means that code changes need to be committed first, and only then can HTML documentation be regenerated and committed in a followup commit.

The second issue is that GitHub is not very good at serving HTML files from the repository itself (and http://htmlpreview.github.io chokes on links to the sources).

The recommended workflow is to use gh-pages, which can be made relatively painless with the git worktree command. The gist of it is to make the doc/ directory a checkout of the branch named gh-pages. There is a good description of this general process. Two commits are needed still, but it is somewhat less painful.

This way the HTML documentation will be available at

http://<username>.github.io/<repo-name>

It is probably a good idea to add sections like the [Links and Systems][ba90] section to allow jumping between the repository and the gh-pages site.

  • [function] MAKE-GITHUB-SOURCE-URI-FN ASDF-SYSTEM GITHUB-URI &KEY GIT-VERSION

    This function is a backward-compatibility wrapper around [MAKE-GIT-SOURCE-URI-FN][587f], which supersedes MAKE-GITHUB-SOURCE-URI-FN. All arguments are passed on to MAKE-GIT-SOURCE-URI-FN, leaving URI-FORMAT-STRING at its default, which is suitable for GitHub.

  • [function] MAKE-GIT-SOURCE-URI-FN ASDF-SYSTEM GIT-FORGE-URI &KEY GIT-VERSION GIT-ROOT (URI-FORMAT-STRING "~A/blob/~A/A#LS")

    Return an object suitable as :SOURCE-URI-FN of a page spec (see the PAGES argument of DOCUMENT). The function looks at the source location of the object passed to it, and if the location is found, the path is made relative to the GIT-ROOT (the top-level directory of the git checkout), and finally an URI pointing to your git forge (such as GitHub) is returned.

    • If ASDF-SYSTEM is non-NIL, then GIT-ROOT defaults to the directory of the .asd file of the system. Similarly, GIT-VERSION defaults current commit id in the checkout at GIT-ROOT.

    • If both GIT-ROOT and GIT-VERSION are explicitly provided, ASDF-SYSTEM has no effect.

    If GIT-ROOT is not specified and cannot be determined from ASDF-SYSTEM, then no links to the git forge will be generated.

    A warning is signalled whenever the source location lookup fails or if the source location points to a directory not below GIT-ROOT.

    If GIT-FORGE-URI is "https://github.com/melisgl/mgl-pax/" and GIT-VERSION is "master", then the returned URI may look like this:

      https://github.com/melisgl/mgl-pax/blob/master/src/pax-early.lisp#L12

    URI-FORMAT-STRING is a [CL:FORMAT][ad78] control string for four arguments:

    • GIT-FORGE-URI,

    • GIT-VERSION,

    • the relative path to the file of the source location of the reference,

    • and the line number.

    The default value of URI-FORMAT-STRING is for GitHub. If using a non-standard git forge, such as Sourcehut or GitLab, simply pass a suitable URI-FORMAT-STRING matching the URI scheme of your forge.

8.11.3 PAX World

PAX World is a registry of documents, which can generate cross-linked HTML documentation pages for all the registered documents. There is an official PAX World at https://fixnum.com/, which is updated every few hours. To have your stuff there, [REGISTER-DOC-IN-PAX-WORLD][f4fd] in your ASDF system and file an issue at https://github.com/melisgl/mgl-pax-world/issues to make PAX World actually load it.

  • [function] REGISTER-DOC-IN-PAX-WORLD ID SECTIONS PAGE-SPECS

    Register SECTIONS and PAGE-SPECS under ID (an arbitrary symbol) in PAX World. By default, [UPDATE-PAX-WORLD][ee51] generates documentation for all of these. SECTIONS and PAGE-SPECS must be lists of [SECTION][5fac]s and PAGE-SPECs (see [PAGES][9c7d]) or designators of functions with no arguments that return such lists.

    Using an existing ID overwrites the previous registration.

For example, this is how PAX registers itself:

(defun pax-sections ()
  (list @pax-manual))

(defun pax-pages ()
  `((:objects ,(pax-sections)
     :source-uri-fn ,(make-git-source-uri-fn
                      "mgl-pax" "https://github.com/melisgl/mgl-pax"))))

(register-doc-in-pax-world :pax 'pax-sections 'pax-pages)

  • [function] UPDATE-PAX-WORLD &KEY (DOCS *REGISTERED-PAX-WORLD-DOCS*) (FORMATS '(:HTML)) DIR UPDATE-CSS-P (STYLE *DOCUMENT-HTML-DEFAULT-STYLE*)

    Generate HTML documentation for all DOCS. Files are created in DIR ((asdf:system-relative-pathname :mgl-pax "world/") by default if DIR is NIL). DOCS is a list of entries of the form (NAME [SECTIONS][5fac] PAGE-SPECS). The default for DOCS is all the sections and pages registered with [REGISTER-DOC-IN-PAX-WORLD][f4fd].

    In the absence of :HEADER-FN :FOOTER-FN, :OUTPUT, every spec in PAGE-SPECS is augmented with HTML headers, footers and output location specifications (based on the name of the section).

    If necessary, a default page spec is created for every section.

9 Transcripts

What are transcripts for? When writing a tutorial, one often wants to include a REPL session with maybe a few defuns and a couple of forms whose output or return values are shown. Also, in a function's docstring an example call with concrete arguments and return values speaks volumes. A transcript is a text that looks like a REPL session, but which has a light markup for printed output and return values, while no markup (i.e. prompt) for Lisp forms. PAX transcripts may include output and return values of all forms, or only selected ones. In either case, the transcript itself can be easily generated from the source code. The main worry associated with including examples in the documentation is that they tend to get out-of-sync with the code. This is solved by being able to parse back and update transcripts.

9.1 Transcribing in Documentation

Consider this function:

(defun foo (x)
  "Return X + 1 and log what happened."
  (format t "Adding 1 to ~S~%" x)
  (1+ x))

Let's add an example to the docstring:

(defun foo (x)
  "Return X + 1 and log what happened.
  For example,

  ```cl
  (foo 7)
  .. Adding 1 to 7
  ..
  => 8
  ```"
  (format t "Adding 1 to ~S~%" x)
  (1+ x))

In this transcript above, output lines are prefixed with ".. " and return values with "=> ". The transcript could have been generated with (TRANSCRIBE "(foo 7)" *STANDARD-OUTPUT*) or interactively by invoking mgl-pax-transcribe-last-expression in Emacs with the point right after (FOO 7) (see [Transcript API][9dbc] and [Transcribing with Emacs][f5bd]).

If we want PAX to check that the transcript is consistent with the code when Generating Documentation, we add the cl-transcript tag to the code block:

(defun foo (x)
  "Return X + 1 and log what happened.
  For example,

  ```cl-transcript
  (foo 7)
  .. Adding 1 to 7
  ..
  => 8
  ```"
  (format t "Adding 1 to ~S~%" x)
  (+ x 2))

Since the code was also changed to (+ X 2), documenting this function causes the following error:

(document #'foo)
.. debugger invoked on TRANSCRIPTION-VALUES-CONSISTENCY-ERROR:
..   Transcription error: Readable value "8" in source is not EQUAL to "9".
..     [While documenting (MGL-PAX::FOO COMMON-LISP:FUNCTION)]
..
..   Form:
..   "(foo 7)"

When [Browsing Live Documentation][a595], any errors signalled during transcription are downgraded to warnings.

All in all, transcripts are a handy tool especially when combined with the Emacs support to update them and with PYTHONIC-STRING-READER's triple-quoted strings, that allow one to work with nested strings with less noise. The triple-quote syntax can be enabled with:

(in-readtable pythonic-string-syntax)

9.2 Transcribing with Emacs

With [Emacs set up][8541], we can have two Elisp functions at our disposal for working with transcripts.

  • mgl-pax-transcribe-last-expression: Evaluates a single form, and insert its output and return values into the current buffer.

  • mgl-pax-retranscribe-region: Retranscribes a region as with [TRANSCRIBE][f1f0] :UPDATE-ONLY T. The region defaults to the innermost enclosing cl-transcript block if any.

These functions work in any major and minor mode where the basic sexp movements are sufficiently Lisp-like. They also work anywhere in the buffer, including docstrings and comments at any indentation level. Like DOCUMENT, they understand the :DYNENV argument of cl-transcript and pass on the other arguments to TRANSCRIBE.

For example, consider this buffer content, which may be part of a docstring. Move the cursor right after the end of the form as if you were to evaluate it with C-x C-e. The cursor is marked by ^:

(values (princ :hello) (list 1 2))^

Now invoke the Elisp function mgl-pax-transcribe where the cursor is to insert the transcribed output and return values:

(values (princ :hello) (list 1 2))
.. HELLO
=> :HELLO
=> (1 2)
^

Then, change the printed message and add comments to the second return value:

(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> ;; This is a list.
   (1
    ;; This value is arbitrary.
    2)

Obviously, the transcript is now out-of-date, and if it is in a cl-transcript code block in a docstring, then DOCUMENT will signal a [TRANSCRIPTION-CONSISTENCY-ERROR][a249]. So, let's update the transcript by marking the region bounded by | and the cursor at ^ in this example:

|(values (princ :hello-world) (list 1 2))
.. HELLO
=> :HELLO
=> ;; This is a list.
   (1
    ;; This value is arbitrary.
    2)
^

Then, invoke the Elisp function mgl-pax-retranscribe-region to get

(values (princ :hello-world) (list 1 2))
.. HELLO-WORLD
=> :HELLO-WORLD
=> ;; This is a list.
   (1
    ;; This value is arbitrary.
    2)
^

Note how the indentation and comments of (1 2) were left alone, but the output and the first return value got updated.

Also note that when the cursor is in cl-transcript code block, mgl-pax-retranscribe-region defaults to the whole block when there is no active region.

Alternatively, C-u 1 mgl-pax-transcribe will emit commented markup:

(values (princ :hello) (list 1 2))
;.. HELLO
;=> :HELLO
;=> (1 2)

C-u 0 mgl-pax-retranscribe-region will turn commented into non-commented markup. In general, the numeric prefix argument is the index of the syntax to be used in [*TRANSCRIBE-SYNTAXES*][ebd3]. Without a prefix argument, mgl-pax-retranscribe-region will not change the markup style.

Finally, not only do both functions work at any indentation level but in comments too:

;;;; (values (princ :hello) (list 1 2))
;;;; .. HELLO
;;;; => :HELLO
;;;; => (1 2)

With mgl-pax-transcribe-last-expression, we strip the longest run of leading spaces and semicolons common to all lines of the expression in the buffer.

For mgl-pax-retranscribe-region, the longest run is truncated so that it does not extend beyond the column of the first form to be transcribed. Without this rule, the syntax used

;;(list 1 2)
;;;=> (1
;;;->  2)

would be ambiguous, as the ;=> could refer to => in the :DEFAULT syntax or to ;=> in :COMMENTED-1.

The dynamic environment of the transcription is determined by the :DYNENV argument of the enclosing cl-transcript code block (see [Controlling the Dynamic Environment][6b59]).

9.3 Controlling the Dynamic Environment

When the [TRANSCRIBE][f1f0] function is called directly, the forms in the transcript are evaluated in the current dynamic environment with the following exceptions.

  • The variables [*][9590], [**][78d1], [***][ea37], [/][9f2f], [//][e433], ///, -, [+][72a7], [++][fbb1], +++ are locally bound and updated as in a new [REPL][f83b] session.

  • [*STANDARD-OUTPUT*][e7ee], [*ERROR-OUTPUT*][66c6], *TRACE-OUTPUT*, *DEBUG-IO*, [*QUERY-IO*][f4bf] and [*TERMINAL-IO*][633c] are redirected to capture the output.

If TRANSCRIBE is invoked [by Emacs][f5bd] or [by the DOCUMENT function][9bd5], a new dynamic environment is established by [STANDARD-TRANSCRIBE-DYNENV][588c] (similar in spirit to WITH-STANDARD-IO-SYNTAX), which binds printer and reader variables to fixed values to make IO more predictable. This default can be overridden with the :DYNENV argument of cl-transcript:

```cl-transcript (:dynenv my-transcript)
...
```

In this case, instead of calling TRANSCRIBE directly, MY-TRANSCRIPT is called with a thunk (a function of no arguments) that wraps a call to TRANSCRIBE. Once MY-TRANSCRIPT establishes the desired dynamic environment, it calls its argument. The following definition of MY-TRANSCRIPT simply packages up oft-used settings to TRANSCRIBE.

(defun my-transcript (fn)
  (standard-transcribe-dynenv
    (let ((*transcribe-check-consistency*
            '((:output my-transcript-output=)
              (:readable equal)
              (:unreadable nil))))
      (funcall fn))))

(defun my-transcript-output= (string1 string2)
  (string= (my-transcript-normalize-output string1)
           (my-transcript-normalize-output string2)))

(defun my-transcript-normalize-output (string)
  (squeeze-whitespace (delete-trailing-whitespace (delete-comments string))))

The default for :DYNENV is STANDARD-TRANSCRIBE-DYNENV, and it is generally a good idea to use it as the above example does, but this is not required. Specify :DYNENV NIL explicitly if you want to use the current dynamic environment without any changes.

A more involved solution could establish bindings for global variables set in transcripts, unintern symbols created or even create a temporary package for evaluation.

  • [function] STANDARD-TRANSCRIBE-DYNENV FN

    Bind printer and reader variables to standard values and call FN.

    The bindings are the same as with WITH-STANDARD-IO-SYNTAX, but

    • [*PACKAGE*][5ed1] and [*READTABLE*][b79a] are unaffected (because both DOCUMENT and Emacs set these up);

    • [*PRINT-READABLY*][8aca] is NIL, [*PRINT-PRETTY*][782a] is T, [*PRINT-CIRCLE*][c8cb] is T and [*PRINT-RIGHT-MARGIN*][5ab6] is 72.

    This function is the default for the :DYNENV argument of cl-transcript. A function that overrides the default may want to call STANDARD-TRANSCRIBE-DYNENV with a lambda that establishes more bindings.

9.4 Transcript API

  • [function] TRANSCRIBE INPUT OUTPUT &KEY UPDATE-ONLY (INCLUDE-NO-OUTPUT UPDATE-ONLY) (INCLUDE-NO-VALUE UPDATE-ONLY) (ECHO T) (CHECK-CONSISTENCY *TRANSCRIBE-CHECK-CONSISTENCY*) DEFAULT-SYNTAX (INPUT-SYNTAXES *TRANSCRIBE-SYNTAXES*) (OUTPUT-SYNTAXES *TRANSCRIBE-SYNTAXES*)

    Read forms from INPUT and write them (iff ECHO) to OUTPUT followed by any output and return values produced by calling EVAL on the form.

    INPUT can be a stream or a string, while OUTPUT can be a stream or NIL, in which case output goes into a string. The return value is the OUTPUT stream or the string that was constructed. As the second value, a generalized boolean indicating whether any form was transcribed is returned.

    Go up to [Transcribing with Emacs][f5bd] for nice examples. A more mind-bending one is this:

    (transcribe "(princ 42) " nil)
=> "(princ 42)
.. 42
=> 42
"
=> T

    However, the above may be a bit confusing since this documentation uses TRANSCRIBE markup syntax in this very example, so let's do it differently. If we have a file with these contents:

    (values (princ 42) (list 1 2))

    it is transcribed to:

    (values (princ 42) (list 1 2))
.. 42
=> 42
=> (1 2)

    Output to all standard streams is captured and printed with the :OUTPUT prefix (".."). The return values above are printed with the :READABLE prefix ("=>"). Note how these prefixes are always printed on a new line to facilitate parsing.

    Updating

    TRANSCRIBE is able to parse its own output. If we transcribe the previous output above, we get it back exactly. However, if we remove all output markers, leave only a placeholder value marker and pass :UPDATE-ONLY T with source:

    (values (princ 42) (list 1 2))
=>

    we get this:

    (values (princ 42) (list 1 2))
=> 42
=> (1 2)

    With UPDATE-ONLY, the printed output of a form is transcribed only if there were output markers in the source. Similarly, with UPDATE-ONLY, return values are transcribed only if there were value markers in the source.

    No Output/Values

    If the form produces no output or returns no values, then whether or not output and values are transcribed is controlled by INCLUDE-NO-OUTPUT and INCLUDE-NO-VALUE, respectively. By default, neither is on so:

    (values)
..
=>

    is transcribed to

    (values)

    With UPDATE-ONLY true, we probably wouldn't like to lose those markers since they were put there for a reason. Hence, with UPDATE-ONLY, INCLUDE-NO-OUTPUT and INCLUDE-NO-VALUE default to true. So, with UPDATE-ONLY the above example is transcribed to:

    (values)
..
=> ; No value

    where the last line is the :NO-VALUE prefix.

    Consistency Checks

    If CHECK-CONSISTENCY is true, then TRANSCRIBE signals a continuable [TRANSCRIPTION-OUTPUT-CONSISTENCY-ERROR][8492] whenever a form's output as a string is different from what was in INPUT, provided that INPUT contained the output. Similarly, for values, a continuable TRANSCRIPTION-VALUES-CONSISTENCY-ERROR is signalled if a value read from the source does not print as the value returned by EVAL. This allows readable values to be hand-indented without failing consistency checks:

    (list 1 2)
=> ;; This is commented, too.
   (1
      ;; Funny indent.
      2)

    See Transcript Consistency Checking for the full picture.

    Unreadable Values

    The above scheme involves [READ][fe58], so consistency of unreadable values cannot be treated the same. In fact, unreadable values must even be printed differently for transcribe to be able to read them back:

    (defclass some-class () ())

(defmethod print-object ((obj some-class) stream)
  (print-unreadable-object (obj stream :type t)
    (format stream "~%~%end")))

(make-instance 'some-class)
==> #<SOME-CLASS 
-->
--> end>

    where "==>" is the :UNREADABLE prefix and "-->" is the :UNREADABLE-CONTINUATION prefix. As with outputs, a consistency check between an unreadable value from the source and the value from EVAL is performed with STRING= by default. That is, the value from EVAL is printed to a string and compared to the source value. Hence, any change to unreadable values will break consistency checks. This is most troublesome with instances of classes with the default PRINT-OBJECT method printing the memory address. See [Finer-Grained Consistency Checks][6e18].

    Errors

    If an [ERROR][d162] condition is signalled, the error is printed to the output and no values are returned.

    (progn
  (print "hello")
  (error "no greeting"))
..
.. "hello" 
.. debugger invoked on SIMPLE-ERROR:
..   no greeting

    To keep the textual representation somewhat likely to be portable, the printing is done with (FORMAT T "#<~S ~S>" (TYPE-OF ERROR) (PRINC-TO-STRING ERROR)). [SIMPLE-CONDITION][f2f5]s are formatted to strings with SIMPLE-CONDITION-FORMAT-CONTROL and [SIMPLE-CONDITION-FORMAT-ARGUMENTS][da14].

    Syntaxes

    Finally, a transcript may employ different syntaxes for the output and values of different forms. When INPUT is read, the syntax for each form is determined by trying to match all prefixes from all syntaxes in INPUT-SYNTAXES against a line. If there are no output or values for a form in INPUT, then the syntax remains undetermined.

    When OUTPUT is written, the prefixes to be used are looked up in DEFAULT-SYNTAX of OUTPUT-SYNTAXES if DEFAULT-SYNTAX is not NIL. If DEFAULT-SYNTAX is NIL, then the syntax used by the same form in the INPUT is used or (if that could not be determined) the syntax of the previous form. If there was no previous form, then the first syntax if OUTPUT-SYNTAXES is used.

    To produce a transcript that's executable Lisp code, use :DEFAULT-SYNTAX :COMMENTED-1:

    (make-instance 'some-class)
;==> #<SOME-CLASS
;-->
;--> end>

(list 1 2)
;=> (1
;->    2)

    To translate the above to uncommented syntax, use :DEFAULT-SYNTAX :DEFAULT. If DEFAULT-SYNTAX is NIL (the default), the same syntax will be used in the output as in the input as much as possible.

  • [variable] *TRANSCRIBE-CHECK-CONSISTENCY* NIL

    The default value of [TRANSCRIBE][f1f0]'s CHECK-CONSISTENCY argument.

  • [variable] *TRANSCRIBE-SYNTAXES* ((:DEFAULT (:OUTPUT "..") (:NO-VALUE "=> ; No value") (:READABLE "=>") (:UNREADABLE "==>") (:UNREADABLE-CONTINUATION "-->")) (:COMMENTED-1 (:OUTPUT ";..") (:NO-VALUE ";=> ; No value") (:READABLE ";=>") (:READABLE-CONTINUATION ";->") (:UNREADABLE ";==>") (:UNREADABLE-CONTINUATION ";-->")) (:COMMENTED-2 (:OUTPUT ";;..") (:NO-VALUE ";;=> ; No value") (:READABLE ";;=>") (:READABLE-CONTINUATION ";;->") (:UNREADABLE ";;==>") (:UNREADABLE-CONTINUATION ";;-->")))

    The default syntaxes used by [TRANSCRIBE][f1f0] for reading and writing lines containing output and values of an evaluated form.

    A syntax is a list of of the form (SYNTAX-ID &REST PREFIXES) where PREFIXES is a list of (PREFIX-ID PREFIX-STRING) elements. For example the syntax :COMMENTED-1 looks like this:

    (:commented-1
 (:output ";..")
 (:no-value ";=>  No value")
 (:readable ";=>")
 (:readable-continuation ";->")
 (:unreadable ";==>")
 (:unreadable-continuation ";-->"))

    All of the above prefixes must be defined for every syntax except for :READABLE-CONTINUATION. If that's missing (as in the :DEFAULT syntax), then the following value is read with [READ][fe58] and printed with [PRIN1][6384] (hence no need to mark up the following lines).

    When writing, an extra space is added automatically if the line to be prefixed is not empty. Similarly, the first space following the prefix is discarded when reading.

    See TRANSCRIBE for how the actual syntax to be used is selected.

  • [condition] TRANSCRIPTION-ERROR [ERROR][d162] CONDITION-CONTEXT-MIXIN

    Represents syntactic errors in the SOURCE argument of [TRANSCRIBE][f1f0] and also serves as the superclass of [TRANSCRIPTION-CONSISTENCY-ERROR][a249].

  • [condition] TRANSCRIPTION-OUTPUT-CONSISTENCY-ERROR [TRANSCRIPTION-CONSISTENCY-ERROR][a249]

    Signalled (with CERROR) by [TRANSCRIBE][f1f0] when invoked with :CHECK-CONSISTENCY and the output of a form is not the same as what was parsed.

  • [condition] TRANSCRIPTION-VALUES-CONSISTENCY-ERROR [TRANSCRIPTION-CONSISTENCY-ERROR][a249]

    Signalled (with CERROR) by [TRANSCRIBE][f1f0] when invoked with :CHECK-CONSISTENCY and the values of a form are inconsistent with their parsed representation.

9.5 Transcript Consistency Checking

The main use case for consistency checking is detecting out-of-date examples in documentation, although using it for writing tests is also a possibility. Here, we focus on the former.

When a Markdown code block tagged cl-transcript is processed during Generating Documentation, the code in it is replaced with the output of (TRANSCRIBE <CODE> NIL :UPDATE-ONLY T :CHECK-CONSISTENCY T). Suppose we have the following example of the function GREET, that prints hello and returns 7.

```cl-transcript
(greet)
.. hello
=> 7
```

Now, if we change GREET to print or return something else, a [TRANSCRIPTION-CONSISTENCY-ERROR][a249] will be signalled during documentation generation. Then we may fix the documentation or CONTINUE from the error.

By default, comparisons of previous to current output, readable and unreadable return values are performed with STRING=, EQUAL, and STRING=, respectively, which is great in the simple case. Non-determinism aside, exact matching becomes brittle as soon as the notoriously unportable pretty printer is used or when unreadable objects are printed with their #<> syntax, especially when [PRINT-UNREADABLE-OBJECT][9439] is used with :IDENTITY T.

9.5.1 Finer-Grained Consistency Checks

To get around this problem, consistency checking of output, readable and unreadable values can be customized individually by supplying [TRANSCRIBE][f1f0] with a CHECK-CONSISTENCY argument like ((:OUTPUT <OUTPUT-CHECK>) (:READABLE <READABLE-CHECK>) (:UNREADABLE <UNREADABLE-CHECK>)). In this case, <OUTPUT-CHECK> may be NIL, T, or a function designator.

  • If it's NIL or there is no :OUTPUT entry in the list, then the output is not checked for consistency.

  • If it's T, then the outputs are compared with the default, STRING=.

  • If it's a function designator, then it's called with two strings and must return whether they are consistent with each other.

The case of <READABLE-CHECK> and <UNREADABLE-CHECK> is similar.

Code blocks tagged cl-transcript can take arguments, which they pass on to TRANSCRIBE. The following shows how to check only the output.

```cl-transcript (:check-consistency ((:output t)))
(error "Oh, no.")
.. debugger invoked on SIMPLE-ERROR:
..   Oh, no.

(make-condition 'simple-error)
==> #<SIMPLE-ERROR {1008A81533}>
```

It is often a good idea to package up these settings in the :DYNENV argument of cl-transcript (see [Controlling the Dynamic Environment][6b59]).

9.5.2 Utilities for Consistency Checking

  • [function] SQUEEZE-WHITESPACE STRING

    Replace consecutive whitespace characters with a single space in STRING and trim whitespace from the right. This is useful to undo the effects of pretty printing when building comparison functions for [TRANSCRIBE][f1f0].

  • [function] DELETE-TRAILING-WHITESPACE STRING

    Delete whitespace characters after the last non-whitespace character in each line in STRING.

  • [function] DELETE-COMMENTS STRING &KEY (PATTERN ";")

    For each line in STRING delete the rest of the line after and including the first occurrence of PATTERN. On changed lines, delete trailing whitespace too. This function does not parse STRING as Lisp forms, hence all occurrences of PATTERN (even those seemingly in string literals) are recognized as comments.

    Let's define a comparison function:

    (defun string=/no-comments (string1 string2)
  (string= (delete-comments string1) (delete-comments string2)))

    And use it to check consistency of output:

      ```cl-transcript (:check-consistency ((:output string=/no-comments)))
  (format t "hello~%world")
  .. hello     ; This is the first line.
  .. world     ; This is the second line.
  ```

    Just to make sure the above example works, here it is without being quoted.

    (format t "hello~%world")
.. hello     ; This is the first line.
.. world     ; This is the second line.

10 Parsing

10.1 Parsing Names

When encountering a [word][d7b0] such as CLASSes in a docstring, PAX needs to find the [name][88cf], and how that's done varies slightly. [Codification][f1ab], for example, looks for [interesting][7445] names, Navigating Sources in Emacs for names with Lisp DEFINITIONS, and Linking for names with any kind of definition.

This is not as straightforward as it sounds because it needs to handle cases like un[READ][fe58]able, [PRINT][d451]ed, and all the various forms of Linking in docstrings as well as in comments, and the (NAME LOCATIVE) syntax in [DEFSECTION][72b4].

  • [glossary-term] word

    A word is a string from which we want to extract a [name][88cf]. When Navigating, the word is slime-sexp-at-point or the label of a [Markdown reference link][8c00] if point is over one. Similarly, when Generating Documentation, it is a non-empty string between whitespace characters in a docstring or the label of a [Markdown reference link][8c00].

  • [glossary-term] raw name

    A raw name is a string from which a [name][88cf] may be read. Raw names correspond to an intermediate parsing step between [word][d7b0]s and [name][88cf]s. See Names in Raw Names.

  • [glossary-term] name

    A name is a DRef name. That is, a symbol, a string or a nested list of the previous associated with a definition, whose kind is given by a locative.

Depending on the context, trimming and depluralization may be enabled (see [Raw Names in Words][f0d5]), while the possible names may be restricted to symbols (see Names in Raw Names).

  • Trimming: Enabled for M-. Defaulting and [Codification][f1ab].

  • Depluralization: Enabled when the [word][d7b0] is part of the normal flow of text (i.e. not for [Specific Reflink with Text][fb17], Unspecific Reflink with Text and various Elisp functions such as mgl-pax-apropos unless they determine their argument from buffer contents).

  • Symbols only: This is the case for [Codification][f1ab] and [Unspecific Autolink][e2a4] to prevent string-based definitions from littering the documentation with links without the control provided by explicitly [IMPORT][8f46]ing symbols.

For a word, a number of [raw name][f5af]s are generated by trimming delimiter characters and plural markers, and for each raw name a number of names are considered until one is found suitable in the context. The following subsections describe the details of the parsing algorithm.

10.1.1 Raw Names in Words

From [word][d7b0]s, [raw name][f5af]s are parsed by trimming some prefixes and suffixes. For a given word, multiple raw names are considered in the following order.

  1. The entire word.

  2. Trimming the following characters from the left of the word:

     #<{;"'`

  3. Trimming the following characters from the right of the word:

     ,;:.>}"'`

  4. Trimming both of the previous two at the same time.

  5. From the result of 4., If a [word][d7b0] ends with what looks like a plural marker (case-insensitive), then a [name][88cf] is created by removing it. For example, from the word BUSES the plural marker ES is removed to produce the name BUS. The list of plural markers considered is SES (e.g. GASSES), ES (e.g. BUSES), S (e.g. CARS), ZES (e.g. FEZZES), and REN (e.g. CHILDREN).

  6. From the result of 4., removing the prefix before the first, and the suffix after the last uppercase character if it contains at least one lowercase character.

10.1.2 Names in Raw Names

For each [raw name][f5af] from [Raw Names in Words][f0d5], various [name][88cf]s may be considered until one is found that's suitable in the context.

The following examples list the names considered for a given raw name, assuming that READTABLE-CASE is :UPCASE as well as that FOO and |Foo| are interned.

  • "foo": FOO, "foo", "FOO" (rules 1, 2, 3)

  • "FOO": FOO, "FOO" (rules 1, 2)

  • "Foo": "Foo", "FOO" (rules 2, 3)

  • "|Foo|": |Foo| (rule 4)

  • "\"foo\"": "foo" (rule 5)

The rules are:

  1. If the raw name is not mixed case (i.e. it doesn't have both upper- and lowercase characters) and it names an interned symbol (subject to the current [Package and Readtable][ab7e]), then that symbol is considered as a name.

  2. The raw name itself (a string) is considered a name.

  3. The raw name upcased or downcased according to READTABLE-CASE (subject to the [current readtable][ab7e]) but still as a string. This is to allow [dref][package] to refer to the "DREF" package regardless of whether the symbol DREF is interned in the current package.

  4. If the raw name is explicitly a symbol (it starts with #\|), and it names an interned symbol (subject to the current [Package and Readtable][ab7e]), then that symbol is considered as a name and nothing else.

  5. If the raw name has an embedded string (it starts with #\") and [READ-FROM-STRING][d813] can read the embedded string from it, then that string is considered as a name and nothing else.

10.1.3 Name Parsing Example

For example, when M-. is pressed while point is over unREADable., the last word of the sentence It may be unREADable., the following [raw name][f5af]s are considered until one is found with a definition:

  1. The entire word, "unREADable.".

  2. Trimming left does not produce a new raw name.

  3. Trimming right removes the dot and gives "unREADable".

  4. Trimming both is the same as trimming right.

  5. No plural markers are found.

  6. The lowercase prefix and suffix is removed around the uppercase core, giving "READ". This names an interned symbol which has a definition, so M-. will visit it.

When Generating Documentation, [Autolink][ec7a]ing behaves similarly.

10.2 Parsing Locatives

Locatives are parsed almost as if by [READ][fe58]. They are found in buffer contents around a [word][d7b0] when M-. Defaulting or Generating Documentation, and in the string entered when [M-. Prompting][ed46], with a similar distinction when [Browsing Live Documentation][a595].

Parsing deviates from READ in the following ways.

  • No new symbols are [INTERN][b4f0]ed during parsing. If an expression contains uninterned symbols, then it is not parsable as a locative.

  • Read-time evaluation ([#.][ffd7]) follows normal READ semantics. Thus, (METHOD ((EQL #.(FIND-PACKAGE 'CL)))) may INTERN the symbol CL.

  • A locative that involves unreadable objects that print using the #< syntax (e.g. (METHOD ((EQL #<PACKAGE DREF>)))) is parsable in the context of a [name][88cf] if each unreadable object in the locative occurs in one of the DEFINITIONS of that name and it prints to an equivalent string (e.g. #<PACKAGE DREF> above).

  • [glossary-term] prints to an equivalent string

    An object with an unreadable representation is said to print to some string S if its [PRIN1][6384] representation (under WITH-STANDARD-IO-SYNTAX but in the current package and with [*PRINT-READABLY*][8aca] NIL) is the same as S, where consecutive whitepace characters are replaced with a single space in both strings, and the comparison is case-insensitive.

    See the related concept of stable printed locative, that requires the printed representation of entire locatives to be unique and non-changing to support Linking.

11 Writing Extensions

11.1 Adding New Locatives

Once everything in Extending DRef has been done, there are only a couple of PAX generic functions left to extend.

  • [generic-function] DOCUMENT-OBJECT* OBJECT STREAM

    Write the documentation of OBJECT in Markdown to STREAM. The output may depend on *FORMAT*. Specialize this on a subclass of DREF if that subclass is not RESOLVEable, else on the type of object it resolves to. This function is for extension only. Don't call it directly.

  • [generic-function] EXPORTABLE-REFERENCE-P PACKAGE SYMBOL LOCATIVE-TYPE LOCATIVE-ARGS

    Return true if SYMBOL is to be exported from PACKAGE when it occurs in a [DEFSECTION][72b4] in a reference with LOCATIVE-TYPE and LOCATIVE-ARGS. SYMBOL is [accessible][9b15] in PACKAGE.

    The default method calls [EXPORTABLE-LOCATIVE-TYPE-P][c930] with LOCATIVE-TYPE and ignores the other arguments.

    By default, [SECTION][5fac]s, [GLOSSARY-TERM][8251]s, CONCEPTs and [NOTE][e2ae]s are not exported although they are EXPORTABLE-LOCATIVE-TYPE-P. To export symbols naming sections from MGL-PAX, the following method could be added:

    (defmethod exportable-reference-p ((package (eql (find-package 'mgl-pax)))
                                   symbol (locative-type (eql 'section))
                                   locative-args)
  t)

  • [generic-function] EXPORTABLE-LOCATIVE-TYPE-P LOCATIVE-TYPE

    Return true if symbols in references with LOCATIVE-TYPE are to be exported by default when they occur in a [DEFSECTION][72b4]. The default method returns T, while the methods for locative types PACKAGE, METHOD, ASDF:SYSTEM, and [INCLUDE][5cd7] return NIL.

    This function is called by the default method of [EXPORTABLE-REFERENCE-P][e51f] to decide what symbols DEFSECTION shall export when its EXPORT argument is true.

Also note that due to the [HOME-SECTION][fda4] logic, especially for locative types with string names, DREF-EXT:DOCSTRING* should probably return a non-NIL package.

11.2 Locative Aliases

DEFINE-LOCATIVE-ALIAS can be used to help M-. and Specific Autolinks disambiguate references based on the context of a [name][88cf] as described on Parsing.

The following example shows how to make docstrings read more naturally by defining an alias.

(defclass my-string ()
  ())

(defgeneric my-string (obj)
  (:documentation "Convert OBJ to MY-STRING."))

;;; This version of FOO has a harder to read docstring because
;;; it needs to disambiguate the MY-STRING reference.
(defun foo (x)
  "FOO takes an argument X, a [MY-STRING][class] object.")

;;; Define OBJECT as an alias for the CLASS locative.
(define-locative-alias object class)

;;; Note how no explicit link is needed anymore.
(defun foo (x)
  "FOO takes an argument X, a MY-STRING object.")

Similarly, defining the indefinite articles as aliases of the CLASS locative can reduce the need for explicit linking.

(define-locative-alias a class)
(define-locative-alias an class)

Since these are unlikely to be universally helpful, make sure not to export the symbols A and AN.

11.3 Extending DOCUMENT

For all definitions that it encounters, DOCUMENT calls [DOCUMENT-OBJECT*][8269] to generate documentation. The following utilities are for writing new DOCUMENT-OBJECT* methods. They emit Markdown encoded in an unspecified format internal to DOCUMENT.

  • [variable] *FORMAT*

    Bound by DOCUMENT to its FORMAT argument, this allows Markdown output to depend on the output format.

  • [macro] WITH-HEADING (STREAM &KEY DREF LINK-TITLE-TO (NUMBERED T)) &BODY BODY

    Write a Markdown heading with the [DOCTITLE][e619] of DREF to STREAM.

    • DREF defaults to the definition for which documentation is currently being generated.

    • Nested WITH-HEADINGs produce nested headings.

    • If *DOCUMENT-LINK-SECTIONS*, generate anchors based on DREF.

    • LINK-TITLE-TO behaves like the LINK-TITLE-TO argument of [DEFSECTION][72b4].

    • NUMBERED controls whether a heading number is assigned to the heading. The heading number reflects the nesting structure of WITH-HEADINGS (e.g. 3.1.2). Descendants of unnumbered headings are unnumbered regardless of NUMBERED.

  • [generic-function] DOCTITLE* OBJECT

    DOCTITLE* extends [DOCTITLE][e619] in the same way as DOCSTRING* extends DOCSTRING.

    The default method returns NIL.

    This function is for extension only. Do not call it directly.

  • [macro] DOCUMENTING-DEFINITION (STREAM &KEY DREF PACKAGE READTABLE ARGLIST) &BODY BODY

    Write DREF to STREAM as described in [*DOCUMENT-MARK-UP-SIGNATURES*][8fb6], and establish DREF as a [Local Definition][9db9] for the processing of BODY.

    • DREF defaults to the definition for which documentation is currently being generated.

    • If DREF has a [DOCTITLE][e619], then it is [PRINC][676d]ed after the LOCATIVE-TYPE (see Markdown in Titles). Else, (DREF-NAME DREF) is printed subject to [*DOCUMENT-DOWNCASE-UPPERCASE-CODE*][a5ee] but with all Markdown and [MathJax][a17d] markup escaped.

    • [*PACKAGE*][5ed1] and [*READTABLE*][b79a] are bound to PACKAGE and READTABLE for the duration of printing the ARGLIST and the processing of BODY. If either is NIL, then a default value is computed as described in [Package and Readtable][ab7e].

    • ARGLIST:

      • If it is not provided, then it defaults to (ARGLIST DREF).

      • If NIL, then it is not printed.

      • If it is a list, then it is must be a [lambda list][98ff] and is printed without the outermost parens and with the package names removed from the argument names. [Keywords][88a0] are printed with a leading colon.

        &AUX variables and SUPPLIED-P-PARAMETERs of &OPTIONAL and &KEY arguments are considered internal and are not printed.

      • If it is a string, then it must be valid Markdown.

    • It is not allowed to have [WITH-HEADING][80e8] within the dynamic extent of BODY.

  • [macro] WITH-DISLOCATED-NAMES NAMES &BODY BODY

    For each name in NAMES, establish a [Local Definition][9db9].

  • [function] DOCUMENT-DOCSTRING DOCSTRING STREAM &KEY (INDENTATION " ") EXCLUDE-FIRST-LINE-P (PARAGRAPHP T)

    Write DOCSTRING to STREAM, [sanitizing the Markdown][7bf5] in it, performing [Codification][f1ab] and Linking, finally prefixing each line with INDENTATION. The prefix is not added to the first line if EXCLUDE-FIRST-LINE-P. If PARAGRAPHP, then add a newline before and after the output.

  • [function] ESCAPE-MARKDOWN STRING &KEY (ESCAPE-INLINE T) (ESCAPE-MATHJAX T) (ESCAPE-HTML T) (ESCAPE-BLOCK T)

    Backslash escape Markdown constructs in STRING.

    • If ESCAPE-INLINE, then escape the following characters:

        *_`[]\

    • If ESCAPE-MATHJAX, then escape $ characters.

    • If ESCAPE-HTML, then escape the following characters:

        <&

    • If ESCAPE-BLOCK, then escape whatever is necessary to avoid starting a new Markdown block (e.g. a paragraph, heading, etc).

  • [function] ENSURE-MD-PARAGRAPH STREAM

    Ensure that output previously written to STREAM is in a separate Markdown paragraph than the output that follows. Calling this function multiple times without writing anything else to STREAM in between is equivalent to calling it once.

11.4 Sections

[SECTION][5fac] objects rarely need to be dissected since [DEFSECTION][72b4] and DOCUMENT cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

  • [class] SECTION

    [DEFSECTION][72b4] stores its NAME, TITLE, PACKAGE, [READTABLE][d646] and ENTRIES arguments in [SECTION][5fac] objects.

  • [reader] SECTION-NAME [SECTION][5fac] (:NAME)

    The name of the global variable whose value is this [SECTION][5fac] object.

  • [reader] SECTION-PACKAGE [SECTION][5fac] (:PACKAGE)

    [*PACKAGE*][5ed1] will be bound to this package when generating documentation for this section.

  • [reader] SECTION-READTABLE [SECTION][5fac] (:READTABLE)

    [*READTABLE*][b79a] will be bound to this when generating documentation for this section.

  • [reader] SECTION-TITLE [SECTION][5fac] (:TITLE)

    A title or NIL. Used in generated documentation (see [Markdown Output][dd29]) and is returned by [DOCTITLE][e619] for [SECTION][5fac] objects and SECTION definitions.

  • [function] SECTION-LINK-TITLE-TO SECTION

  • [function] SECTION-ENTRIES SECTION

    A list of Markdown docstrings and XREFs in the order they occurred in [DEFSECTION][72b4].

11.5 Glossary Terms

[GLOSSARY-TERM][8251] objects rarely need to be dissected since [DEFINE-GLOSSARY-TERM][8ece] and DOCUMENT cover most needs. However, it is plausible that one wants to subclass them and maybe redefine how they are presented.

  • [class] GLOSSARY-TERM

    See [DEFINE-GLOSSARY-TERM][8ece].

  • [reader] GLOSSARY-TERM-NAME [GLOSSARY-TERM][8251] (:NAME)

    The name of the global variable whose value is this [GLOSSARY-TERM][8251] object.

  • [reader] GLOSSARY-TERM-TITLE [GLOSSARY-TERM][8251] (:TITLE)

    A title or NIL. Used in generated documentation (see [Markdown Output][dd29]) and is returned by [DOCTITLE][e619] for [GLOSSARY-TERM][8251] objects and GLOSSARY-TERM definitions.

  • [reader] GLOSSARY-TERM-URL [GLOSSARY-TERM][8251] (:URL)

    A string or NIL.

Truncated — view the full README on GitHub.

common-lisp
documentation-generator
documentation-tools
illiterate-programming
rubber-doc-development
untangled-lp

Similar repos

elixir-lang/ex_doc

ExDoc produces HTML, EPUB, and Markdown documentation for Erlang and Elixir projects

1,615

sphinx-doc/sphinx

The Sphinx documentation generator

8,023

atgreen/icl

Interactive Common Lisp: an enhanced REPL

201

machow/quartodoc

Generate API documentation with quarto

265

susam/emacs4cl

A tiny DIY kit to set up vanilla Emacs for Common Lisp programming

394

iamgio/quarkdown

🪐 Markdown with superpowers: from ideas to papers, presentations, websites, books, and knowledge…

16,210

jupyter/papyri

87

tefkah/emacs-docs

The documentation website Emacs deserves

356

Contributors

melisgl

982 commits

svetlyak40wt

2 commits

fare

1 commits

kingcons

1 commits

Languages

Common Lisp

87.1%

Emacs Lisp

10.3%

CSS

2.0%