Usage

To use this extension, first enable it on your conf.py file for Sphinx:

extensions += ["auto_intersphinx"]

You do not need to both enable this extension and sphinx.ext.intersphinx because this extension will load that automatically.

Once you have enabled the extension, create a single variable with the names of the packages you want to cross-reference:

auto_intersphinx_packages = ["sphinx", "requests", "packaging"]

Optionally, add a catalog file path (relative to conf.py) that will be used to cache any lookups we may have to do during Sphinx document building.

auto_intersphinx_catalog = "catalog.json"

How does it work?

Auto-intersphinx works by automatically populating intersphinx_mapping dictionary using URLs that are stored in the built-in JSON catalog of URLs or your own. If URLs for a package are not found on these catalogs, auto-intersphinx checks the installed Python packages and then tries to go online searching for those cross-references. Here is the rough algorithm, performed for each entry in the auto_intersphinx_packages list:

  1. If an intersphinx URL for the given package on the user catalog, that is used prioritarily, and the intersphinx_mapping for that package is filled and the algorithm continues with the next package.

  2. Else, we check the built-in catalog, distributed with auto-intersphinx. If an URL for the package is found there, then we fill-in intersphinx_mapping and continue.

  3. Else, we check the current Python environment, searching for the package metadata and attached documentation URLs the project may declare. Many projects declare their Sphinx documents this way. If an URL is found, it is checked for an objects.inv file existing there (not downloaded at this point), and if that exists, it added to the user catalog, allowing it to be added to the intersphinx_mapping. If successful, the algorithm then goes to the next package.

  4. If you get to this point, then auto-intersphinx will look for the package’s “versions” page in https://readthedocs.org. If one is found, then we download the information on that page, which includes documentation pointers and store these in the user catalog. We do not check for objects.inv file at those URLs since readthedocs documents are all Sphinx-compatible. If successful, the algorithm adds the intersphinx_mapping entry for the package, and then goes to the next package.

  5. Finally, if nothing else worked, we will look at the project metadata for the package at https://pypi.org. PyPI metadata often contains documentation URL for packages. If that is the case, we test it for the presence of an objects.inv file and, if successful, add that information to the user catalog. If successful, the algorithm populates the intersphinx_mapping, and goes to the next package.

    At this point, if we still do not find the relevant documentation for the package, auto-intersphinx will generate a build error which will cause Sphinx to exit with an error condition at the end of the processing.

By the end of the processing, if the user catalog was changed with respect to the state registered on the disk, and a filename has been defined by setting auto_intersphinx_catalog, it is saved back to that file (a backup is created if needed), so that the next lookup will not require probing remote resources. You may commit this JSON file to your repository and keep it there.

The Catalog

The lookup algorithm above may be optimised/improved/fixed in several ways. To avoid unnecessary lookups, it is recommended to setup your own catalog of entries to covering all your documentation needs. Setting up auto_intersphinx_catalog may help you kick-starting this work.

Moreover, often, names of packages as they are known in Python, at readthedocs or PyPI, may differ from the package name you typically use while importing such package. This may lead the lookup algorithm to search for incorrect entries in these environments. Here is a list of issues one can typically find while trying to create a coherent set of intersphinx_mapping entries:

  • Package names may be different than what you use for importing them into your code. For example, you may import docker in your code, however the readthedocs entry for this package lives under the name docker-py. The PyPI version of the package is placed under docker. The famous machine learning package pytorch is distributed as torch inside PyPI.

  • Packages may not explicitly define their documentation URLs, or be available on readthedocs_. For example, pytorch has a broken set of links on readthedocs, and self-hosts their own documentation.

  • Packges may not be available while you build your documentation, and therefore may not be looked-up at the current environment.

The URL catalog format has been designed to accomodate these exceptions and skip automated (online) searches if possible. It is a JSON file that contains a dictionary, mapping names you’d use as inputs in auto_intersphinx_packages to another dictionary composed of two keys: versions and sources. Here is an example:

{
  "click": {
    "versions": {
      "latest": "https://click.palletsprojects.com/en/latest/",
      "8.1.x": "https://click.palletsprojects.com/en/8.1.x/",
      "8.0.x": "https://click.palletsprojects.com/en/8.0.x/",
    },
    "sources": {}
  },
  "docker": {
    "versions": {
      "latest": "https://docker-py.readthedocs.io/en/latest/",
      "stable": "https://docker-py.readthedocs.io/en/stable/",
      "5.0.3": "https://docker-py.readthedocs.io/en/5.0.3/",
      "5.0.2": "https://docker-py.readthedocs.io/en/5.0.2/",
      "5.0.1": "https://docker-py.readthedocs.io/en/5.0.1/",
      "5.0.0": "https://docker-py.readthedocs.io/en/5.0.0/",
    },
    "sources": {
      "readthedocs": "docker-py"
    }
  },
  "numpy": {
    "versions": {
      "latest": "https://numpy.org/devdocs/",
      "stable": "https://numpy.org/doc/stable/",
      "1.23.x": "https://numpy.org/doc/1.23/",
      "1.22.x": "https://numpy.org/doc/1.22/",
      "1.21.x": "https://numpy.org/doc/1.21/"
    },
    "sources": {
      "pypi": "numpy"
    }
  }
}

The versions entry of each package determine version name to URL mappings. When you do not explicitly request a particular version number catalog to be linked for a package, the stable (or latest) versions are used. The sources entry indicate where information for this package can be found. Examples are environment, readthedocs or pypi. The values correspond to the name that should ber looked up on those services for finding information about that package.

In the example above, the package click has 3 versions encoded in the catalog, the user can ask for. Furthermore, no online services have information about this package (probably the user hard-coded those URLs after inspection). The various versions of the package docker documentation may be obtained on readthedocs under the name docker-py. Finally, numpy information may be obtained at PyPI, under the name numpy.

Bootstrapping a Catalog

A new user catalog is typically saved on the directory containing the file conf.py, if the user specified the Sphinx configuration parameter auto_intersphinx_catalog with a relative path name (e.g. catalog.json). However, the user may create and maintain their own catalog file using command-line utilities shiped with this package.

To bootstrap a new catalog from scratch, specifying the list of packages to lookup and populate it with, and use the program update-catalog:

auto-intersphinx-update-catalog -vvv --catalog=catalog.json numpy requests click

This will create a new file called catalog.json on your directory, and will try to apply the lookup algorithm explained above to find sources of documentation for these packages. You may hand-edit or improve this catalog later.

Tip

You may bootstrap a new catalog from either a list of packages, as above, or by providing the path (or URL) of a pip-requirements style file. The command-line application auto-intersphinx-update-catalog will parse such a file, search for resources, and fill-in a catalog.

Tip

Read the application documentation with auto-intersphinx-update-catalog --help for usage instructions.

Updating a Catalog

To update an existing catalog, use the command-line application update-catalog:

auto-intersphinx update-catalog -vvv --self --catalog=catalog.json

This will read the current information available in the existing catalog, and will search the sources once more for updated information. Naturally, packages with no sources (empty sources) entries, will not be updated.

Browsing for Documentation

You may ask the command-line application check-packages to display where documentation information may be found for a package. For example:

$ auto-intersphinx check-packages numpy
Found numpy in builtin catalog:
| {
|   "latest": "https://numpy.org/devdocs/",
|   "stable": "https://numpy.org/doc/stable/",
|   "1.23.4": "https://numpy.org/doc/1.23/",
|   "1.23.x": "https://numpy.org/doc/1.23/",
|   "1.22.x": "https://numpy.org/doc/1.22/",
|   "1.21.x": "https://numpy.org/doc/1.21/",
|   "1.20.x": "https://numpy.org/doc/1.20/"
| }

You may ask this tool to keep going and find all sources of information for a package:

$ auto-intersphinx-check-packages numpy --keep-going
Found numpy in builtin catalog:
| {
|   "latest": "https://numpy.org/devdocs/",
|   "stable": "https://numpy.org/doc/stable/",
|   "1.23.4": "https://numpy.org/doc/1.23/",
|   "1.23.x": "https://numpy.org/doc/1.23/",
|   "1.22.x": "https://numpy.org/doc/1.22/",
|   "1.21.x": "https://numpy.org/doc/1.21/",
|   "1.20.x": "https://numpy.org/doc/1.20/"
| }
Found numpy documentation in readthedocs.org:
| {
|   "latest": "https://numpy.readthedocs.io/en/latest/",
|   "main": "https://numpy.readthedocs.io/en/main/"
| }
Looking up all PyPI versions of numpy - this may be long...
Found numpy documentation in PyPI:
| {
|   "1.23.4": "https://numpy.org/doc/1.23/"
| }

Which may display more information about the package’s various sources. The flag --keep-going may also be used at auto-intersphinx-update-catalog to complete information about a package. While here the versions dictionary are displayed in sections, while updating the catalogs, the resulting versions dictionaries are merged together. Of course, you still have the option to copy-and-paste information from the above output, directly to your catalog.

Inspecting intersphinx Inventories

It is sometimes useful to inspect the contents of objects.inv files available locally or remotely. To this end, we provide a CLI application called dump-objects, that can be used to dump the contents of both local or remote inventories. Read the application documentation with --help, to discover usage examples.