Commit 3df5affb authored by Jonathan Corbet's avatar Jonathan Corbet
Browse files

Merge branch 'build-script' into docs-mw

Quoth Mauro:

This series should probably be called:

    "Move the trick-or-treat build hacks accumulated over time
     into a single place and document them."

as this reflects its main goal. As such:

- it places the jobserver logic on a library;
- it removes sphinx/parallel-wrapper.sh;
- the code now properly implements a jobserver-aware logic
  to do the parallelism when called via GNU make, failing back to
  "-j" when there's  no jobserver;
- converts check-variable-fonts.sh to Python and uses it via
  function call;
- drops an extra script to generate man pages, adding a makefile
  target for it;
- ensures that return code is 0 when PDF successfully builds;
- about half of the script is comments and documentation.

I tried to do my best to document all tricks that are inside the
script. This way, the docs build steps is now documented.

It should be noticed that it is out of the scope of this series
to change the implementation. Surely the process can be improved,
but first let's consolidate and document everything on a single
place.

Such script was written in a way that it can be called either
directly or via a Makefile. Running outside Makefile is
interesting specially when debug is needed. The command line
interface replaces the need of having lots of env vars before
calling sphinx-build:

    $ ./tools/docs/sphinx-build-wrapper --help
    usage: sphinx-build-wrapper [-h]
	   [--sphinxdirs SPHINXDIRS [SPHINXDIRS ...]] [--conf CONF]
	   [--builddir BUILDDIR] [--theme THEME] [--css CSS] [--paper {,a4,letter}] [-v]
	   [-j JOBS] [-i] [-V [VENV]]
	   {cleandocs,linkcheckdocs,htmldocs,epubdocs,texinfodocs,infodocs,mandocs,latexdocs,pdfdocs,xmldocs}

    Kernel documentation builder

    positional arguments:
      {cleandocs,linkcheckdocs,htmldocs,epubdocs,texinfodocs,infodocs,mandocs,latexdocs,pdfdocs,xmldocs}
			    Documentation target to build

    options:
      -h, --help            show this help message and exit
      --sphinxdirs SPHINXDIRS [SPHINXDIRS ...]
			    Specific directories to build
      --conf CONF           Sphinx configuration file
      --builddir BUILDDIR   Sphinx configuration file
      --theme THEME         Sphinx theme to use
      --css CSS             Custom CSS file for HTML/EPUB
      --paper {,a4,letter}  Paper size for LaTeX/PDF output
      -v, --verbose         place build in verbose mode
      -j, --jobs JOBS       Sets number of jobs to use with sphinx-build
      -i, --interactive     Change latex default to run in interactive mode
      -V, --venv [VENV]     If used, run Sphinx from a venv dir (default dir: sphinx_latest)

the only mandatory argument is the target, which is identical with
"make" targets.

The call inside Makefile doesn't use the last four arguments. They're
there to help identifying problems at the build:

    -v makes the output verbose;
    -j helps to test parallelism;
    -i runs latexmk in interactive mode, allowing to debug PDF
       build issues;
    -V is useful when testing it with different venvs.

When used with GNU make (or some other make which implements jobserver),
a call like:

    make -j <targets> htmldocs

will make the wrapper to automatically use POSIX jobserver to claim
the number of available job slots, calling sphinx-build with a
"-j" parameter reflecting it. ON such case, the default can be
overriden via SPHINXDIRS argument.

Visiable changes when compared with the old behavior:

When V=0, the only visible difference is that:
- pdfdocs target now returns 0 on success, 1 on failures.
  This addresses an issue over the current process where we
  it always return success even on failures;
- it will now print the name of PDF files that failed to build,
  if any.

In verbose mode, sphinx-build-wrapper and sphinx-build command lines
are now displayed.
parents d0841b87 e123e00a
Loading
Loading
Loading
Loading
+34 −120
Original line number Diff line number Diff line
@@ -23,164 +23,76 @@ SPHINXOPTS =
SPHINXDIRS    = .
DOCS_THEME    =
DOCS_CSS      =
_SPHINXDIRS   = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
SPHINX_CONF   = conf.py
RUSTDOC       =
PAPER         =
BUILDDIR      = $(obj)/output
PDFLATEX      = xelatex
LATEXOPTS     = -interaction=batchmode -no-shell-escape

PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)

# Wrapper for sphinx-build

BUILD_WRAPPER = $(srctree)/tools/docs/sphinx-build-wrapper

# For denylisting "variable font" files
# Can be overridden by setting as an env variable
FONTS_CONF_DENY_VF ?= $(HOME)/deny-vf

ifeq ($(findstring 1, $(KBUILD_VERBOSE)),)
SPHINXOPTS    += "-q"
endif

# User-friendly check for sphinx-build
HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)

ifneq ($(wildcard $(srctree)/.config),)
ifeq ($(CONFIG_RUST),y)
	RUSTDOC=--rustdoc
endif
endif

ifeq ($(HAVE_SPHINX),0)

.DEFAULT:
	$(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
	@echo
	@$(srctree)/scripts/sphinx-pre-install
	@$(srctree)/tools/docs/sphinx-pre-install
	@echo "  SKIP    Sphinx $@ target."

else # HAVE_SPHINX

# User-friendly check for pdflatex and latexmk
HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
# Common documentation targets
htmldocs mandocs infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
	$(Q)PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
		$(srctree)/tools/docs/sphinx-pre-install --version-check
	+$(Q)PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
		$(PYTHON3) $(BUILD_WRAPPER) $@ \
		--sphinxdirs="$(SPHINXDIRS)" $(RUSTDOC) \
		--builddir="$(BUILDDIR)" --deny-vf=$(FONTS_CONF_DENY_VF) \
		--theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)

ifeq ($(HAVE_LATEXMK),1)
	PDFLATEX := latexmk -$(PDFLATEX)
endif #HAVE_LATEXMK

# Internal variables.
PAPEROPT_a4     = -D latex_elements.papersize=a4paper
PAPEROPT_letter = -D latex_elements.papersize=letterpaper
ALLSPHINXOPTS   = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
ALLSPHINXOPTS   += $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
ifneq ($(wildcard $(srctree)/.config),)
ifeq ($(CONFIG_RUST),y)
	# Let Sphinx know we will include rustdoc
	ALLSPHINXOPTS   +=  -t rustdoc
endif
endif
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .

# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;

# $2 sphinx builder e.g. "html"
# $3 name of the build subfolder / e.g. "userspace-api/media", used as:
#    * dest folder relative to $(BUILDDIR) and
#    * cache folder relative to $(BUILDDIR)/.doctrees
# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man
# $5 reST source folder relative to $(src),
#    e.g. "userspace-api/media" for the linux-tv book-set at ./Documentation/userspace-api/media

PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)

quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
      cmd_sphinx = \
	PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
	BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(src)/$5/$(SPHINX_CONF)) \
	$(PYTHON3) $(srctree)/scripts/jobserver-exec \
	$(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \
	$(SPHINXBUILD) \
	-b $2 \
	-c $(abspath $(src)) \
	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
	-D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
	$(ALLSPHINXOPTS) \
	$(abspath $(src)/$5) \
	$(abspath $(BUILDDIR)/$3/$4) && \
	if [ "x$(DOCS_CSS)" != "x" ]; then \
		cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(DOCS_CSS)) $(BUILDDIR)/$3/_static/; \
	fi

htmldocs:
	@$(srctree)/scripts/sphinx-pre-install --version-check
	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
# The following targets are independent of HAVE_SPHINX, and the rules should
# work or silently pass without Sphinx.

htmldocs-redirects: $(srctree)/Documentation/.renames.txt
	@tools/docs/gen-redirects.py --output $(BUILDDIR) < $<

# If Rust support is available and .config exists, add rustdoc generated contents.
# If there are any, the errors from this make rustdoc will be displayed but
# won't stop the execution of htmldocs

ifneq ($(wildcard $(srctree)/.config),)
ifeq ($(CONFIG_RUST),y)
	$(Q)$(MAKE) rustdoc || true
endif
endif

texinfodocs:
	@$(srctree)/scripts/sphinx-pre-install --version-check
	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))

# Note: the 'info' Make target is generated by sphinx itself when
# running the texinfodocs target define above.
infodocs: texinfodocs
	$(MAKE) -C $(BUILDDIR)/texinfo info

linkcheckdocs:
	@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))

latexdocs:
	@$(srctree)/scripts/sphinx-pre-install --version-check
	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))

ifeq ($(HAVE_PDFLATEX),0)

pdfdocs:
	$(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
	@echo "  SKIP    Sphinx $@ target."

else # HAVE_PDFLATEX

pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
pdfdocs: latexdocs
	@$(srctree)/scripts/sphinx-pre-install --version-check
	$(foreach var,$(SPHINXDIRS), \
	   $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; \
	   mkdir -p $(BUILDDIR)/$(var)/pdf; \
	   mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUILDDIR)/$(var)/pdf/; \
	)

endif # HAVE_PDFLATEX

epubdocs:
	@$(srctree)/scripts/sphinx-pre-install --version-check
	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))

xmldocs:
	@$(srctree)/scripts/sphinx-pre-install --version-check
	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))

endif # HAVE_SPHINX

# The following targets are independent of HAVE_SPHINX, and the rules should
# work or silently pass without Sphinx.

refcheckdocs:
	$(Q)cd $(srctree);scripts/documentation-file-ref-check

cleandocs:
	$(Q)rm -rf $(BUILDDIR)

# Used only on help
_SPHINXDIRS   = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))

dochelp:
	@echo  ' Linux kernel internal documentation in different formats from ReST:'
	@echo  '  htmldocs        - HTML'
	@echo  '  htmldocs-redirects - generate HTML redirects for moved pages'
	@echo  '  texinfodocs     - Texinfo'
	@echo  '  infodocs        - Info'
	@echo  '  mandocs         - Man pages'
	@echo  '  latexdocs       - LaTeX'
	@echo  '  pdfdocs         - PDF'
	@echo  '  epubdocs        - EPUB'
@@ -194,11 +106,13 @@ dochelp:
	@echo  '  make SPHINXDIRS="s1 s2" [target] Generate only docs of folder s1, s2'
	@echo  '  valid values for SPHINXDIRS are: $(_SPHINXDIRS)'
	@echo
	@echo  '  make SPHINX_CONF={conf-file} [target] use *additional* sphinx-build'
	@echo  '  configuration. This is e.g. useful to build with nit-picking config.'
	@echo
	@echo  '  make DOCS_THEME={sphinx-theme} selects a different Sphinx theme.'
	@echo
	@echo  '  make DOCS_CSS={a .css file} adds a DOCS_CSS override file for html/epub output.'
	@echo
	@echo  '  make PAPER={a4|letter} Specifies the paper size used for LaTeX/PDF output.'
	@echo
	@echo  '  make FONTS_CONF_DENY_VF={path} sets a deny list to block variable Noto CJK fonts'
	@echo  '  for PDF build. See tools/docs/lib/latex_fonts.py for more details'
	@echo
	@echo  '  Default location for the generated documents is Documentation/output'
+5 −10
Original line number Diff line number Diff line
@@ -18,8 +18,6 @@ import sphinx
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath("sphinx"))

from load_config import loadConfig               # pylint: disable=C0413,E0401

# Minimal supported version
needs_sphinx = "3.4.3"

@@ -93,8 +91,12 @@ def config_init(app, config):
    # LaTeX and PDF output require a list of documents with are dependent
    # of the app.srcdir. Add them here

    # When SPHINXDIRS is used, we just need to get index.rst, if it exists
    # Handle the case where SPHINXDIRS is used
    if not os.path.samefile(doctree, app.srcdir):
        # Add a tag to mark that the build is actually a subproject
        tags.add("subproject")

        # get index.rst, if it exists
        doc = os.path.basename(app.srcdir)
        fname = "index"
        if os.path.exists(os.path.join(app.srcdir, fname + ".rst")):
@@ -583,13 +585,6 @@ pdf_documents = [
kerneldoc_bin = "../scripts/kernel-doc.py"
kerneldoc_srctree = ".."

# ------------------------------------------------------------------------------
# Since loadConfig overwrites settings from the global namespace, it has to be
# the last statement in the conf.py file
# ------------------------------------------------------------------------------
loadConfig(globals())


def setup(app):
    """Patterns need to be updated at init time on older Sphinx versions"""

+16 −13
Original line number Diff line number Diff line
@@ -579,20 +579,23 @@ source.
How to use kernel-doc to generate man pages
-------------------------------------------

If you just want to use kernel-doc to generate man pages you can do this
from the kernel git tree::
To generate man pages for all files that contain kernel-doc markups, run::

  $ scripts/kernel-doc -man \
    $(git grep -l '/\*\*' -- :^Documentation :^tools) \
    | scripts/split-man.pl /tmp/man
  $ make mandocs

Some older versions of git do not support some of the variants of syntax for
path exclusion.  One of the following commands may work for those versions::
Or calling ``script-build-wrapper`` directly::

  $ scripts/kernel-doc -man \
    $(git grep -l '/\*\*' -- . ':!Documentation' ':!tools') \
    | scripts/split-man.pl /tmp/man
  $ ./tools/docs/sphinx-build-wrapper mandocs

  $ scripts/kernel-doc -man \
    $(git grep -l '/\*\*' -- . ":(exclude)Documentation" ":(exclude)tools") \
    | scripts/split-man.pl /tmp/man
The output will be at ``/man`` directory inside the output directory
(by default: ``Documentation/output``).

Optionally, it is possible to generate a partial set of man pages by
using SPHINXDIRS:

  $ make SPHINXDIRS=driver-api/media mandocs

.. note::

   When SPHINXDIRS={subdir} is used, it will only generate man pages for
   the files explicitly inside a ``Documentation/{subdir}/.../*.rst`` file.
+2 −2
Original line number Diff line number Diff line
@@ -106,7 +106,7 @@ There's a script that automatically checks for Sphinx dependencies. If it can
recognize your distribution, it will also give a hint about the install
command line options for your distro::

	$ ./scripts/sphinx-pre-install
	$ ./tools/docs/sphinx-pre-install
	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
	Warning: better to also install "texlive-luatex85".
	You should run:
@@ -116,7 +116,7 @@ command line options for your distro::
		. sphinx_2.4.4/bin/activate
		pip install -r Documentation/sphinx/requirements.txt

	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
	Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-pre-install line 468.

By default, it checks all the requirements for both html and PDF, including
the requirements for images, math expressions and LaTeX build, and assumes
+1 −1
Original line number Diff line number Diff line
@@ -220,7 +220,7 @@
	    If you want them, please install non-variable ``Noto Sans CJK''
	    font families along with the texlive-xecjk package by following
	    instructions from
	    \sphinxcode{./scripts/sphinx-pre-install}.
	    \sphinxcode{./tools/docs/sphinx-pre-install}.
	    Having optional non-variable ``Noto Serif CJK'' font families will
	    improve the looks of those translations.
	\end{sphinxadmonition}}
Loading