Sphinx supports parallelized build (-j) option. Setting it to `auto`
makes use of all available cores. This option seems to speed up the
build significantly on multi-core machines.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Fixes: #31486
Sourcing `cmake/extensions.cmake` to have Zephyr functions available
when including Zephyr CMake files into doc build system.
This is a follow up on #30904 by supporting generated Kconfig files
that might contain lines as `osource "$(ZEPHYR_<MODULE_NAME>_KCONFIG)"`
to be properly parsed for doc building.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Fix documentation build issue where generating the kconfig rST was
failing due to missing Kconfig.shield* files. Add an extra step in
CMakeLists.txt to properly generate them, similarly to what is already
done for Kconfig.soc*.
Signed-off-by: Fabio Utzig <fabio.utzig@nordicsemi.no>
Add generated documentation content for known devicetree bindings
using the Binding abstraction which was just added to edtlib. This
works similarly to the way Kconfig content is generated, so extract a
bit of common helper code for doing that out and rename the relevant
files to keep the distinction clear.
Make the documentation build system respect a preset DTS_ROOT. In this
way, out of tree bindings can be added to the generated content by
telling the documentation build system where to find them, identically
to how out of tree bindings can be added to a Zephyr application.
Similarly, make the output directory configurable.
Fixes: #28865
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Rather than doing a doc build on every PR, limit it to PRs that either
touch a file in doc/, *.rst and what's listed as DOXY_SOURCES in
doc/CMakeLists.txt.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Fixes commit 15fbf707ca ("doc: revert to copy files with
extract_content.py directly") that removed (2 years ago!) the definition
of EXTRACT_CONTENT_OUTPUTS while leaving it in use by sphinx-html and
sphinx-latex.
On my Linux box "make sphinx-html" is more than 5 times faster than
"htmldocs", likely because it does not build doxygen nor kconfig.
Also add missing WORKING_DIRECTORY for consistency with regular
targets and to fix this error:
Exception occurred:
File "~/.local/lib/python3.8/site-packages/sphinx/
search/__init__.py", line 284, in __init__
with open(scoring, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'scorer.js'
I don't know when that second regression appeared.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
This commit introduces support for multiple SOC_ROOT.
This means that additional SOC_ROOTs specified using -DSOC_ROOT as
argument to CMake will be forming a list together with ${ZEPHYR_BASE}.
This allows for greater flexibility, as developers can now specify
multiple out-of-tree SoCs and not worry about the SoC used for the
board they compile for.
Also it avoid code, such as:
if(BOARD STREQUAL my_board_using_out_of_tree_soc)
set(SOC_ROOT some/out/of/tree/soc/path)
endif()
in application CMakeLists.txt.
Finally, allowing multiple SOC_ROOTs prepares for specifying SOC_ROOTs
in Zephyr modules.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This provides a better error message when building with CMake and
forgetting ZEPHYR_BASE or not registering Zephyr in the CMake package
registry. See parent commit for more details (split from parent for
better readability).
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Fixes: #24903
This commit now includes ${ZEPHYR_BASE}/cmake/python.cmake to locate
python3.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
Sometimes you are only interested in doxygen, so need to wait 10 minutes
for everything to generate in this case. Now just do:
make doxygen
and get only the doxygen output.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
With the introduction of ZephyrConfig.cmake all parts of CMake code
should rely on the CMake ZEPHYR_BASE variable instead of the environment
setting.
This ensures that after the first CMake invocation, then all subsequent
invocation in same build folder will use same zephyr base.
Signed-off-by: Torsten Rasmussen <Torsten.Rasmussen@nordicsemi.no>
This was still set to 3.4, and does not include REQUIRED to fail the
build in case of an old version.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Add an option --separate-all-index that makes genrest.py generate a
separate index-all.rst index page that lists all symbols, instead of
listing all symbols in index.rst. index-all.rst is linked from
index.rst.
This was originally motivated by an external project where index.rst
becomes the top-level page, which runs into a Sphinx bottleneck with
sphinx_rtd_theme. See https://github.com/sphinx-doc/sphinx/issues/6909.
This turned out pretty nice after some feedback from Ruth Fuchss, so use
it for Zephyr too.
Also unclutter the generated documentation a bit by removing some
headings. This makes the navigation menu on the left nicer too.
Piggyback making genrest.py executable, which is handy when running it
manually.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Using ':' as the separator in --modules breaks module specifications
like
nrfxlib:nrfxlib:C:/Users/Carles/src/ncs/nrfxlib
The ':' in 'C:' gets seen as a separator.
Use ',' instead, which is unlikely to appear in paths (at least in
--modules). Treat a doubled ',,' as a literal ','.
Another option would be ';', but it clashes with how CMake represents
lists, which is awkward.
Also make quoting of arguments with spaces more robust by passing
VERBATIM to add_custom_target().
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Use --modules to split the Kconfig reference up by where symbols are
defined. See the argparse docstring for --modules in
doc/scripts/genrest.py.
Not sure what the best way to split things up is, so feedback would be
appreciated. I just pulled out some top-level directories. It could
always be tweaked later.
If a symbol is defined in more than one module (by being defined in
multiple locations), it appears on multiple index pages.
To build the documentation, do
$ mkdir doc/b && cd doc/b
$ cmake -GNinja ..
$ ninja htmldocs
(output in html/reference/kconfig/index.html)
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Need to add the kernel_arch_interface.h header here
to match zephyr.doxyfile.in.
This is the header which contains the formal kernel-to-
architecture interface and we want docs generated for it.
Signed-off-by: Andrew Boie <andrew.p.boie@intel.com>
Paths like soc//arm/... showed up in the output due to the extra '/' at
the end of the SOC_DIR value. Remove the redundant '/' in SOC_DIR and
other environment variables that get referenced in the Kconfig files.
Kconfiglib never normalizes paths, because it avoids some gotchas, e.g.
with symlinks.
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Add a --modules flag to genrest.py for generating separate index pages
for symbols defined within certain paths.
Passing
--modules Shell🐚subsys/shell Storage:storage:subsys/storage
will generate these index pages, instead of a single index.rst file:
- index-shell.rst: Lists the symbols defined in subsys/shell
- index-storage.rst: Lists the symbols defined in subsys/storage
- index-main.rst: Lists all symbols that are not in subsys/shell or
subsys/storage
- index-all.rst: Lists all symbols
- index.rst: Contains links to the other index pages
The string before the first ':' ('Shell' and 'Storage' above) is used
when generating the title of the index page. 'Shell' gives
'Shell Configuration Options', for example.
The title for index-main.rst can be set by passing
'--non-module-title <title>'. It defaults to "Zephyr".
By default, paths in symbol information pages that are within modules
are shown as '<title>/<path relative to module>'. This can be disabled
by passing --keep-module-paths.
--keep-module-paths would make sense for the example above, but
stripping the path to the module is nice when dealing with modules
defined outside the Zephyr repository.
If a symbol is defined in multiple modules (or both in a module and
outside all modules), it will appear on multiple index pages.
This commit also simplifies how genrest.py is called a bit, making the
Kconfig filename optional (default: Kconfig).
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Correctly set the separator as a semicolon on Windows when
constructing the PYTHONPATH environment variable so that CMake
doesn't intepret it and swallow it.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
The main change is the elimination of the bootstrapper, a design flaw
/ misfeature.
Update the documentation to be compatible with the 0.6.x releases as
well. This has to be done atomically, as there were incompatible
changes. Make use of the versionchanged and versionadded directives
to begin keeping track of how these APIs are evolving.
(Note that west 0.6.0 will remain compatible with the extension
commands in Zephyr v1.14 LTS as long as that is still alive. This
change is targeted towards Zephyr 2.0 users.)
This requires a bump in the shippable container and allows us to
simplify the west_commands test procedure.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Clean up the INPUT paths for doxygen scanning:
- directory scans include sub-directories so no need
to include them explicitly
- remove excluded (legacy) includes that no longer exist
Sync directory changes in the CMakeLists.txt (as noted in the comments)
Add content to the home page of the doxygen-generated HTML, and add a
Zephyr branding theme.
Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
Fixes: #6773
The module processing logic in Zephyr requires Python, since it uses a
Python script to locate and process the modules. It is therefore
required to find the Python executable using CMake before actually
including the CMake script that processes the modules. In order to
prevent this from happening again, move the inclusion of version.cmake
as well in case that script chooses to use Python in the future.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
With the addition of a local CMake variable ZEPHYR_BASE that mirrors the
value of the environment variable the documentation build broke on
Windows. This is because CMake was incorrectly interpreting backslashes
present in the Windows path as escaping characters. In order to avoid
this, use CMake's built-in file(TO_CMAKE_PATH) functionality, which
converts the path to a canonical version using forward slashes only.
At the same time rework boilerplate.cmake to use the same mechanism
instead of manually replacing backslashes and removing trailing ones.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
This fixes a corner case.
The ZEPHYR_BASE variable might be set in the environment at generation
time, but unset at build time. Save it in the cmake -E env calls to
scripts that need it.
Signed-off-by: Marti Bolivar <marti.bolivar@nordicsemi.no>
Update the files which contain no license information with the
'Apache-2.0' SPDX license identifier. Many source files in the tree are
missing licensing information, which makes it harder for compliance
tools to determine the correct license.
By default all files without license information are under the default
license of Zephyr, which is Apache version 2.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
When generating the Kconfig reference documentation, enable the warnings
for undefined Kconfig options in order to make sure all references are
met.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
In order to parse all the Kconfig files that make up the whole Kconfig
tree we need to process the modules, since this can optionally provide
Kconfig files. In order to do so, include the modules CMake file so that
they are listed and a Kconfig.modules is generated.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Currently, the Kconfig.modules file is placed in the build directory
relative to the CMake "project". But technically, the file is not
project-specific, but global, or build-directory specific.
So we move it up one level to the CMAKE_BINARY_DIR instead. Currently,
there is only one project, so this change has no effect, but this
enables us to have multiple projects in the future, which again
enables multi-image builds.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
Signed-off-by: Håkon Øye Amundsen <haakon.amundsen@nordicsemi.no>
Avoids having it look like a Kconfig symbol reference (which gets
detected by an upcoming CI check).
Signed-off-by: Ulf Magnusson <Ulf.Magnusson@nordicsemi.no>
Do not load the GENERATED_DTS_BOARD_CONF if in doc mode, since it will
not defined as it doesn't apply. No need to defined it to a dummy value.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Use file(GLOB_RECURSE ...) to generate a superset of doxygen input
files. Use this list as a dependency for a new empty file
doc/_build/last_doxy_run_tstamp.
The incremental build time to re-run "make htmldocs-fast" after a small
.rst doc change is brought down from from ~8s to ~4s on my Linux system
and from ~30s to less than 10s on my MacBook.
(Doxygen changes are still slow for a couple reasons recently isolated
and documented in doc/CMakeLists.txt)
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Clearly shows where time is spent. More specifically helps work on the
next commit while removing noise from it.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Use a variable to set the doxygen output directory so we can refer to it
later when fixing mtimes instead of hardcoding it.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Doxygen doesn't support incremental builds. It could be ok because it
doesn't take much time in this codebase. However it's not because it
makes old output look new which has a cascade effect on sphinx:
https://sourceforge.net/p/doxygen/mailman/message/36580807/
Make doxygen artificially smarter by saving and restoring modify
timestamps on the filesystem for doxygen output files that haven't
changed.
On my system this brings down the time to run an incremental "make
htmldocs" from 75s down to 8s (cmake -DKCONFIG_TURBO_MODE=1 used in both
cases)
This optimization speeds-up non-doxygen documentation work only.
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Make kconfigfunctions.py agnostic to where GENERATED_DTS_BOARD_CONF is
located.
We don't want to encode the path of GENERATED_DTS_BOARD_CONF into more
places than necessary. Also, re-using GENERATED_DTS_BOARD_CONF makes
it easier to change where generated_dts_board.conf is located.
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
This file is now needed by Kconfig and was not created during doc
creation. So touch an empty file to make doc build go through.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
On (at least) Fedora the /usr/bin/sphinx-build binary always belong to
python2-sphinx which results in a confusing "cannot import name
DEVNULL" error below (DEVNULL is available from python 3.3).
Tell find_program() to look for sphinx-build-3 first and then
sphinx-build next.
(shorten) Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/sphinx/config.py",
line 161, in __init__ execfile_(filename, config)
...
File "conf.py", line 17, in <module>
from subprocess import CalledProcessError, check_output, DEVNULL
ImportError: cannot import name DEVNULL
Signed-off-by: Marc Herbert <marc.herbert@intel.com>
Introduces the ARCH_ROOT argument, similar to BOARD_ROOT and SOC_ROOT.
This enables support for out-of-tree architectures.
The ARCH_ROOT out-of-tree layout is expected to be the following:
* ${ARCH_ROOT}/arch/${ARCH}/
* ${ARCH_ROOT}/include/arch/${ARCH}/ (Optional)
Signed-off-by: Klaus Petersen <kape@oticon.com>
Export a new KCONFIG_DOC_MODE environment variable when building the doc
and invoking Kconfig, so that the functions that expect a build folder
can accordingly return a hardcoded value.
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Building the documentation for all the Kconfig options significantly
adds to the total doc build time. When making and testing major changes
to the documentation, we provide an option to temporarily stub-out
the auto-generated configuration documentation so the doc build process
runs much faster.
To enable this mode, set the following option when invoking cmake
-DKCONFIG_TURBO_MODE=1
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
The 'doc' build has been invoking 'project' with the argument "VERSION
${PROJECT_VERSION}". This is cargo-cult code inherited from the
toplevel build system.
This can be safely removed because in the doc build the versioning
variables that are set by 'project' are unused and PROJECT_VERSION is
not available to be de-referenced at that point in time any way ...
This fixes#12282
Signed-off-by: Sebastian Bøe <sebastian.boe@nordicsemi.no>
Move to latest cmake version with many bug fixes and enhancements.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
Adding ninja targets based on the files that are to be copied into the
build folder doesn't seem to speed up the build, so simplify the CMake
script by relying on extract_content.py to do the copy.
Numbers on my machine:
* master
clean: 6m45.541s, 6m8.113s
incremental: 1m24.233s, 1m32.720s
* revert
clean: 6m25.221s, 6m19.751s
incremental:1m20.893s, 1m20.337s
Signed-off-by: Carles Cufi <carles.cufi@nordicsemi.no>
In a clean build, Ninja is creating copies of the extracted
documentation outputs regardless of whether they already exist or
not. I can't explain this behavior from the documentation of the build
command, but it's so, even if extract_content.py is patched to use
copyfile() instead of copy2() so that outputs (copied .rst files in
the build directory) have strictly newer mtimes than inputs (source
tree .rst files).
This extra copying doesn't happen for incremental builds, for some
reason.
To account for this behavior, add and use an extract_content.py option
that skips content extraction completely and leaves that all up to the
add_custom_command() which tracks each individual input->output build
command.
Reported-by: Carles Cufi <carles.cufi@nordicsemi.no>
Signed-off-by: Marti Bolivar <marti@foundries.io>
Most of time (i.e. when just writing to .rst files or editing images,
rather than editing Doxygen or Kconfig), a full rebuild of the docs
pipeline is not necessary.
This is unfortunate, because it takes over a minute to re-run doxygen
etc. (on my system), whereas an incremental sphinx-build run should
only take a few seconds.
As a band-aid to help developers who have already generated a docs
build and know the consequences of their actions, add "sphinx-html"
and "sphinx-latex" targets. These just re-extract content files into
the build directory and run sphinx-build. This makes for much faster
documentation edit/compile/read loops.
Add some extra COMMENTs while we're here, to make it more obvious
what's going on when the docs build system crunches on something.
A "proper" solution for changing "ninja html" to have similar
performance would need, at least, to:
- only re-run doxygen if any source files have changed
- only re-generate Kconfig if any of those files have changed
That's all left to future work, as this is good enough to make
rebuilding the docs bearable for me.
Signed-off-by: Marti Bolivar <marti@foundries.io>
The logic which copies source documentation files into the build
directory could use some improvements to its dependency management, so
that when a source file changes, extract_content.py gets re-run.
Make these changes as follows:
- Add an --outputs flag to extract_content.py, so that the
sources it depends on and generates can be saved into a file and
thus made known to the build system
- Change the way the sources and destination are specified in the
extract_content.py command line so that the entire job can be done
in a single command, rather than multiple (to avoid having to
collate multiple --outputs files in CMake)
- Extract the content at configure time with execute_process(),
tracking all inputs and outputs within the build system itself. Use
this information to make sure that each individual output depends on
just its exact input file, ensuring updated inputs produce updated
outputs without having to call extract_content.py again.
- Ensure that the "content" build system target depends on all the
outputs, transitively triggering a rebuild any time an input
file (e.g. .rst documentation file or included image/source file)
changes.
Signed-off-by: Marti Bolivar <marti@foundries.io>
Minor cleanups:
- Allow the user to run extract.content.py -h with ZEPHYR_BASE unset
without crashing
- Replace the use of the ZEPHYR_BUILD environment variable with an
--ignore argument (that can be given multiple times) which specifies
a path to a directory whose files should *not* be copied, and use
it to ignore the build directory
- rename, reorder, and tweak get_files() arguments a bit to go from
source to dest, with extra configuration at the end, getting rid of
local shadowing of global built-in functions
Signed-off-by: Marti Bolivar <marti@foundries.io>
Gerard Marull-Paretas