.. |br| raw:: html
Developer Guide
***************
This documentation is intended for developers of the core FleCSI
library.
-----
Style Guide
+++++++++++
Unless otherwise specified, follow the `Boost "Design and Programming"
guidelines`__ and `header guidelines`__. They do not address purely stylistic
matters of indentation, spacing, and brace placement; those are addressed for
FleCSI by ``clang-format``.
__ https://www.boost.org/development/requirements.html#Design_and_Programming
__ https://www.boost.org/development/header.html
Exceptions
^^^^^^^^^^
Starting from the 2019 version:
* Optimization does matter for FleCSI, but only where the overhead it
introduces is expected to be a significant fraction of the application's
runtime.
* Obviously FleCSI uses certain additional libraries (like Legion). Do not
add further dependencies without discussion. As a single component, there
is no need to limit use of one part of FleCSI by another.
* The ABI concerns, especially for Windows, are irrelevant.
* ``noexcept`` has replaced exception-specifications and should be used as
appropriate, especially for move operations.
* FleCSI uses its own unit-testing framework, not Boost's.
* The preferable line-length limit is 78 characters, so that even a diff of
such a file avoids the line continuations used in some displays.
* Individual documentation files do not get their own copyright/license.
* The source-file header is of course different.
* FleCSI uses ``flog_assert``, not ``BOOST_ASSERT``.
* ``min`` and ``max`` do not require special treatment.
* Header and source files are suffixed with ``.hh`` and ``.cc``.
* There are no "primary directories".
* Code documentation is generated by Doxygen, prose by Sphinx.
Additional Rules
^^^^^^^^^^^^^^^^
* Changes purely for stylistic reasons should be rare and must appear in a
separate commit (preferably a separate merge request).
* Class and enumeration names do not get a suffix ``_t``.
* Header names are "absolute" (*i.e.*, begin with ``flecsi/``).
* Function documentation uses the imperative mood.
Directory Structure
^^^^^^^^^^^^^^^^^^^
The source code for the core FleCSI infrastructure is located in the
*top-level/flecsi* directory. For the most part, the subdirectories of
this directory correspond to the different namespaces in the core
infrastructure. Each of these subdirectories must contain a valid
CMakeLists.txt file. However, none of their children should have a
CMakeLists.txt file, i.e., the build system will not recurse beyond the
first level of subdirectories. Developers should use relative paths
within a CMakeLists.txt file to identify source in subdirectories.
Unit test files should be placed in the *test* subdirectory of each
namespace subdirectory. By convention, developers should not create
subdirectories within the test subdirectory.
-----
Versioning
++++++++++
FleCSI's versioning system uses three branch types (descibed below) that
define the purpose and provenance of the code at a particular commit,
with tags that label points of interest in the development cycle, e.g,.
releases or stable features.
Branch Types
^^^^^^^^^^^^
* **incompatible** |br|
The *devel* branch is where work on the next major release takes
place, potentially with interface and feature changes that are
*incompatible* with previous versions.
* **feature** |br|
Feature branches (named for their *major* version number, e.g., 1, 2,
3) are for feature development on the current major version.
* **release** |br|
Release branches (named for their *major.minor* version number, e.g.,
1.1, 1.2) are stable versions of the code base. These can only be
modified with bug fixes. At appropriate points, tags (named for their
*major.minor.patch* version number, e.g., 1.1.2) are used to identify
patched versions.
.. tip::
At the time of writing, FleCSI has the following branches:
**devel** (incompatible) |br|
**1** (feature) |br|
**1.4** (release)
Tags
^^^^
The form of a tag is determined by the underlying branch type and commit
it is intended to reference:
* **devel branch** |br|
In general, a commit on the devel branch should only be tagged if it
is the first commit *after* a new feature branch has been created, and
should be named with *d* and the major version of the feature branch,
e.g., when feature branch *2* is created, a new devel tag *d2* should
be created on the next devel commit.
* **feature branch** |br|
Feature tags should be created on the next commit after a new release
branch is created, with *f* and the major.minor version of the release
branch, e.g., when release branch *1.5* is created, a new feature tag
*f1.5* should be created on the next feature commit.
* **release branch** |br|
Release tags should be created for each new release, with *v* and the
release version, e.g., *v1.4.1*.
.. tip::
Tags mark a *whence* or *origin* for development rather than a
*whither* or *completion*. The first letter of the tag is important,
i.e., *d*, *f*, or *v*, as it is used by FleCSI to determine the
branch type of the tag during CMake configuration.
Workflow
^^^^^^^^
FleCSI development uses a *devel -> feature -> release* forking workflow
that can be visualized as in :numref:`branch`. Bugfixes and features can
be back-merged into *feature* or *devel*, as appropriate.
.. _branch:
.. figure:: images/branch.png
Workflow diagram illustrating basic workflow using the three branch
types described above. (Figure due to Angela Herring.)
-----
Container Images
++++++++++++++++
.. important::
These sections assumes that you are familiar with container management
using a container engine like *Docker*. If you are not, there is
excellent documentation starting `here`__. Additionally, we include
instructions and links for installation and basic usage below.
__ https://docs.docker.com/get-started
.. include:: install_docker.rst
Tutorial Image
^^^^^^^^^^^^^^
This section details the steps to modify and update the docker image for
the FleCSI tutorials.
.. tip::
These instructions are only relevant if you want to modify or
rename the automatically generated tutorial images. The docker files in
this directory are automatically built on Docker Hub, whenever they get
modified on GitHub. These instructions are primarily intended for
developers who are modifying the image scripts and need to test them.
Building and Testing Locally
============================
The tutorial docker files are located in *flecsi/tutorial/docker*. These
instructions assume that you are in that directory.
To build a new docker image, you can execute the following:
.. code-block:: console
$ docker build -t laristra/flecsi-tutorial:TAG -f RUNTIME .
where *TAG* and *RUNTIME* are replaced with a tag and a specific runtime
backend, e.g., legion or mpi. The *TAG* argument may be any name that
you would like to give the image. The default is *latest*.
Docker will try to cache build steps and information that it doesn't
think has changed. Sometimes it is useful to force docker to rebuild
everything from scratch. To do this, you can pass the *--no-cache* flag:
.. code-block:: console
$ docker build --no-cache -t laristra/flecsi-tutorial:TAG -f RUNTIME .
This is particularly useful if a repository has changed.
The default repository and branch for the container are
*https://gitlab.lanl.gov/laristra/flecsi.git* and *devel*, respectively.
If you are in the process of updating the container, and would like to
use a different repository and branch, you can specifiy them like:
.. code-block:: console
$ docker build --build-arg REPO=https://github.com/tuxfan/flecsi.git \
--build-arg BRANCH=tutorial-update --no-cache \
-t laristra/flecsi-tutorial:TAG -f RUNTIME .
where *REPO* is the git repo to use, and *BRANCH* is the branch name to
use.
Once you have successfully built the imager, you can test it using the
*run* command:
.. code-block:: console
$docker run -it --shm-size=512m -u flecsi laristra/flecsi-tutorial:TAG /bin/bash
This will open an interactive shell (/bin/bash) on the running image.
Pushing to hub.docker.com
=========================
If you want to push images to the docker hub repository, you will need
to log in:
.. code-block:: console
$ docker login
This will prompt you for a username and password. You will need to be a
member of the *laristra* organization to successfully push images to the
docker hub. Once you are logged in, you can push an image to Docker Hub
with:
.. code-block:: console
$ docker push laristra/flecsi-tutorial:TAG
where TAG is the tag that you specified during the build.
In general, pushing the image manually is unnecessary because it will
automatically be rebuilt when the docker file is updated on GitHub.
-----
Gitlab CI
^^^^^^^^^
FleCSI has a special branch *gitlab-ci* that provides images and
configuration files that are used for Gitlab's continuous integration
(CI) services. The current set of utilties include:
* **Images** |br|
Operating system, build environment, and FleCSI images. These use
dockerfile syntax, and can be built and run with docker, and podman
engines, or potentially, with any engine that is compatible with
docker-generated images.
The images are modular, and are intended to be used to build up
capabilities:
* **OS Images** |br|
Located in the *os* directory, these are designed to provide a
standard base of functionality, i.e., a standard set of system
packages, which can be used interchangeably with downstream
environment images.
* **Environment** |br|
Located in a directory named for the corresponding branch, e.g., the
environment dockerfile file for *devel* is located in the *devel*
directory, these are designed to provide a standard spack
environment for building FleCSI.
It is intended that developers add new directories when necessary to
support custom configuration of a branch that they are testing.
* **Build** |br|
Also located in the *branch* directory, the build image provides a
pre-built FleCSI installation that can be used for debugging, or as
a building block for other projects.
* **Spack Configuration Files** |br|
Located in the *spack* subdirectory of an environment directory, these
allow customization of the spack install for a particular environment.
In particular, the *spack/packages.py* file should specify explicit
versions for as many packages as is feasible. Package versions should
only be elevated when necessary for new feature support. Following
this guideline will greatly increase stability.
Build System
============
The *gitlab-ci* branch uses a standard CMake build system. There are
several important configuration options that you need to be aware of:
* **CI_BRANCH** |br|
This is the branch name of the branch or fork of *gitlab-ci* that you
wish to use. The default is *gitlab-ci*. However, it is useful to be
able to set this to a personal branch name, so that you can test local
changes without doing a merge request to the central FleCSI Gitlab
project. Use this in conjunction with the *REPOSITORY* option.
* **REPOSITORY** |br|
This is the url for the repository where the build system will look
for the *CI_BRANCH* option that you specified. The default is the main
FleCSI Gitlab repository
*https://gitlab.lanl.gov/laristra/flecsi.git*. However, it is useful
to set this to your personal fork for testing new images.
* **REGISTRY** |br|
This is the container registry that will be used for *push-*, and
*pull-* targets. In general, this needs to be set to the registry that
is being used by the CI to pull images, e.g., laristra/flecsi-ci (this
is on dockerhub.com), or gitlab.lanl.gov:5050/laristra/flecsi
(internal container registry).
* **CONTAINER_ENGINE** |br|
This should be the name of the container engine that is installed on
your system. Currently, only *docker*, and *podman* are supported.
Additional engines may be added in the future.
As an example configuration, consider the following:
.. code-block:: console
$ mkdir build
$ cd build
$ cmake .. -DCI_BRANCH=ci-environment -DREPOSITORY=https://github.com/tuxfan/flecsi.git -DREGISTRY=gitlab.lanl.gov:5050/laristra/flecsi -DENGINE=podman
This configuration will use the author's *ci-environment* branch on
Github with the *podman* engine.
To build an image, try:
.. code-block:: console
$ make centos-8
This will build the *OS* image defined in *os/centos-8*.
.. warning::
A note of caution on building the images under the *gitlab-ci* branch.
Simply typing *make* after configuration will attempt to build **all
of the images in the project!** This can take several hours to complete!
If this is not your intention, you can list the build targets by
typing :code:`make help`. To build only a specific target, type
:code:`make TARGET`, where *TARGET* is the target that you would like
to build. Note also that building targets with dependencies will build
all of the dependencies of the target (This is unaviodable.)
.. tip::
The build system also supports passing runtime options to the
container engine. These are of the form *STAGE*\_EXTRA, e.g.,
BUILD_EXTRA, PUSH_EXTRA, and CLEAN_EXTRA. For example, BUILD_EXTRA can
be used to force complete rebuild of an image like:
.. code-block:: console
$ make BUILD_EXTRA="--pull --no-cache" centos-8
This will force the engine to start from scratch, i.e., pulling the
base image from the registry, and ignoring cached stages.
-----
Spack Cheat Sheet
+++++++++++++++++
To remove all cached sources:
.. code-block:: console
$ spack clean -d
To remove cached sources for a particular spec:
.. code-block:: console
$ spack clean -d spec
To uninstall all spack packages:
.. code-block:: console
$ spack uninstall -fay
To keep temporary staging files in /tmp/*$USER*:
.. code-block:: console
$ spack install --keep-stage ...
-----
Git Cheat Sheet
+++++++++++++++
To lookup the hash referenced by a tag:
.. code-block:: console
$ git rev-list -n 1 $TAG
To get the message for an annotated tag:
.. code-block:: console
$ git tag -nX (X specifies lines of annotation)
To sync tags:
.. code-block:: console
$ git fetch --prune --prune-tags
-----
Building for Darwin
+++++++++++++++++++
This is a recipe for building the current development version of FleCSI
on the LANL Darwin cluster.
Checkout spack and flecsi:
.. code-block:: console
$ git clone --branch devel git@gitlab.lanl.gov:laristra/flecsi.git
$ git clone --single-branch --branch develop git@github.com:spack/spack.git
Source the spack environment script:
.. code-block:: console
$ source spack/share/spack/setup-env.sh
Add the flecsi spack repo:
.. code-block:: console
$ spack repo add flecsi/spack-repo
Load gcc module and update spack compilers:
.. code-block:: console
$ module load gcc/9.3.0
$ spack compilers
Install the flecsi dependencies:
.. code-block:: console
$ spack install --only dependencies flecsi@devel backend=legion +hdf5 +graphviz +flog ^mpich
Create a spack environment and install the flecsi dependencies. This is
not redundant, spack will not rebuild anything, but it needs to install
the dependencies in the environment:
.. code-block:: console
$ spack env create legion
$ spack env activate legion
$ spack install --only dependencies flecsi@devel backend=legion +hdf5 +graphviz +flog ^mpich
$ spack install cmake
Load clang and doxygen, install sphinx, and update your path:
.. code-block:: console
$ module load clang/8.0.1
$ module load doxygen
$ pip3 install --user Sphinx
$ pip3 install --user recommonmark
$ pip3 install --user sphinx_rtd_theme
$ export PATH=$HOME/.local/bin:$PATH
Configure and build flecsi:
.. code-block:: console
$ cd flecsi && mkdir -p build/legion && cd build/legion
$ ../../tools/configure clang legion
$ make -j 16
-----
Graphviz Notes
++++++++++++++
FleCSI uses the `libcgraph`__ interface to `Graphviz`__ to create
control model visualizations. The *libcgraph* interface is fairly
counterintuitive. One particular gotcha is that graph, node, and edge
attributes can only be set on attributes that have been defined for the
graph. If an attribute type has not been defined, the graph will ignore
it. There is not easy to remedy to this problem: attributes that are
added after initialization will reset all previously added elements to
whatever the default of the new attribute is. Therefore, if you need to
add an attribute, the best thing to do is to look at the *graphviz.hh*
file in 'flecsi/utils', and add it there with a reasonable default.
__ https://graphviz.gitlab.io/_pages/pdf/libguide.pdf
__ https://www.graphviz.org
-----
Sphinx
++++++
Sphinx documentation is `here`__.
The following are some examples of frequently-used elements.
However, a good practice is to just look at the existing documentation
to figure out how something was done.
__ https://www.sphinx-doc.org/en/master
Links
^^^^^
.. code-block::
For more information look at `sphinx`__.
__ https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html
This will be rendered like:
For more information look at `sphinx`__.
__ https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html
Figures
^^^^^^^
.. code-block::
.. _undersea:
.. figure:: images/undersea.png
:align: center
:width: 70%
A colorful image resembling a cosmic version of an undersea world.
This will be rendered like:
.. _undersea:
.. figure:: images/undersea.png
:align: center
:width: 70%
A colorful image resembling a cosmic version of an undersea world.
You can reference the figure using its label *undersea* like:
.. code-block::
As can be seen in :numref:`undersea`...
This will be rendered like:
As can be seen in :numref:`undersea`...
Code Blocks
^^^^^^^^^^^
Syntax highlighting for codes blocks uses `pygments`__, which supports
many programming and markup languages.
__ https://pygments.org
Here are two examples:
Console
=======
.. code-block::
.. code-block:: console
$ xterm -hold -fs 10 -bg black -fg white -geometry 128x40 -e curl wttr.in
This will be rendered like:
.. code-block:: console
$ xterm -hold -fs 10 -bg black -fg white -geometry 128x40 -e curl wttr.in
C++
===
.. code-block::
.. code-block:: cpp
template
using Baz = Foo;
This will be rendered like:
.. code-block:: cpp
template
using Baz = Foo;
Literal Includes
^^^^^^^^^^^^^^^^
Literal includes allow you to directly include source code or other
inputs from the actual file you are referencing. This is useful because
any changes to the file will automatically be captured in the
documentation.
.. code-block::
.. literalinclude:: ../../../tutorial/standalone/standalone.cc
:language: cpp
:lines: 23-45
This will be rendered like:
.. literalinclude:: ../../../tutorial/standalone/standalone.cc
:language: cpp
:lines: 23-45
.. vim: set tabstop=2 shiftwidth=2 expandtab fo=cqt tw=72 :