Table of contents
- The Caliper CLI
- Installing from NPM
- Using the Docker image
- Installing locally from source
Caliper is published as the @hyperledger/caliper-cli NPM package and the hyperledger/caliper Docker image, both containing the CLI binary. Refer to the Installing from NPM and Using the Docker image sections for the available versions and their intricacies.
Installing and running Caliper usually consists of the following steps, thoroughly detailed by the remaining sections:
- Acquire the Caliper CLI either from NPM or from DockerHub.
- Execute a bind command through the CLI. This step pulls the specified version of SDK packages for the selected platform.
- Start the benchmark through the CLI or by starting the Docker container.
The examples in the rest of the documentation use the caliper-benchmarks repository as the Caliper workspace since it contains many sample artifacts for benchmarking.
Important: make sure you check out the appropriate tag/commit of the repository, matching the version of Caliper you use.
To clone the
caliper-benchmarks repository, run:
git clone https://github.com/hyperledger/caliper-benchmarks.git cd caliper-benchmarks git checkout <your Caliper version>
Note: If you are running your custom benchmark, then change this directory path (and other related configurations) accordingly in the examples.
The Caliper CLI
Unless you are embedding the Caliper packages in your own application, you will probably use Caliper through its command line interface (CLI). The other sections will introduce the different ways of acquiring and calling the Caliper CLI. This section simply focuses on the API it provides.
Note: The following examples assume a locally installed CLI in the
~/caliper-benchmarksdirectory, hence the
npxcall before the
caliperbinary. Refer to the Local NPM install section for the specifics.
The entry point of the CLI is the
caliper binary. You can confirm whether the CLI is installed correctly by checking its version:
user@ubuntu:~/caliper-benchmarks$ npx caliper --version v0.4.2
The CLI provides multiple commands to perform different tasks. To check the available commands and their descriptions, execute:
user@ubuntu:~/caliper-benchmarks$ npx caliper --help caliper <command> Commands: caliper.js bind [options] Bind Caliper to a specific SUT and its SDK version caliper.js launch <subcommand> Launch a Caliper process either in a manager or worker role. caliper.js unbind [options] Unbind Caliper from a previously bound SUT and its SDK version caliper.js completion generate completion script Options: --help, -h Show usage information [boolean] --version Show version information [boolean] Examples: caliper bind caliper unbind caliper launch manager caliper launch worker For more information on Hyperledger Caliper: https://hyperledger.github.io/caliper/
You can also request the help page of a specific command, as demonstrated by the next subsections.
Note: the command options can be set either through the command line, or from various other sources supported by the configuration mechanism of Caliper. This flexibility makes it easy to embed the CLI in different environments.
The bind command
Acquiring Caliper is as easy as installing a single NPM package, or pulling a single Docker image. However, this single point of install necessitates an additional step of telling Caliper which platform to target and which platform SDK version to use. This step is called binding, provided by the
bind CLI command.
To have a look at the help page of the command, execute:
user@ubuntu:~/caliper-benchmarks$ npx caliper bind --help Usage: caliper bind --caliper-bind-sut fabric:1.4 --caliper-bind-cwd ./ --caliper-bind-args="-g" Options: --help, -h Show usage information [boolean] --version Show version information [boolean] --caliper-bind-sut The name and version of the platform and its SDK to bind to [string] --caliper-bind-cwd The working directory for performing the SDK install [string] --caliper-bind-args Additional arguments to pass to "npm install". Use the "=" notation when setting this parameter [string] --caliper-bind-file Yaml file to override default (supported) package versions when binding an SDK [string]
The binding step technically consists of an extra
npm install call with the appropriate packages and install settings, fully managed by the CLI. The following parameters can be set for the command:
- SUT/platform name and SDK version: specifies the name of the target platform and its SDK version to install e.g.,
- Working directory: the directory from which the
npm installcommand must be performed. Defaults to the current working directory
- User arguments: additional arguments to pass to
npm install, e.g.,
The following SUT name and SDK version combinations are supported:
1.4.11 [1.4, latest],
2.2.3 [2.2, latest-v2]
Note: Ensure that the SDK you are binding is compatible with the the SUT version that you intend to target.
Note: all patch versions of Besu are supported for each version. The
1.3.2SUT is deprecated and it is recommended you use
bind command is useful when you plan to run multiple benchmarks against the same SUT version. Bind once, then run different benchmarks without the need to bind again. As you will see in the next sections, the launcher commands for the manager and worker processes can also perform the binding step if the required parameter is present.
The built-in bindings can be overridden by setting the
caliper-bind-file parameter to a YAML file path. The file must match the structure of the default binding file, documented here. This way you can use experimental SDK versions that are not (yet) officially supported by Caliper. This also means that we cannot provide help for such SDK versions!
The unbind command
It might happen that you would like to switch between different SUT SDK versions/bindings during your measurements or project development. Depending on the SUT SDK, simply rebinding to a different version might leave behind unwanted packages, resulting in obscure errors.
To avoid this, the CLI provides an
unbind command, that behaves exactly like the
bind command (even uses the same arguments), but instead of installing the packages present in the binding specification, it removes them, leaving no trace of the previous binding.
To have a look at the help page of the command, execute:
user@ubuntu:~/caliper-benchmarks$ npx caliper unbind --help Usage: caliper unbind --caliper-bind-sut fabric:1.4 --caliper-bind-cwd ./ --caliper-bind-args="-g" Options: --help, -h Show usage information [boolean] --version Show version information [boolean] --caliper-bind-sut The name and version of the platform and its SDK to unbind [string] --caliper-bind-cwd The working directory for performing the SDK removal [string] --caliper-bind-args Additional arguments to pass to "npm remove". Use the "=" notation when setting this parameter [string] --caliper-bind-file Yaml file to override default (supported) package versions when unbinding an SDK [string]
Note: It is recommended to either bind/unbind globally (as done by the Caliper Docker image), or use the
--caliper-bind-args="--save-dev"argument when performing the binding/unbinding. This ensures that
npmwill correctly remove the packages.
The launch command
Caliper runs a benchmark by using worker processes to generate the workload, and by using a manager process to coordinate the different benchmark rounds among the worker processes. Accordingly, the CLI provides commands for launching both manager and worker processes.
To have a look at the help page of the command, execute:
user@ubuntu:~/caliper-benchmarks$ npx caliper launch --help caliper launch <subcommand> Launch a Caliper process either in a manager or worker role. Commands: caliper launch manager [options] Launch a Caliper manager process to coordinate the benchmark run caliper launch worker [options] Launch a Caliper worker process to generate the benchmark workload Options: --help, -h Show usage information [boolean] --version Show version information [boolean]
The launch manager command
The Caliper manager process can be considered as the entry point of a distributed benchmark run. It coordinates (and optionally spawns) the worker processes throughout the benchmark run.
To have a look at the help page of the command, execute:
user@ubuntu:~/caliper-benchmarks$ npx caliper launch manager --help Usage: caliper launch manager --caliper-bind-sut fabric:1.4 [other options] Options: --help, -h Show usage information [boolean] --version Show version information [boolean] --caliper-bind-sut The name and version of the platform to bind to [string] --caliper-bind-cwd The working directory for performing the SDK install [string] --caliper-bind-args Additional arguments to pass to "npm install". Use the "=" notation when setting this parameter [string] --caliper-bind-file Yaml file to override default (supported) package versions when binding an SDK [string]
As you can see, the
launch manager command can also process the parameters of the
bind command, just in case you would like to perform the binding and the benchmark run in one step.
However, the command requires the following parameters to be set:
- caliper-workspace: the directory serving as the root of your project. Every relative path in other configuration files or settings will be resolved from this directory. The workspace concept was introduced to make Caliper projects portable across different machines.
- caliper-benchconfig: the path of the file containing the configuration of the test rounds, as detailed in the Architecture page. Should be relative to the workspace path.
- caliper-networkconfig: the path of the file containing the network configuration/description for the selected SUT, detailed in the configuration pages of the respective adapters. Should be relative to the workspace path.
The launch worker command
The Caliper worker processes are responsible for generating the workload during the benchmark run. Usually more than one worker process is running, coordinated by the single manager process.
To have a look at the help page of the command, execute:
user@ubuntu:~/caliper-benchmarks$ npx caliper launch worker --help Usage: caliper launch manager --caliper-bind-sut fabric:1.4 [other options] Options: --help, -h Show usage information [boolean] --version Show version information [boolean] --caliper-bind-sut The name and version of the platform to bind to [string] --caliper-bind-cwd The working directory for performing the SDK install [string] --caliper-bind-args Additional arguments to pass to "npm install". Use the "=" notation when setting this parameter [string] --caliper-bind-file Yaml file to override default (supported) package versions when binding an SDK [string]
As you can see, you can configure the worker processes the same way as the manager process. Including the optional binding step, but also the three mandatory parameters mentioned in the previous section.
Caliper test phase control
Caliper commands are capable of passing all runtime configuration settings. A subset of these commands are for flow control that provide direct control over the following Caliper phases:
It is possible to skip, or perform only one of the above phases through use of the correct flag. For instance, it is common to have an existing network that may be targeted by Caliper through the provision of a
Installing from NPM
Caliper is published as the @hyperledger/caliper-cli NPM package, providing a single point of install for every supported adapter.
Before explaining the steps for installing Caliper, let’s take a look at the
Versions page of the CLI package. You will see a list of tags and versions. If you are new to NPM, think of versions as immutable pointers to a specific version (duh) of the source code, while tags are mutable pointers to a specific version. So tags can change where they point to. Easy, right?
But why is all this important to you? Because Caliper is still in its pre-release life-cycle (< v1.0.0), meaning that even minor version bumps are allowed to introduce breaking changes. And if you use Caliper in your project, you might run into some surprises depending on how you install Caliper from time to time.
Note: Until Caliper reaches v1.0.0, always use the explicit version numbers when installing from NPM. So let’s forget about the
unstabletags, as of now they are just a mandatory hindrance of NPM. As you will see, we deliberately do not provide such tags for the Docker images.
Now that we ignored the tags, let’s see the two types of version numbers you will encounter:
0.2.0: Version numbers of this form denote releases deemed stable by the maintainers. Such versions have a corresponding GitHub tag, both in the
caliper-benchmarksrepositories. Moreover, the latest stable version is documented by the matching version of the documentation page. So make sure to align the different versions if you run into some issue.
0.4.2-unstable-20201127140758: Such version “numbers” denote unstable releases that are published upon every merged pull request (hence the timestamp at the end), and eventually will become a stable version, e.g.,
0.4.2. This way you always have access to the NPM (and Docker) artifacts pertaining to the
mainbranch of the repository. Let’s find and fix the bugs of new features before they make it to the stable release!
Note: The newest unstable release always corresponds to the up-to-date version of the related repositories, and the
vNextversion of the documentation page!
The following tools are required to install the CLI from NPM:
- node-gyp, python2, make, g++ and git (for fetching and compiling some packages during install)
- Node.js v10.X LTS (for running Caliper)
- Docker and Docker Compose (only needed when running local examples, or using Caliper through its Docker image)
Local NPM install
Note: this is the highly recommended way to install Caliper for your project. Keeping the project dependencies local makes it easier to setup multiple Caliper projects. Global dependencies would require re-binding every time before a new benchmark run (to ensure the correct global dependencies).
- Set your NPM project details with
npm init(or just execute
npm init -y) in your workspace directory (if you haven’t done this already, i.e., you don’t have a
- Install the Caliper CLI as you would any other NPM package. It is highly recommended to explicitly specify the version number, e.g.,
- Bind the CLI to the required platform SDK (e.g.,
- Invoke the local CLI binary (using npx) with the appropriate parameters. You can repeat this step for as many benchmarks as you would like.
Putting it all together:
user@ubuntu:~/caliper-benchmarks$ npm init -y user@ubuntu:~/caliper-benchmarks$ npm install --only=prod \ @firstname.lastname@example.org user@ubuntu:~/caliper-benchmarks$ npx caliper bind \ --caliper-bind-sut fabric:2.2 user@ubuntu:~/caliper-benchmarks$ npx caliper launch manager \ --caliper-workspace . \ --caliper-benchconfig benchmarks/scenario/simple/config.yaml \ --caliper-networkconfig networks/fabric/test-network.yaml
We could also perform the binding automatically when launching the manager process (note the extra parameter for
caliper launch manager):
user@ubuntu:~/caliper-benchmarks$ npm init -y user@ubuntu:~/caliper-benchmarks$ npm install --only=prod \ @email@example.com user@ubuntu:~/caliper-benchmarks$ npx caliper launch manager \ --caliper-bind-sut fabric:2.2 \ --caliper-workspace . \ --caliper-benchconfig benchmarks/scenario/simple/config.yaml \ --caliper-networkconfig networks/fabric/test-network.yaml
Note: specifying the
--only=prodparameter in step 2 will ensure that the default latest SDK dependencies for every platform will not be installed. Since we perform an explicit binding anyway (and only for a single platform), this is the desired approach, while also saving some storage and time.
Note: always make sure that the versions of the SUT, the bound SDK and the used artifacts match!
Global NPM install
Note: make sure that you have a really good reason for installing the Caliper CLI globally. The recommended approach is the local install. That way your project is self-contained and you can easily setup multiple projects (in multiple directories) that each target a different SUT (or just different SUT versions). Installing or re-binding dependencies globally can get tricky.
There are some minor differences compared to the local install:
- You don’t need a
- You can perform the install, bind and run steps from anywhere (just specify the workspace accordingly).
- You need to install the CLI globally (
- You need to tell the binding step to install the packages also globally (
- You can omit the
caliperwill be in your
user@ubuntu:~$ npm install -g --only=prod @firstname.lastname@example.org user@ubuntu:~$ caliper bind \ --caliper-bind-sut fabric:2.2 \ --caliper-bind-args=-g user@ubuntu:~$ caliper launch manager \ --caliper-workspace ~/caliper-benchmarks \ --caliper-benchconfig benchmarks/scenario/simple/config.yaml \ --caliper-networkconfig networks/fabric/test-network.yaml
Note: for global install you don’t need to change the directory to your workspace, you can simply specify
--caliper-workspace ~/caliper-benchmarks. But this way you can’t utilize the auto complete feature of your commandline for the relative paths of the artifacts.
Depending on your NPM settings, your user might need write access to directories outside of its home directory. This usually results in “Access denied” errors. The following pointers here can guide you to circumvent the problem.
Using the Docker image
Caliper is published as the hyperledger/caliper Docker image, providing a single point of usage for every supported adapter. The image builds upon the node:10.16-alpine image to keep the image size as low as possible.
The important properties of the image are the following:
- Working directory:
- The commands are executed by the
nodeuser (created in the base image)
- The environment variable
CALIPER_WORKSPACEis set to the
- The entry point is the globally installed
- The environment variable
CALIPER_BIND_ARGSis set to
-g, so the binding step also occurs globally.
- The default command is set to
--version. This must be overridden when using the image.
This has the following implications:
- It is recommended to mount your local workspace to the
/hyperledger/caliper/workspacecontainer directory. The default
CALIPER_WORKSPACEenvironment variable value points to this location, so you don’t need to specify it explicitly, one less setting to modify.
- You need to choose a command to execute, either
launch worker. Check the Docker and Docker-Compose examples for the exact syntax.
- The binding step is still necessary, similarly to the NPM install approach. Whether you use the
launch workercommand, you only need to set the required binding parameter. The easiest way to do this is through the
- You need to set the required parameters for the launched manager or worker. The easiest way to do this is through the
Starting a container
Parts of starting a Caliper container (following the recommendations above):
- Pick the required image version
- Mount your local working directory to a container directory
- Set the required binding and run parameters
Note: the latest (or any other) tag is not supported, i.e, you explicitly have to specify the image version you want:
hyperledger/caliper:0.4.2, just like it’s the recommended approach for the NPM packages.
Putting it all together, split into multiple lines for clarity, and naming the container
user@ubuntu:~/caliper-benchmarks$ docker run \ -v $PWD:/hyperledger/caliper/workspace \ -e CALIPER_BIND_SUT=fabric:2.2 \ -e CALIPER_BENCHCONFIG=benchmarks/scenario/simple/config.yaml \ -e CALIPER_NETWORKCONFIG=networks/fabric/test-network.yaml \ --name caliper hyperledger/caliper:0.4.2 launch manager
Note: the above network configuration file contains a start script to spin up a local Docker-based Fabric network, which will not work in this form. So make sure to remove the start (and end) script, and change the node endpoints to remote addresses.
The above command is more readable when converted to a
version: '2' services: caliper: container_name: caliper image: hyperledger/caliper:0.4.2 command: launch manager environment: - CALIPER_BIND_SUT=fabric:2.2 - CALIPER_BENCHCONFIG=benchmarks/scenario/simple/config.yaml - CALIPER_NETWORKCONFIG=networks/fabric/test-network.yaml volumes: - ~/caliper-benchmarks:/hyperledger/caliper/workspace
Once you navigate to the directory containing the
docker-compose.yaml file, just execute:
Note: if you would like to test a locally deployed SUT, then you also need to add the necessary SUT containers to the above file and make sure that Caliper starts last (using the
Installing locally from source
Note: this section is intended only for developers who would like to modify the Caliper code-base and experiment with the changes locally before raising pull requests. You should perform the following steps every time you make a modification you want to test, to correctly propagate any changes.
The workflow of modifying the Caliper code-base usually consists of the following steps:
- Bootstrapping the repository
- Modifying and testing the code
- Publishing package changes locally
- Building the Docker image
Bootstrapping the Caliper repository
To install the basic dependencies of the repository, and to resolve the cross-references between the different packages in the repository, you must execute the following commands from the root of the repository directory:
npm i: Installs development-time dependencies, such as Lerna and the license checking package.
npm run repoclean: Cleans up the
node_modulesdirectory of all packages in the repository. Not needed for a freshly cloned repository.
npm run bootstrap: Installs the dependencies of all packages in the repository and links any cross-dependencies between the packages. It will take some time to finish installation. If it is interrupted by
ctrl+c, please recover the
package.jsonfile first and then run
npm run bootstrapagain.
Or as a one-liner:
user@ubuntu:~/caliper$ npm i && npm run repoclean -- --yes && npm run bootstrap
Note: do not run any of the above commands with
sudo, as it will cause the bootstrap process to fail.
Testing the code
The easiest way to test your changes is to run the CI process locally. Currently, the CI process runs benchmarks for specific adapters. You can trigger these tests by running the following script from the root directory of the repository, setting the
BENCHMARK environment variable to the platform name:
user@ubuntu:~/caliper$ BENCHMARK=fabric ./.travis/benchmark-integration-test-direct.sh
The following platform tests (i.e., valid
BENCHMARK values) are available:
The scripts will perform the following tests (also necessary for a successful pull request):
- Linting checks
- Licence header checks
- Unit tests
- Running sample benchmarks
If you would like to run other examples, then you can directly access the CLI in the
packages/caliper-cli directory, without publishing anything locally.
Note: the SDK dependencies in this case are fixed (the binding step is not supported with this approach), and you can check (and change) them in the
package.jsonfiles of the corresponding packages. In this case the repository needs to be bootstrapped again.
user@ubuntu:~/caliper$ node ./packages/caliper-cli/caliper.js launch manager \ --caliper-workspace ~/caliper-benchmarks \ --caliper-benchconfig benchmarks/scenario/simple/config.yaml \ --caliper-networkconfig networks/fabric/test-network.yaml
Publishing to local NPM repository
The NPM publishing and installing steps for the modified code-base can be tested through a local NPM proxy server, Verdaccio. The steps to perform are the following:
- Start a local Verdaccio server to publish to
- Publish the packages from the local (and possible modified) Caliper repository to the Verdaccio server
- Install and bind the CLI from the Verdaccio server
- Run the integration tests or any sample benchmark
packages/caliper-publish directory contains an internal CLI for easily managing the following steps. So the commands of the following sections must be executed from the
user@ubuntu:~/caliper$ cd ./packages/caliper-publish
Note: use the
--helpflag for the following CLI commands and sub-commands to find out more details.
To setup and start a local Verdaccio server, simply run the following command:
user@ubuntu:~/caliper/packages/caliper-publish$ ./publish.js verdaccio start ... [PM2] Spawning PM2 daemon with pm2_home=.pm2 [PM2] PM2 Successfully daemonized [PM2] Starting /home/user/projects/caliper/packages/caliper-tests-integration/node_modules/.bin/verdaccio in fork_mode (1 instance) [PM2] Done. ┌───────────┬────┬──────┬────────┬────────┬─────────┬────────┬─────┬───────────┬────────┬──────────┐ │ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │ ├───────────┼────┼──────┼────────┼────────┼─────────┼────────┼─────┼───────────┼────────┼──────────┤ │ verdaccio │ 0 │ fork │ 115203 │ online │ 0 │ 0s │ 3% │ 25.8 MB │ user │ disabled │ └───────────┴────┴──────┴────────┴────────┴─────────┴────────┴─────┴───────────┴────────┴──────────┘ Use `pm2 show <id|name>` to get more details about an app
The Verdaccio server is now listening on the following address:
Publishing the packages
Once Verdaccio is running, you can run the following command to publish every Caliper package locally:
user@ubuntu:~/caliper/packages/caliper-publish$ ./publish.js npm --registry "http://localhost:4873" ... + @email@example.com [PUBLISH] Published package @firstname.lastname@example.org ... + @email@example.com [PUBLISH] Published package @firstname.lastname@example.org ... + @email@example.com [PUBLISH] Published package @firstname.lastname@example.org
Take note of the dynamic version number you see in the logs, you will need it to install you modified Caliper version from Verdaccio (the
unstable tag is also present on NPM, so Verdaccio would probably pull that version instead of your local one).
Since the published packages include a second-precision timestamp in their versions, you can republish any changes immediately without restarting the Verdaccio server and without worrying about conflicting packages.
Running package-based tests
Once the packages are published to the local Verdaccio server, we can use the usual NPM install approach. The only difference is that now we specify the local Verdaccio registry as the install source instead of the default, public NPM registry:
user@ubuntu:~/caliper-benchmarks$ npm init -y user@ubuntu:~/caliper-benchmarks$ npm install --registry=http://localhost:4873 --only=prod \ @email@example.com user@ubuntu:~/caliper-benchmarks$ npx caliper bind --caliper-bind-sut fabric:2.2 user@ubuntu:~/caliper-benchmarks$ npx caliper launch manager \ --caliper-workspace . \ --caliper-benchconfig benchmarks/scenario/simple/config.yaml \ --caliper-networkconfig networks/fabric/test-network.yaml
Note: we used the local registry only for the Caliper packages. The binding happens through the public NPM registry. Additionally, we performed the commands through npx and the newly installed CLI binary (i.e., not directly calling the CLI code file).
Building the Docker image
Once the modified packages are published to the local Verdaccio server, you can rebuild the Docker image. The Dockerfile is located in the
To rebuild the Docker image, execute the following:
user@ubuntu:~/caliper/packages/caliper-publish$ ./publish.js docker ... Successfully tagged hyperledger/caliper:manager-unstable-20200206065953 [BUILD] Built Docker image "hyperledger/caliper:manager-unstable-20200206065953"
Now you can proceed with the Docker-based benchmarking as described in the previous sections.
Note: once you are done with the locally published packages, you can clean them up the following way:
user@ubuntu:~/caliper/packages/caliper-publish$ ./publish.js verdaccio stop
The Caliper codebase is released under the Apache 2.0 license. Any documentation developed by the Caliper Project is licensed under the Creative Commons Attribution 4.0 International License. You may obtain a copy of the license, titled CC-BY-4.0, at http://creativecommons.org/licenses/by/4.0/.