2. Usage

This section describes how to use the forwarder beyond the quick introduction in Quickstart.

2.1. Installation using pipx

The recommended way to install the exporter is by using pipx.

Pipx creates a `virtual Python environment`_ under the covers, installs the Python package into that environment and makes the zhmc_os_forwarder command available in a directory that is in the PATH. The zhmc_os_forwarder command will be available that way, regardless of whether or not you have a virtual Python environment active (that you may need for other purposes).

1. Prerequisite: Install pipx as an OS-level package

   Follow the steps at https://pipx.pypa.io/stable/installation/ to install pipx as an OS-level package to your local system.

2. Install the exporter using pipx

   To install the latest released version of the exporter:

   $ pipx install zhmc-os-forwarder
   

   To install a specific released version of the exporter, e.g. 1.0.0:

   $ pipx install zhmc-os-forwarder==1.0.0
   

   To install a specific development branch of the exporter, e.g. master:

   $ pipx install git+https://github.com/zhmcclient/zhmc-os-forwarder.git@master
   

   To install the exporter with a non-default Python version, e.g. 3.10:

   $ pipx install zhmc-os-forwarder --python python3.10
   

2.2. Installation into a virtual Python environment

In some cases it may be useful to install the exporter into your own `virtual Python environment`_. That avoids the dependency to pipx, but it requires you to activate the virtual environment every time you want to use the zhmc_os_forwarder command.

There is a number of ways how virtual Python environments can be created. This documentation describes the use of “virtualenv”:

1. Prerequisite: Install virtualenv into system Python:

   $ pip install virtualenv
   

2. Create and activate a virtual Python environment:

   $ virtualenv ~/.virtualenvs/zhmcpe
   $ source ~/.virtualenvs/zhmcpe/bin/activate
   

3. Install the exporter into the virtual Python environment:

   To install the latest released version of the exporter so that it uses your default Python version:

   (zhmcpe) $ pip install zhmc-os-forwarder
   

   To install a specific released version of the exporter, e.g. 1.0.0:

   (zhmcpe) $ pip install zhmc-os-forwarder==1.0.0
   

   To install a specific development branch of the exporter, e.g. master:

   (zhmcpe) $ pip install git+https://github.com/zhmcclient/zhmc-os-forwarder.git@master
   

2.3. Installation into a system Python

Your system Python version(s) are installed using OS-level packages for all the Python functionality.

Adding packages to your system Python using Python packages from `Pypi`_ may create issues. This is why recent versions of pip raise a warning when attempting to install into the system Python. Even if you install a Python package from Pypi into your user’s space, this may create issues.

The main issue is that the more Python packages you install into the system Python, the more likely there will be incompatible Python package dependencies.

Another issue is when you replace OS-level packages with Python packages.

In order to avoid these issues, you should install the exporter into the system Python only in cases where the system has a well-defined scope and you have full control over the set of OS-level and Python-level packages, for example when building a Docker container.

2.4. Running in a Docker container

If you want to run the forwarder in a Docker container you can create the container as follows, using the Dockerfile provided in the Git repository.

• Clone the Git repository of the forwarder and switch to the clone’s root directory:

  $ git clone https://github.com/zhmcclient/zhmc-os-forwarder
  $ cd zhmc-os-forwarder
  

• Build a local Docker image as follows:

  $ make docker
  

  This builds a container image named zhmc_os_forwarder:latest in your local Docker environment.

  The image does not contain the forwarder config file, so that needs to be specified when starting the container.

• Run the local Docker image as follows:

  $ docker run --rm -v $(pwd)/myconfig:/root/myconfig zhmc_os_forwarder -p 514:514 -c /root/myconfig/config.yaml -v
  

  In this command, the forwarder config file is provided on the local system as ./myconfig/config.yaml. The -v option of ‘docker run’ mounts the ./myconfig directory to /root/myconfig in the container’s file system. The -c option of the forwarder references the forwarder config file as it appears in the container’s file system.

  The command above maps port 514 in the docker container to port 514 of the system running docker. That is the default port used by syslog. If your remote syslog servers use different ports, they need to be mapped using the -p option of the docker run command.

2.5. zhmc_os_forwarder command

The zhmc_os_forwarder command supports the following arguments:

usage: zhmc_os_forwarder [-h] [-c CONFIG_FILE] [--log DEST] [--log-comp COMP[=LEVEL]]
                         [--syslog-facility TEXT] [--verbose] [--version] [--help-config]

IBM Z HMC OS Message Forwarder

optional arguments:

  -h, --help            show this help message and exit

  -c CONFIG_FILE        path name of config file. Use --help-config for details. Default:
                        /etc/zhmc-os-forwarder/config.yaml

  --log DEST            enable logging and set a log destination (stderr, syslog, FILE). Default:
                        no logging

  --log-comp COMP[=LEVEL]
                        set a logging level (error, warning, info, debug, off, default: warning)
                        for a component (forwarder, hmc, jms, all). May be specified multiple
                        times; options add to the default of: all=warning

  --syslog-facility TEXT
                        syslog facility (user, local0, local1, local2, local3, local4, local5,
                        local6, local7) when logging to the system log. Default: user

  --verbose, -v         increase the verbosity level (max: 2)

  --version             show versions of forwarder and zhmcclient library and exit

  --help-config         show help for forwarder config file and exit

2.6. Setting up the HMC

Usage of this package requires that the HMC in question is prepared accordingly:

• The Web Services API must be enabled on the HMC.

  You can do that in the HMC GUI by selecting “HMC Management” in the left pane, then opening the “Configure API Settings” icon on the pain pane, then selecting the “Web Services” tab on the page that comes up, and finally enabling the Web Services API on that page.

  The above is on a z16 HMC, it may be different on older HMCs.

  If you cannot find this icon, then your userid does not have permission for the respective task on the HMC. In that case, there should be some other HMC admin you can go to to get the Web Services API enabled.

2.7. Setting up firewalls or proxies

If you have to configure firewalls or proxies between the system where you run the zhmc_os_forwarder command and the HMC, the following ports need to be opened:

• 6794 (TCP) - for the HMC API HTTP server

• 61612 (TCP) - for the HMC API message broker via JMS over STOMP

For details, see sections “Connecting to the API HTTP server” and “Connecting to the API message broker” in the HMC API book.

2.8. HMC userid requirements

This section describes the requirements on the HMC userid that is used by the zhmc_os_forwarder command.

The HMC userid must have the following permissions:

• Object access permission to the following objects:

  • LPARs for which OS messages should be forwarded

  • CPCs containing these LPARs

• Task permission for the following tasks:

  • “Operating System Messages” task (view-only mode is sufficient)

2.9. HMC certificate

By default, the HMC is configured with a self-signed certificate. That is the X.509 certificate presented by the HMC as the server certificate during TLS handshake at its Web Services API.

The ‘zhmc_os_forwarder’ command will reject self-signed certificates by default.

The HMC should be configured to use a CA-verifiable certificate. This can be done in the HMC task “Certificate Management”. See also the HMC Security book and Chapter 3 “Invoking API operations” in the HMC API book.

The ‘zhmc_os_forwarder’ command provides control knobs for the verification of the HMC certificate via the verify_cert attribute in the Forwarder config file, as follows:

• Not specified or specified as true (default): Verify the HMC certificate using the CA certificates from the first of these locations:

  • The certificate file or directory in the REQUESTS_CA_BUNDLE environment variable, if set

  • The certificate file or directory in the CURL_CA_BUNDLE environment variable, if set

  • The Python ‘certifi’ package (which contains the Mozilla Included CA Certificate List).

• Specified with a string value: An absolute path or a path relative to the directory of the forwarder config file. Verify the HMC certificate using the CA certificates in the specified certificate file or directory.

• Specified as false: Do not verify the HMC certificate. Not verifying the HMC certificate means that hostname mismatches, expired certificates, revoked certificates, or otherwise invalid certificates will not be detected. Since this mode makes the connection vulnerable to man-in-the-middle attacks, it is insecure and should not be used in production environments.

If a certificate file is specified (using any of the ways listed above), that file must be in PEM format and must contain all CA certificates that are supposed to be used. Usually they are in the order from leaf to root, but that is not a hard requirement. The single certificates are concatenated in the file.

If a certificate directory is specified (using any of the ways listed above), it must contain PEM files with all CA certificates that are supposed to be used, and copies of the PEM files or symbolic links to them in the hashed format created by the OpenSSL command c_rehash.

An X.509 certificate in PEM format is base64-encoded, begins with the line -----BEGIN CERTIFICATE-----, and ends with the line -----END CERTIFICATE-----. More information about the PEM format is for example on this www.ssl.com page or in this serverfault.com answer.

Note that setting the REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE environment variables influences other programs that use these variables, too.

For more information, see the Security section in the documentation of the ‘zhmcclient’ package.

2.10. Forwarder config file

The forwarder config file tells the forwarder which HMC to use, and for which CPCs and LPARs it should forward OS messages to which syslog servers.

The forwarder config file is in YAML format and has the following structure:

hmc:
  host: {hmc-ip-address}
  userid: {hmc-userid}
  password: {hmc-password}
  verify_cert: {verify-cert}

forwarding:
  # list of forwarding definitions
  - syslogs:
     # list of remote syslog servers
     - host: {syslog-ip-address}
       port: {syslog-port}
       port_type: {syslog-port-type}
       facility: {syslog-facility}
    cpcs:
      # list of CPCs
      - cpc: {cpc-pattern}
        partitions:
          # list of LPARs
          - partition: {partition-pattern}

Where:

• {hmc-ip-address} is the IP address of the HMC.

• {hmc-userid} is the userid on the HMC to be used for logging on.

• {hmc-password} is the password of that userid.

• {verify-cert} controls whether and how the HMC server certificate is verified. For details, see HMC certificate.

• {cpc-pattern} is a regular expression for the CPC name, to select CPCs from the set of CPCs managed by the targeted HMC.

• {partition-pattern} is a regular expression for the LPAR name, to select LPARs from the CPC (or set of CPCs) specified in {cpc-pattern}.

Each item in the forwarding list is a forwarding definition that specifies a list of remote syslog servers and a list of LPARs (along with their CPCs) The OS messages of the specified LPARs will be forwarded to the remote syslog servers. In other words, the forwarding definitions are organized by the targeted syslog servers.

2.11. Example forwarder config file

The following is an example forwarder config file.

The file can be downloaded from the Git repo as examples/config_example.yaml.

# Config file for Z HMC OS Message Forwarder

hmc:
  host: 10.11.12.13
  userid: "myuser"
  password: "mypassword"
  verify_cert: false

forwarding:
  # Each item in this list is a forwarding definition that specifies which
  # LPARs on which CPCs should forward their OS messages to which set of
  # remote syslog servers.
  - syslogs:
     - host: 10.11.12.14
       port: 514
       port_type: udp
       facility: user
    cpcs:
      - cpc: MYCPC    # Can be a regular expression
        partitions:
          - partition: ".*"    # Can be a regular expression

2.12. Logging

The forwarder supports logging its own activities and the interactions with the HMC. Note that this kind of logging has nothing to do with the forwarding of OS messages to remote syslog servers.

By default, logging is disabled.

Logging is enabled by using the --log DEST option that controls the logging destination as follows:

• --log stderr - log to the Standard Error stream

• --log syslog - log to the System Log (see Logging to the System Log)

• --log FILE - log to the log file with path name FILE.

There are multiple components that can log. By default, all of them log at the warning level. This can be fine tuned by using the --log-comp COMP[=LEVEL] option. This option can be specified multiple times, and the specified options add in sequence to the default of all=warning.

The components that can be specified in COMP are:

• forwarder - activities of the forwarder. Logger name: zhmcforwarder.

• hmc - HTTP interactions with the HMC performed by the zhmcclient library. Logger name: zhmcclient.hmc.

• jms - JMS notifications from the HMC received by the zhmcclient library. Logger name: zhmcclient.jms.

• all - all of these components.

The log levels that can be specified in LEVEL are:

• error - Show only errors for the component. Errors are serious conditions that need to be fixed by the user. Some errors may need to be reported as issues. The forwarder retries with the HMC in case of certain errors, but some errors cause the forwarder to terminate.

• warning - Show errors and warnings for the component. Warnings never cause the forwarder to terminate, but should be analyzed and may need to be fixed.

• info - Show informations, warnings and errors for the component. Informations are useful to understand what is going on.

• debug - Show debug info, informations, warnings and errors for the component. Debug info provides a very detailed amount of information that may be useful foo analyzing problems.

• off - Show no log messages for the component.

The LEVEL part can be omitted in the --log-comp option, and its default is warning. This is for compatibility with older versions of the forwarder.

The default log level for each component is warning, and specifying other log levels changes that level only for the specified components but keeps the default for those components that are not specified.

Examples:

# log to Standard Error with all=warning
$ zhmc_os_forwarder --log stderr ...

# log to file mylog.log with all=warning
$ zhmc_os_forwarder --log mylog.log ...

# log to file mylog.log with forwarder=info, hmc=warning (by default), jms=warning (by default)
$ zhmc_os_forwarder --log mylog.log --log-comp forwarder=info

# log to file mylog.log with forwarder=info, hmc=warning (by default), jms=debug
$ zhmc_os_forwarder --log mylog.log --log-comp forwarder=info --log-comp jms=debug

# log to file mylog.log with forwarder=debug, hmc=debug, jms=debug
$ zhmc_os_forwarder --log mylog.log --log-comp all=debug

# log to file mylog.log with forwarder=info, hmc=off, jms=off
$ zhmc_os_forwarder --log mylog.log --log-comp all=off --log-comp forwarder=info

2.12.1. Logging to the System Log

When logging its own activities and the interactions with the HMC to the System Log, the syslog address used by the forwarder depends on the operating system the forwarder runs on, as follows:

• Linux: /dev/log

• macOS: /var/run/syslog

• Windows: UDP port 514 on localhost (requires a syslog demon to run)

• CygWin: /dev/log (requires the syslog-ng package to be installed)

For other operating systems, UDP port 514 on localhost is used.

Messages logged to the system log will only show up there if the syslog configuration has enabled the syslog facility and the syslog severity levels that are used by the forwarder. The configuration of the syslog depends on the operating system or syslog demon that is used and is therefore not described here.

The syslog facility that will be used by the forwarder can be specified with the --syslog-facility option and defaults to user.

The syslog severity levels (not to be confused with syslog priorities) that will be used by the forwarder are derived from the Python log levels using the default mapping defined by Python logging, which is:

Python log level	Syslog severity
ERROR	3 (Error)
WARNING	4 (Warning)
INFO	6 (Informational)
DEBUG	7 (Debug)

On some systems, the syslog rejects messages that exceed a certain limit. For this reason, the forwarder truncates the message text to somewhat below 2048 Bytes, when logging to the system log. Messages are not truncated when logging to the Standard Error stream or to a file.