.. Copyright 2023 IBM Corp. All Rights Reserved. .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://www.apache.org/licenses/LICENSE-2.0 .. .. Unless required by applicable law or agreed to in writing, software .. distributed under the License is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. Usage ===== This section describes how to use the forwarder beyond the quick introduction in :ref:`Quickstart`. Installation using pipx ----------------------- The recommended way to install the exporter is by using pipx. Pipx creates a `virtual Python environment`_ under the covers, installs the Python package into that environment and makes the ``zhmc_os_forwarder`` command available in a directory that is in the PATH. The ``zhmc_os_forwarder`` command will be available that way, regardless of whether or not you have a virtual Python environment active (that you may need for other purposes). 1. Prerequisite: Install pipx as an OS-level package Follow the steps at https://pipx.pypa.io/stable/installation/ to install pipx as an OS-level package to your local system. 2. Install the exporter using pipx To install the latest released version of the exporter: .. code-block:: bash $ pipx install zhmc-os-forwarder To install a specific released version of the exporter, e.g. 1.0.0: .. code-block:: bash $ pipx install zhmc-os-forwarder==1.0.0 To install a specific development branch of the exporter, e.g. master: .. code-block:: bash $ pipx install git+https://github.com/zhmcclient/zhmc-os-forwarder.git@master To install the exporter with a non-default Python version, e.g. 3.10: .. code-block:: bash $ pipx install zhmc-os-forwarder --python python3.10 Installation into a virtual Python environment ---------------------------------------------- In some cases it may be useful to install the exporter into your own `virtual Python environment`_. That avoids the dependency to pipx, but it requires you to activate the virtual environment every time you want to use the ``zhmc_os_forwarder`` command. There is a number of ways how virtual Python environments can be created. This documentation describes the use of "virtualenv": 1. Prerequisite: Install virtualenv into system Python: .. code-block:: bash $ pip install virtualenv 2. Create and activate a virtual Python environment: .. code-block:: bash $ virtualenv ~/.virtualenvs/zhmcpe $ source ~/.virtualenvs/zhmcpe/bin/activate 3. Install the exporter into the virtual Python environment: To install the latest released version of the exporter so that it uses your default Python version: .. code-block:: bash (zhmcpe) $ pip install zhmc-os-forwarder To install a specific released version of the exporter, e.g. 1.0.0: .. code-block:: bash (zhmcpe) $ pip install zhmc-os-forwarder==1.0.0 To install a specific development branch of the exporter, e.g. master: .. code-block:: bash (zhmcpe) $ pip install git+https://github.com/zhmcclient/zhmc-os-forwarder.git@master Installation into a system Python --------------------------------- Your system Python version(s) are installed using OS-level packages for all the Python functionality. Adding packages to your system Python using Python packages from `Pypi`_ may create issues. This is why recent versions of pip raise a warning when attempting to install into the system Python. Even if you install a Python package from Pypi into your user's space, this may create issues. The main issue is that the more Python packages you install into the system Python, the more likely there will be incompatible Python package dependencies. Another issue is when you replace OS-level packages with Python packages. In order to avoid these issues, you should install the exporter into the system Python only in cases where the system has a well-defined scope and you have full control over the set of OS-level and Python-level packages, for example when building a Docker container. Running in a Docker container ----------------------------- If you want to run the forwarder in a Docker container you can create the container as follows, using the Dockerfile provided in the Git repository. * Clone the Git repository of the forwarder and switch to the clone's root directory: .. code-block:: bash $ git clone https://github.com/zhmcclient/zhmc-os-forwarder $ cd zhmc-os-forwarder * Build a local Docker image as follows: .. code-block:: bash $ make docker This builds a container image named ``zhmc_os_forwarder:latest`` in your local Docker environment. The image does not contain the forwarder config file, so that needs to be specified when starting the container. * Run the local Docker image as follows: .. code-block:: bash $ docker run --rm -v $(pwd)/myconfig:/root/myconfig zhmc_os_forwarder -p 514:514 -c /root/myconfig/config.yaml -v In this command, the forwarder config file is provided on the local system as ``./myconfig/config.yaml``. The ``-v`` option of 'docker run' mounts the ``./myconfig`` directory to ``/root/myconfig`` in the container's file system. The ``-c`` option of the forwarder references the forwarder config file as it appears in the container's file system. The command above maps port 514 in the docker container to port 514 of the system running docker. That is the default port used by syslog. If your remote syslog servers use different ports, they need to be mapped using the ``-p`` option of the ``docker run`` command. zhmc_os_forwarder command ------------------------- The ``zhmc_os_forwarder`` command supports the following arguments: .. When updating the command help, use a 100 char wide terminal .. code-block:: text usage: zhmc_os_forwarder [-h] [-c CONFIG_FILE] [--log DEST] [--log-comp COMP[=LEVEL]] [--syslog-facility TEXT] [--verbose] [--version] [--help-config] IBM Z HMC OS Message Forwarder optional arguments: -h, --help show this help message and exit -c CONFIG_FILE path name of config file. Use --help-config for details. Default: /etc/zhmc-os-forwarder/config.yaml --log DEST enable logging and set a log destination (stderr, syslog, FILE). Default: no logging --log-comp COMP[=LEVEL] set a logging level (error, warning, info, debug, off, default: warning) for a component (forwarder, hmc, jms, all). May be specified multiple times; options add to the default of: all=warning --syslog-facility TEXT syslog facility (user, local0, local1, local2, local3, local4, local5, local6, local7) when logging to the system log. Default: user --verbose, -v increase the verbosity level (max: 2) --version show versions of forwarder and zhmcclient library and exit --help-config show help for forwarder config file and exit Setting up the HMC ------------------ Usage of this package requires that the HMC in question is prepared accordingly: * The Web Services API must be enabled on the HMC. You can do that in the HMC GUI by selecting "HMC Management" in the left pane, then opening the "Configure API Settings" icon on the pain pane, then selecting the "Web Services" tab on the page that comes up, and finally enabling the Web Services API on that page. The above is on a z16 HMC, it may be different on older HMCs. If you cannot find this icon, then your userid does not have permission for the respective task on the HMC. In that case, there should be some other HMC admin you can go to to get the Web Services API enabled. Setting up firewalls or proxies ------------------------------- If you have to configure firewalls or proxies between the system where you run the ``zhmc_os_forwarder`` command and the HMC, the following ports need to be opened: * 6794 (TCP) - for the HMC API HTTP server * 61612 (TCP) - for the HMC API message broker via JMS over STOMP For details, see sections "Connecting to the API HTTP server" and "Connecting to the API message broker" in the :term:`HMC API` book. HMC userid requirements ----------------------- This section describes the requirements on the HMC userid that is used by the ``zhmc_os_forwarder`` command. The HMC userid must have the following permissions: * Object access permission to the following objects: - LPARs for which OS messages should be forwarded - CPCs containing these LPARs * Task permission for the following tasks: - "Operating System Messages" task (view-only mode is sufficient) HMC certificate --------------- By default, the HMC is configured with a self-signed certificate. That is the X.509 certificate presented by the HMC as the server certificate during TLS handshake at its Web Services API. The 'zhmc_os_forwarder' command will reject self-signed certificates by default. The HMC should be configured to use a CA-verifiable certificate. This can be done in the HMC task "Certificate Management". See also the :term:`HMC Security` book and Chapter 3 "Invoking API operations" in the :term:`HMC API` book. The 'zhmc_os_forwarder' command provides control knobs for the verification of the HMC certificate via the ``verify_cert`` attribute in the :ref:`forwarder config file`, as follows: * Not specified or specified as ``true`` (default): Verify the HMC certificate using the CA certificates from the first of these locations: - The certificate file or directory in the ``REQUESTS_CA_BUNDLE`` environment variable, if set - The certificate file or directory in the ``CURL_CA_BUNDLE`` environment variable, if set - The `Python 'certifi' package `_ (which contains the `Mozilla Included CA Certificate List `_). * Specified with a string value: An absolute path or a path relative to the directory of the forwarder config file. Verify the HMC certificate using the CA certificates in the specified certificate file or directory. * Specified as ``false``: Do not verify the HMC certificate. Not verifying the HMC certificate means that hostname mismatches, expired certificates, revoked certificates, or otherwise invalid certificates will not be detected. Since this mode makes the connection vulnerable to man-in-the-middle attacks, it is insecure and should not be used in production environments. If a certificate file is specified (using any of the ways listed above), that file must be in PEM format and must contain all CA certificates that are supposed to be used. Usually they are in the order from leaf to root, but that is not a hard requirement. The single certificates are concatenated in the file. If a certificate directory is specified (using any of the ways listed above), it must contain PEM files with all CA certificates that are supposed to be used, and copies of the PEM files or symbolic links to them in the hashed format created by the OpenSSL command ``c_rehash``. An X.509 certificate in PEM format is base64-encoded, begins with the line ``-----BEGIN CERTIFICATE-----``, and ends with the line ``-----END CERTIFICATE-----``. More information about the PEM format is for example on this `www.ssl.com page `_ or in this `serverfault.com answer `_. Note that setting the ``REQUESTS_CA_BUNDLE`` or ``CURL_CA_BUNDLE`` environment variables influences other programs that use these variables, too. For more information, see the `Security `_ section in the documentation of the 'zhmcclient' package. Forwarder config file --------------------- The *forwarder config file* tells the forwarder which HMC to use, and for which CPCs and LPARs it should forward OS messages to which syslog servers. The forwarder config file is in YAML format and has the following structure: .. code-block:: yaml hmc: host: {hmc-ip-address} userid: {hmc-userid} password: {hmc-password} verify_cert: {verify-cert} forwarding: # list of forwarding definitions - syslogs: # list of remote syslog servers - host: {syslog-ip-address} port: {syslog-port} port_type: {syslog-port-type} facility: {syslog-facility} cpcs: # list of CPCs - cpc: {cpc-pattern} partitions: # list of LPARs - partition: {partition-pattern} Where: * ``{hmc-ip-address}`` is the IP address of the HMC. * ``{hmc-userid}`` is the userid on the HMC to be used for logging on. * ``{hmc-password}`` is the password of that userid. * ``{verify-cert}`` controls whether and how the HMC server certificate is verified. For details, see :ref:`HMC certificate`. * ``{cpc-pattern}`` is a :term:`regular expression` for the CPC name, to select CPCs from the set of CPCs managed by the targeted HMC. * ``{partition-pattern}`` is a :term:`regular expression` for the LPAR name, to select LPARs from the CPC (or set of CPCs) specified in ``{cpc-pattern}``. Each item in the ``forwarding`` list is a forwarding definition that specifies a list of remote syslog servers and a list of LPARs (along with their CPCs) The OS messages of the specified LPARs will be forwarded to the remote syslog servers. In other words, the forwarding definitions are organized by the targeted syslog servers. Example forwarder config file ----------------------------- The following is an example forwarder config file. The file can be downloaded from the Git repo as `examples/config_example.yaml `_. .. literalinclude:: ../examples/config_example.yaml :language: yaml Logging ------- The forwarder supports logging its own activities and the interactions with the HMC. Note that this kind of logging has nothing to do with the forwarding of OS messages to remote syslog servers. By default, logging is disabled. Logging is enabled by using the ``--log DEST`` option that controls the logging destination as follows: * ``--log stderr`` - log to the Standard Error stream * ``--log syslog`` - log to the System Log (see :ref:`Logging to the System Log`) * ``--log FILE`` - log to the log file with path name ``FILE``. There are multiple components that can log. By default, all of them log at the warning level. This can be fine tuned by using the ``--log-comp COMP[=LEVEL]`` option. This option can be specified multiple times, and the specified options add in sequence to the default of ``all=warning``. The components that can be specified in ``COMP`` are: * ``forwarder`` - activities of the forwarder. Logger name: ``zhmcforwarder``. * ``hmc`` - HTTP interactions with the HMC performed by the zhmcclient library. Logger name: ``zhmcclient.hmc``. * ``jms`` - JMS notifications from the HMC received by the zhmcclient library. Logger name: ``zhmcclient.jms``. * ``all`` - all of these components. The log levels that can be specified in ``LEVEL`` are: * ``error`` - Show only errors for the component. Errors are serious conditions that need to be fixed by the user. Some errors may need to be reported as issues. The forwarder retries with the HMC in case of certain errors, but some errors cause the forwarder to terminate. * ``warning`` - Show errors and warnings for the component. Warnings never cause the forwarder to terminate, but should be analyzed and may need to be fixed. * ``info`` - Show informations, warnings and errors for the component. Informations are useful to understand what is going on. * ``debug`` - Show debug info, informations, warnings and errors for the component. Debug info provides a very detailed amount of information that may be useful foo analyzing problems. * ``off`` - Show no log messages for the component. The ``LEVEL`` part can be omitted in the ``--log-comp`` option, and its default is ``warning``. This is for compatibility with older versions of the forwarder. The default log level for each component is ``warning``, and specifying other log levels changes that level only for the specified components but keeps the default for those components that are not specified. Examples: .. code-block:: bash # log to Standard Error with all=warning $ zhmc_os_forwarder --log stderr ... # log to file mylog.log with all=warning $ zhmc_os_forwarder --log mylog.log ... # log to file mylog.log with forwarder=info, hmc=warning (by default), jms=warning (by default) $ zhmc_os_forwarder --log mylog.log --log-comp forwarder=info # log to file mylog.log with forwarder=info, hmc=warning (by default), jms=debug $ zhmc_os_forwarder --log mylog.log --log-comp forwarder=info --log-comp jms=debug # log to file mylog.log with forwarder=debug, hmc=debug, jms=debug $ zhmc_os_forwarder --log mylog.log --log-comp all=debug # log to file mylog.log with forwarder=info, hmc=off, jms=off $ zhmc_os_forwarder --log mylog.log --log-comp all=off --log-comp forwarder=info Logging to the System Log ^^^^^^^^^^^^^^^^^^^^^^^^^ When logging its own activities and the interactions with the HMC to the System Log, the syslog address used by the forwarder depends on the operating system the forwarder runs on, as follows: * Linux: ``/dev/log`` * macOS: ``/var/run/syslog`` * Windows: UDP port 514 on localhost (requires a syslog demon to run) * CygWin: ``/dev/log`` (requires the syslog-ng package to be installed) For other operating systems, UDP port 514 on localhost is used. Messages logged to the system log will only show up there if the syslog configuration has enabled the syslog facility and the syslog severity levels that are used by the forwarder. The configuration of the syslog depends on the operating system or syslog demon that is used and is therefore not described here. The syslog facility that will be used by the forwarder can be specified with the ``--syslog-facility`` option and defaults to ``user``. The syslog severity levels (not to be confused with syslog priorities) that will be used by the forwarder are derived from the Python log levels using the default mapping defined by Python logging, which is: ================ ================= Python log level Syslog severity ================ ================= ``ERROR`` 3 (Error) ``WARNING`` 4 (Warning) ``INFO`` 6 (Informational) ``DEBUG`` 7 (Debug) ================ ================= On some systems, the syslog rejects messages that exceed a certain limit. For this reason, the forwarder truncates the message text to somewhat below 2048 Bytes, when logging to the system log. Messages are not truncated when logging to the Standard Error stream or to a file.