ACRN-Crashlog
#############
Introduction
************
``ACRN-Crashlog`` is a collective name for various tools (``acrnprobe``,
``usercrash_s``, ``usercrash_c``, ``debugger``, and more) and a overall
control utility called ``crashlogctl``. Together these tools collect logs
and information after each crash or event on an ACRN platform, including
the hypervisor, Service OS (SOS), and Android as a Guest (AaaG).
``ACRN-Crashlog`` provides a flexible way to configure which events are
of interest, by using an XML configuration file.
Building
********
Build dependencies
==================
The ``ACRN-Crashlog`` tool depends on the following libraries
(build and runtime):
- libevent
- OpenSSL
- libxml2
- systemd
- telemetrics-client-dev (optional, detected at build time)
- libblkid
- e2fsprogs
Refer to the :ref:`getting_started` for instructions on how to set-up your
build environment, and follow the instructions below to build and configure the
``ACRN-Crashlog`` tool.
Build
=====
To build the ``ACRN-Crashlog``, run:
.. code-block:: none
$ cd acrn-crashlog
$ make
To remove all generated files and return the folder to its clean state,
use:
.. code-block:: none
$ cd acrn-crashlog
$ make clean
Installing
**********
To install the build:
.. code-block:: none
$ cd acrn-crashlog
$ sudo make install
Enabling/Disabling
******************
To enable this tool:
.. code-block:: none
$ sudo crashlogctl enable
Then it will show:
.. code-block:: console
... Backup core pattern to /var/log/crashlog/default_core_pattern
'/usr/share/defaults/telemetrics/telemetrics.conf' ->
'/etc/telemetrics/telemetrics.conf'
... Set server_delivery_enabled=false in /etc/telemetrics/telemetrics.conf
... Set record_retention_enabled=true in /etc/telemetrics/telemetrics.conf
'/usr/share/acrn/crashlog/40-watchdog.conf' ->
'/etc/systemd/system.conf.d/40-watchdog.conf'
'/usr/share/acrn/crashlog/80-coredump.conf' ->
'/etc/sysctl.d/80-coredump.conf'
Created symlink /etc/systemd/system/hprobe.timer -> /dev/null.
Created symlink /etc/systemd/system/telemd-update-trigger.service -> /dev/null.
Created symlink /etc/systemd/system/pstore-clean.service -> /dev/null.
Created symlink /etc/systemd/system/pstore-probe.service -> /dev/null.
Created symlink /etc/systemd/system/oops-probe.service -> /dev/null.
Created symlink /etc/systemd/system/klogscanner.service -> /dev/null.
Created symlink /etc/systemd/system/journal-probe.service -> /dev/null.
Created symlink /etc/systemd/system/bert-probe.service -> /dev/null.
Created symlink /etc/systemd/system/multi-user.target.wants/acrnprobe.service -> /usr/lib/systemd/system/acrnprobe.service.
Created symlink /etc/systemd/system/multi-user.target.wants/usercrash.service -> /usr/lib/systemd/system/usercrash.service.
*** Please reboot your system. ***
Follow the hints to reboot the system:
.. code-block:: none
$ sudo reboot
To disable this tool:
.. code-block:: none
$ sudo crashlogctl disable
Then it will show:
.. code-block:: console
Removed /etc/systemd/system/multi-user.target.wants/acrnprobe.service.
Removed /etc/systemd/system/multi-user.target.wants/usercrash.service.
Removed /etc/systemd/system/hprobe.timer.
Removed /etc/systemd/system/telemd-update-trigger.service.
Removed /etc/systemd/system/pstore-clean.service.
Removed /etc/systemd/system/pstore-probe.service.
Removed /etc/systemd/system/oops-probe.service.
Removed /etc/systemd/system/klogscanner.service.
Removed /etc/systemd/system/journal-probe.service.
Removed /etc/systemd/system/bert-probe.service.
removed '/etc/sysctl.d/80-coredump.conf'
removed '/etc/systemd/system.conf.d/40-watchdog.conf'
... Set server_delivery_enabled=true in /etc/telemetrics/telemetrics.conf
... Set record_retention_enabled=false in /etc/telemetrics/telemetrics.conf
*** Please reboot your system. ***
Follow the hints to reboot the system:
.. code-block:: none
$ sudo reboot
To check the status of this tool:
.. code-block:: none
$ sudo crashlogctl is-active
It will show the status of the related services like:
.. code-block:: console
telemprobd : active
telempostd : active
acrnprobe : inactive
usercrash : inactive
Usage
*****
The ``acrnprobe`` tool can work in two ways according to the existence of
telemetrics-client on the system:
1. If telemetrics-client doesn't exist on the system, ``acrnprobe`` provides
``history_event`` (under ``/var/log/crashlog/history_event``) to manage the
crash and events records on the platform. But in this case, the records
can't be delivered to the backend.
2. If telemetrics-client exists on the system, ``acrnprobe`` works as a probe
of the telemetrics-client: it runs as a daemon autostarted when the system
boots, and sends the crashlog path to the telemetrics-client that records
events of interest and reports them to the backend using ``telemd`` the
telemetrics daemon. The work flow of ``acrnprobe`` and
telemetrics-client is shown in :numref:`crashlog-workflow`:
.. graphviz:: images/crashlog-workflow.dot
:name: crashlog-workflow
:align: center
:caption: acrnprobe and telemetrics-client workflow
Crashlog can be retrieved with ``telem_journal`` command:
.. code-block:: none
$ telem_journal -i
.. note::
For more details of telemetrics, please refer the `telemetrics-client`_ and
`telemetrics-backend`_ website.
``ACRN-Crashlog`` also provides a tool ``debugger`` to dump the specific
process information:
.. code-block:: none
$ debugger <pid>
.. note::
You need to be ``root`` to use the ``debugger``.
Source Code
***********
The source code structure:
.. code-block:: none
acrn-crashlog/
├── acrnprobe
│ └── include
├── common
│ └── include
├── data
└── usercrash
└── include
- ``acrnprobe``: to gather all the crash and event logs on the platform, and
probe on telemetrics-client. For the logs on hypervisor, it's collected with
acrnlog. For the log on SOS, the userspace crash log is collected with
usercrash, and the kernel crash log is collected with the inherent mechanism
like ``ipanic``, ``pstore`` and etc. For the log on AaaG, it's collected with
monitoring the change of related folders on the sos image, like
``/data/logs/``. ``acrnprobe`` also provides a flexible way to allow users to
configure which crash or event they want to collect through the xml file
easily.
- ``common``: some utils for logs, command and string.
- ``data``: configuration file, service files and shell script.
- ``usercrash``: to implement the tool which get the crash information for the
crashing process in userspace.
acrnprobe
=========
The ``acrnprobe`` detects all critical events on the platform and collects
specific information for debug purpose. These information would be saved as
logs, and the log path would be delivered to telemetrics-client as a record if
the telemetrics-client existed on the system.
For more detail on acrnprobe, please refer :ref:`acrnprobe_doc`.
usercrash
=========
The ``usercrash`` is a tool to get the crash info of the crashing process in
userspace. It works in Client/Server model. Server is autostarted, and client is
configured in ``core_pattern`` or ``coredump-wrapper``, which will be
triggered once crash occurs in userspace.
For more detail on ``usercrash``, please refer :ref:`usercrash_doc`.
.. _`telemetrics-client`: https://github.com/clearlinux/telemetrics-client
.. _`telemetrics-backend`: https://github.com/clearlinux/telemetrics-backend
e7605fad7d