zephyr/scripts/ci/es_upload/README.md

ElasticSearch index map examples

This directory contains ElasticSearch data store index map examples for use by upload_test_results_es.py script for Twister JSON reports upload.

An index map file allows to create destination index files on ElasticSearch server with explicit scheme having all required Twister JSON report objects associated to proper data types, eventually to store the expected document structure. Besides, it allows to track changes in Twister JSON report scheme and the corresponding data scheme in the same source code repository. An alternative way is to have index files created at the ElasticSearch server by other tools, or rely on default data mapping with potential type misalignments.

The command examples below are simplified with parameters essential for data transformations. For other command line options and more details run the upload script with --help, or read its source code with inline comments there.

Use these examples as a reference for your setup. Always pay attention to possible changes in the current Twister report format and in the upload script itself. Tune resulting data scheme and size depending on your particular needs.

It is recommended to try the upload script with --dry-run option first to check the resulting data without its actual upload.

You should set environment variables ELASTICSEARCH_SERVER and ELASTICSEARCH_KEY to connect with your server running the upload script.

Index creation with a map file

Execute the upload script once for the first time to create your index at the ElasticSearch server with the appropriate map file, for example:

python3 ./scripts/ci/upload_test_results_es.py --create-index \
  --index YOUR_INDEX_NAME \
  --map-file YOUR_INDEX_MAP.json

Upload transformations

The upload script has several command line options to change twister.json data: exclude excess data, extract substrings by regular expressions, change data structure making it suitable for Kibana default visual components (see --help for more details):

• --exclude removes excess testsuite properties not needed to store them at Elasticsearch index.

• --transform applies regexp group parsing rules to string properties extracting derived object properties.

• --flatten changes testsuite data structure in regard of one of its list components: either testcases or recording, so each item there becomes an independent data record inheriting all other testsuite properties, whereas the children object's properties are renamed with the parent object's name as a prefix: testcases_ or recording_ respectively. Only one testsuite property object can be flattened this way per index upload. All other testsuite objects will be treated according to the index structure.

• Other command line options: --flatten-dict-name, --flatten-list-names, --flatten-separator, --transpose-separator, --escape-separator.

Index map examples and use cases

Twister test results

Create the index:

python3 ./scripts/ci/upload_test_results_es.py --create-index \
    --index zephyr-test-example \
    --map-file zephyr_twister_index.json

Upload Twister test results 'as-is', without additional transformations:

python3 ./scripts/ci/upload_test_results_es.py \
    --index zephyr-test-example \
    ./twister-out/**/twister.json

Twister tests with recording

Store test results with recording data entries, for example from Kernel Timer Behavior test suite.

Create the index:

python3 ./scripts/ci/upload_test_results_es.py --create-index \
    --index zephyr-test-recording-example-1 \
    --map-file zephyr_twister_flat_recording_metrics_index.json

Upload data with 'flattened' test suites creating documents for each recording data entry.

python3 ./scripts/ci/upload_test_results_es.py \
    --index zephyr-test-recording-example-1 \
    --exclude path run_id \
    --flatten recording \
    ./twister-out/**/twister.json

Twister test with recording and extracting more data

Store test results with recording data entries which are also scanned by regular expressions to extract embedded values, for example Kernel Latency Benchmarks test suite.

Create the index:

python3 ./scripts/ci/upload_test_results_es.py --create-index \
    --index zephyr-test-recording-example-2 \
    --map-file zephyr_twister_flat_recording_index.json

Upload data with 'flattened' test suites creating documents for each record data entry with 3 additional data properties extracted by regular expressions.

python3 ./scripts/ci/upload_test_results_es.py \
    --index zephyr-test-recording-example-2 \
    --exclude path run_id \
    --flatten recording \
    --transform "{ 'recording_metric': '(?P<recording_metric_object>[^\.]+)\.(?P<recording_metric_action>[^\.]+)\.(?P<recording_metric_details>[^ -]+)' }" \
    ./twister-out/**/twister.json

Memory Footprint results

To store Memory Footprint data reported in twister-footprint.json (see Twister --footprint-report).

Create the index:

python3 ./scripts/ci/upload_test_results_es.py --create-index \
    --index zephyr-memory-footprint-example \
    --map-file zephyr_twister_flat_footprint_index.json

Upload data with 'flattened' footprint entries creating documents for each of them.

python3 ./scripts/ci/upload_test_results_es.py  \
    --index zephyr-memory-footprint-example \
    --exclude path run_id \
    --flatten footprint \
    --flatten-list-names "{'children':'name'}" \
    --transform "{ 'footprint_name': '^(?P<footprint_area>([^\/]+\/){0,2})(?P<footprint_path>([^\/]*\/)*)(?P<footprint_symbol>[^\/]*)$' }" \
    ../footprint_data/**/twister_footprint.json