Mainflux.mainflux/pkg/transformers/json
Manuel Imperiale b18c9e79dc
NOISSUE - Add timestamp transformation rules for specifc JSON fields (#1514)
* NOISSUE - Add timestamp transformation rules for specifc JSON fields

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Add tests and defaults

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix Created

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix READMEs

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix Config file

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Improve the timestamp Config structure

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix typos

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix smpp

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Rm duplicated Unmarshal

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Extract the error var

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix review

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Return error if time field transformation fails

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2021-12-14 11:17:23 +01:00
..
README.md MF-1264 - Add support for JSON readers (#1295) 2020-12-30 15:43:04 +01:00
example_test.go MF-1264 - Add support for JSON readers (#1295) 2020-12-30 15:43:04 +01:00
message.go MF-1264 - Add support for JSON readers (#1295) 2020-12-30 15:43:04 +01:00
time.go NOISSUE - Add timestamp transformation rules for specifc JSON fields (#1514) 2021-12-14 11:17:23 +01:00
transformer.go NOISSUE - Add timestamp transformation rules for specifc JSON fields (#1514) 2021-12-14 11:17:23 +01:00
transformer_test.go NOISSUE - Add timestamp transformation rules for specifc JSON fields (#1514) 2021-12-14 11:17:23 +01:00

README.md

JSON Message Transformer

JSON Transformer provides Message Transformer for JSON messages. To transform Mainflux Message successfully, the payload must be a JSON object.

For the messages that contain JSON array as the root element, JSON Transformer does normalization of the data: it creates a separate JSON message for each JSON object in the root. In order to be processed and stored properly, JSON messages need to contain message format information. For the sake of the simpler storing of the messages, nested JSON objects are flatten to a single JSON object, using composite keys with the default separator /. This implies that the separator character (/) is not allowed in the JSON object key. For example, the following JSON object:

{
    "name": "name",
    "id":8659456789564231564,
    "in": 3.145,
    "alarm": true,
    "ts": 1571259850000,
    "d": {
        "tmp": 2.564,
        "hmd": 87,
        "loc": {
            "x": 1,
            "y": 2
        }
    }
}

will be transformed to:


{
    "name": "name",
    "id":8659456789564231564,
    "in": 3.145,
    "alarm": true,
    "ts": 1571259850000,
    "d/tmp": 2.564,
    "d/hmd": 87,
    "d/loc/x": 1,
    "d/loc/y": 2
}

The message format is stored in the subtopic. It's the last part of the subtopic. In the example:

http://localhost:8185/channels/<channelID>/messages/home/temperature/myFormat

the message format is myFormat. It can be any valid subtopic name, JSON transformer is format-agnostic. The format is used by the JSON message consumers so that they can process the message properly. If the format is not present (i.e. message subtopic is empty), JSON Transformer will report an error. Since the Transformer is agnostic to the format, having format in the subtopic does not prevent the publisher to send the content of different formats to the same subtopic. It's up to the consumer to handle this kind of issue. Message writers, for example, will store the message(s) in the table/collection/measurement (depending on the underlying database) with the name of the format (which in the example is myFormat). Mainflux writers will try to save any format received (whether it will be successful depends on the writer implementation and the underlying database), but it's recommended that the publisher takes care not to send different formats to the same subtopic.

Having a message format in the subtopic means that the subscriber has an option to subscribe to only one message format. This is a nice feature because message subscribers know what's the expected format of the message so that they can process it. If the message format is not important, wildcard subtopic can always be used to subscribe to any message format:

http://localhost:8185/channels/<channelID>/messages/home/temperature/*