Mainflux.mainflux/twins
b1ackd0t 43a263d7da
NOISSUE - Introduce NATS Jetstream as Default ES (#1907)
* Refactor message broker implementation

This commit refactors the Nats message broker implementation to include pubsub options. These changes include:

- Adding `Option` func that takes in the URL and prefix
- Implement `WithStream` option which can create a different stream for nats stream
- Implement `WithExchange` option which can create a different exchaange for rabbitmq channel
- Implement `WithPrefix` option which allows to you change the publisher prefix

These changes improve the organization and readability of the codebase.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Use redis URL to configure username, password and db

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Make event store configurable on dev deployment

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix adds options to messaging `PubSub` interface

Adding options to PubSub interface allows the use of messaging
package to do es.

The changes in this commit ensure that the code handles errors
properly and provides more informative error messages when
encountering unexpected types.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Add NATS event publisher implementation

This commit adds the implementation of the NATS event publisher.

The NATS event publisher is responsible for publishing events
to a NATS messaging system. It uses the `messaging`
package to interact with the messaging system.

The implementation includes the following features:

- Publishing events to NATS using the `Publish` method.
- Marshaling events to JSON before publishing.
- Setting the message subject and headers based on the event.
- Handling errors during publishing.

This implementation is built with the `!rabbitmq` build tag,
which means it will only be compiled if the `rabbitmq` build tag
is not present.

The NATS event publisher is part of the Mainflux events package
and provides support for the Mainflux NATS events source service functionality.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Add RabbitMQ event publisher implementation

This commit adds the implementation of the RabbitMQ event publisher.

The RabbitMQ event publisher is responsible for publishing events
to a RabbitMQ messaging system. It uses the `messaging`
package to interact with the messaging system.

The implementation includes the following features:

- Publishing events to RabbitMQ using the `Publish` method.
- Marshaling events to JSON before publishing.
- Setting the message subject and headers based on the event.
- Handling errors during publishing.

This implementation is built with the `rabbitmq` build tag,
which means it will only be compiled if the `rabbitmq` build tag
is present.

The RabbitMQ event publisher is part of the Mainflux events package
and provides support for the Mainflux RabbitMQ events source service functionality.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Add configurable implementation for events store

This commit adds a new file `brokers_*.go` which contains the implementation for the different event store. The file includes functions for creating a new publisher and subscriber using different es store.

This commit also includes an `init` function that logs a message indicating that the binary was built using the respective package as the events store.

The purpose of this commit is to add support for alternative events store.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix build flags

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Refactor Makefile and Semaphore configuration

The Makefile has been refactored to include the `MF_ES_STORE_TYPE` tag in the `go build` command. Additionally, the Semaphore configuration has been updated to include a new task for compiling with Redis as the broker type.

This commit addresses the need to compile the codebase with Redis as the event store type and includes the necessary changes in the Makefile and Semaphore configuration.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Reduced due to memory on testing

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix tests for es

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix grammar

Co-authored-by: Sammy Kerata Oina <44265300+SammyOina@users.noreply.github.com>
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix linting

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* feat(docker): update environment variables for message broker

The commit updates the environment variable `MF_ES_STORE_TYPE` in the `docker/.env` file. The variable is changed from `${MF_MQTT_BROKER_TYPE}` to `${MF_MESSAGE_BROKER_TYPE}` to accurately reflect the type of message broker being used. This change ensures that the correct message broker is configured for the Event Store.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* feat: Update docker environment variables

- Removed the unused MF_ES_URL variable in the .env file
- Updated the MF_ES_STORE_TYPE and MF_ES_STORE_URL variables in the .env file to match the MF_MESSAGE_BROKER_TYPE and MF_NATS_URL variables respectively

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix after rebase

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Add godocs for option parameters for brokers

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* pass by value exchange and prefix names

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Rename option functions

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* move variables to constants

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* fix: option example comment

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

---------

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
Co-authored-by: Sammy Kerata Oina <44265300+SammyOina@users.noreply.github.com>
2023-10-24 17:22:53 +02:00
..
api NOISSUE - Fix Twins Tests (#1931) 2023-10-20 23:06:03 +02:00
events NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
mocks NOISSUE - Fix Twins Tests (#1931) 2023-10-20 23:06:03 +02:00
mongodb NOISSUE - Change import name aliases (#1868) 2023-08-11 11:30:25 +02:00
tracing NOISSUE - Change import name aliases (#1868) 2023-08-11 11:30:25 +02:00
README.md NOISSUE - Refactor Redis URI (#1898) 2023-10-23 17:25:57 +02:00
doc.go MF-995 - Add Twins tests for endpoint list twins and list states (#1174) 2020-05-18 18:46:50 +02:00
service.go NOISSUE - Switch to Google Zanzibar Access control approach (#1919) 2023-10-15 22:02:13 +02:00
service_test.go NOISSUE - Fix Twins Tests (#1931) 2023-10-20 23:06:03 +02:00
states.go MF-1506 - Group-based Access Control (#1716) 2023-06-14 12:40:37 +02:00
twins.go MF-1506 - Group-based Access Control (#1716) 2023-06-14 12:40:37 +02:00

README.md

Twins

Service twins is used for CRUD and update of digital twins. Twin is a semantic representation of a real world data system consisting of data producers and consumers. It stores the sequence of attribute based definitions of a system and refers to a time series of definition based states that store the system historical data.

Configuration

The service is configured using the environment variables presented in the following table. Note that any unset variables will be replaced with their default values.

Variable Description Default
MF_TWINS_LOG_LEVEL Log level for twin service (debug, info, warn, error) info
MF_TWINS_HTTP_PORT Twins service HTTP port 9018
MF_TWINS_SERVER_CERT Path to server certificate in PEM format
MF_TWINS_SERVER_KEY Path to server key in PEM format
MF_JAEGER_URL Jaeger server URL http://jaeger:14268/api/traces
MF_TWINS_DB Database name mainflux
MF_TWINS_DB_HOST Database host address localhost
MF_TWINS_DB_PORT Database host port 27017
MF_THINGS_STANDALONE_ID User ID for standalone mode (no gRPC communication with users)
MF_THINGS_STANDALONE_TOKEN User token for standalone mode that should be passed in auth header
MF_TWINS_CLIENT_TLS Flag that indicates if TLS should be turned on false
MF_TWINS_CA_CERTS Path to trusted CAs in PEM format
MF_TWINS_CHANNEL_ID Message broker notifications channel ID
MF_MESSAGE_BROKER_URL Mainflux Message broker URL nats://localhost:4222
MF_AUTH_GRPC_URL Users service gRPC URL localhost:7001
MF_AUTH_GRPC_TIMEOUT Users service gRPC request timeout in seconds 1s
MF_TWINS_CACHE_URL Cache database URL redis://localhost:6379/0
MF_SEND_TELEMETRY Send telemetry to mainflux call home server true

Deployment

The service itself is distributed as Docker container. Check the twins service section in docker-compose to see how service is deployed.

To start the service outside of the container, execute the following shell script:

# download the latest version of the service
go get github.com/mainflux/mainflux

cd $GOPATH/src/github.com/mainflux/mainflux

# compile the twins
make twins

# copy binary to bin
make install

# set the environment variables and run the service
MF_TWINS_LOG_LEVEL=[Twins log level] \
MF_TWINS_HTTP_PORT=[Service HTTP port] \
MF_TWINS_SERVER_CERT=[String path to server cert in pem format] \
MF_TWINS_SERVER_KEY=[String path to server key in pem format] \
MF_JAEGER_URL=[Jaeger server URL] \
MF_TWINS_DB=[Database name] \
MF_TWINS_DB_HOST=[Database host address] \
MF_TWINS_DB_PORT=[Database host port] \
MF_THINGS_STANDALONE_EMAIL=[User email for standalone mode (no gRPC communication with auth)] \
MF_THINGS_STANDALONE_TOKEN=[User token for standalone mode that should be passed in auth header] \
MF_TWINS_CLIENT_TLS=[Flag that indicates if TLS should be turned on] \
MF_TWINS_CA_CERTS=[Path to trusted CAs in PEM format] \
MF_TWINS_CHANNEL_ID=[Message broker notifications channel ID] \
MF_MESSAGE_BROKER_URL=[Mainflux Message broker URL] \
MF_AUTH_GRPC_URL=[Users service gRPC URL] \
MF_AUTH_GRPC_TIMEOUT=[Users service gRPC request timeout in seconds] \
MF_TWINS_CACHE_URL=[Cache database URL] \
$GOBIN/mainflux-twins

Usage

Starting twins service

The twins service publishes notifications on a Message broker subject of the format channels.<MF_TWINS_CHANNEL_ID>.messages.<twinID>.<crudOp>, where crudOp stands for the crud operation done on twin - create, update, delete or retrieve - or state - save state. In order to use twin service notifications, one must inform it - via environment variables - about the Mainflux channel used for notification publishing. You must use an already existing channel, since you cannot know in advance or set the channel ID (Mainflux does it automatically).

To set the environment variable, please go to .env file and set the following variable:

MF_TWINS_CHANNEL_ID=

with the corresponding values of the desired channel. If you are running mainflux natively, than do the same thing in the corresponding console environment.

For more information about service capabilities and its usage, please check out the API documentation.