Go to file
b1ackd0t 43a263d7da
NOISSUE - Introduce NATS Jetstream as Default ES (#1907)
* Refactor message broker implementation

This commit refactors the Nats message broker implementation to include pubsub options. These changes include:

- Adding `Option` func that takes in the URL and prefix
- Implement `WithStream` option which can create a different stream for nats stream
- Implement `WithExchange` option which can create a different exchaange for rabbitmq channel
- Implement `WithPrefix` option which allows to you change the publisher prefix

These changes improve the organization and readability of the codebase.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Use redis URL to configure username, password and db

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Make event store configurable on dev deployment

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix adds options to messaging `PubSub` interface

Adding options to PubSub interface allows the use of messaging
package to do es.

The changes in this commit ensure that the code handles errors
properly and provides more informative error messages when
encountering unexpected types.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Add NATS event publisher implementation

This commit adds the implementation of the NATS event publisher.

The NATS event publisher is responsible for publishing events
to a NATS messaging system. It uses the `messaging`
package to interact with the messaging system.

The implementation includes the following features:

- Publishing events to NATS using the `Publish` method.
- Marshaling events to JSON before publishing.
- Setting the message subject and headers based on the event.
- Handling errors during publishing.

This implementation is built with the `!rabbitmq` build tag,
which means it will only be compiled if the `rabbitmq` build tag
is not present.

The NATS event publisher is part of the Mainflux events package
and provides support for the Mainflux NATS events source service functionality.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Add RabbitMQ event publisher implementation

This commit adds the implementation of the RabbitMQ event publisher.

The RabbitMQ event publisher is responsible for publishing events
to a RabbitMQ messaging system. It uses the `messaging`
package to interact with the messaging system.

The implementation includes the following features:

- Publishing events to RabbitMQ using the `Publish` method.
- Marshaling events to JSON before publishing.
- Setting the message subject and headers based on the event.
- Handling errors during publishing.

This implementation is built with the `rabbitmq` build tag,
which means it will only be compiled if the `rabbitmq` build tag
is present.

The RabbitMQ event publisher is part of the Mainflux events package
and provides support for the Mainflux RabbitMQ events source service functionality.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Add configurable implementation for events store

This commit adds a new file `brokers_*.go` which contains the implementation for the different event store. The file includes functions for creating a new publisher and subscriber using different es store.

This commit also includes an `init` function that logs a message indicating that the binary was built using the respective package as the events store.

The purpose of this commit is to add support for alternative events store.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix build flags

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Refactor Makefile and Semaphore configuration

The Makefile has been refactored to include the `MF_ES_STORE_TYPE` tag in the `go build` command. Additionally, the Semaphore configuration has been updated to include a new task for compiling with Redis as the broker type.

This commit addresses the need to compile the codebase with Redis as the event store type and includes the necessary changes in the Makefile and Semaphore configuration.

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Reduced due to memory on testing

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix tests for es

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix grammar

Co-authored-by: Sammy Kerata Oina <44265300+SammyOina@users.noreply.github.com>
Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix linting

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* feat(docker): update environment variables for message broker

The commit updates the environment variable `MF_ES_STORE_TYPE` in the `docker/.env` file. The variable is changed from `${MF_MQTT_BROKER_TYPE}` to `${MF_MESSAGE_BROKER_TYPE}` to accurately reflect the type of message broker being used. This change ensures that the correct message broker is configured for the Event Store.

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* feat: Update docker environment variables

- Removed the unused MF_ES_URL variable in the .env file
- Updated the MF_ES_STORE_TYPE and MF_ES_STORE_URL variables in the .env file to match the MF_MESSAGE_BROKER_TYPE and MF_NATS_URL variables respectively

Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Fix after rebase

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Add godocs for option parameters for brokers

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* pass by value exchange and prefix names

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* Rename option functions

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* move variables to constants

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

* fix: option example comment

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

---------

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>
Signed-off-by: Rodney Osodo <28790446+rodneyosodo@users.noreply.github.com>
Signed-off-by: rodneyosodo <blackd0t@protonmail.com>
Co-authored-by: Sammy Kerata Oina <44265300+SammyOina@users.noreply.github.com>
2023-10-24 17:22:53 +02:00
.github Remove codeball (#1806) 2023-05-31 14:57:21 +02:00
.semaphore NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
api NOISSUE: Listing of shared things with users & Update SDK (#1923) 2023-10-17 15:38:06 +02:00
auth NOISSUE - Fix Linting (#1938) 2023-10-21 23:26:47 +02:00
bootstrap NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
certs NOISSUE - Uncomment Code (#1926) 2023-10-18 16:45:08 +02:00
cli NOISSUE: Listing of shared things with users & Update SDK (#1923) 2023-10-17 15:38:06 +02:00
cmd NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
coap NOISSUE - Add Subscriber Config (#1896) 2023-10-23 15:27:15 +02:00
consumers NOISSUE - Add Subscriber Config (#1896) 2023-10-23 15:27:15 +02:00
docker NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
http UV-267 - Update http messaging to mproxy (#1916) 2023-10-23 14:35:15 +02:00
internal NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
logger NOISSUE - Add More Linters (#1924) 2023-10-16 11:43:33 +02:00
lora NOISSUE - Refactor Redis URI (#1898) 2023-10-23 17:25:57 +02:00
mqtt NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
opcua NOISSUE - Refactor Redis URI (#1898) 2023-10-23 17:25:57 +02:00
pkg NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
provision NOISSUE - Add More Linters (#1924) 2023-10-16 11:43:33 +02:00
readers NOISSUE - Fix Readers Tests (#1933) 2023-10-23 15:36:42 +02:00
scripts NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
things NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
tools NOISSUE - Fix Linting (#1938) 2023-10-21 23:26:47 +02:00
twins NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
users NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
vendor UV-267 - Update WS service to mproxy (#1917) 2023-10-23 16:22:09 +02:00
ws UV-267 - Update WS service to mproxy (#1917) 2023-10-23 16:22:09 +02:00
.dockerignore Fixed docs instructions in README (#1010) 2020-01-17 14:44:13 +01:00
.gitignore MF-1506 - Group-based Access Control (#1716) 2023-06-14 12:40:37 +02:00
.golangci.yml NOISSUE - Uncomment Code (#1926) 2023-10-18 16:45:08 +02:00
ADOPTERS.md MF-1365 - Add ADOPTERS.md file (#1371) 2021-02-25 13:46:07 +01:00
CHANGELOG.md NOISSUE - Update changelog for release 0.13.0 (#1595) 2022-04-15 15:51:35 +02:00
CONTRIBUTING.md Corrected grammar and rephrased a few sentences to read nicely (#641) 2019-03-07 16:25:37 +01:00
LICENSE Fix copyright year (#1009) 2020-01-17 12:51:01 +01:00
MAINTAINERS Update MAINTAINERS 2018-08-28 22:42:47 +02:00
Makefile NOISSUE - Introduce NATS Jetstream as Default ES (#1907) 2023-10-24 17:22:53 +02:00
README.md MF-1830 - Switch to Semaphore 2.0 (#1829) 2023-07-26 16:27:11 +02:00
api.go MF-788 - Remove date and minimize copyright comments (#876) 2019-10-07 16:14:47 +02:00
auth.pb.go NOISSUE - Switch to Google Zanzibar Access control approach (#1919) 2023-10-15 22:02:13 +02:00
auth.proto NOISSUE - Switch to Google Zanzibar Access control approach (#1919) 2023-10-15 22:02:13 +02:00
auth_grpc.pb.go NOISSUE - Switch to Google Zanzibar Access control approach (#1919) 2023-10-15 22:02:13 +02:00
config.toml NOISSUE - Update Auth ENV Vars in Docker-Compose (#1937) 2023-10-18 22:36:41 +02:00
doc.go MF-788 - Remove date and minimize copyright comments (#876) 2019-10-07 16:14:47 +02:00
go.mod UV-267 - Update WS service to mproxy (#1917) 2023-10-23 16:22:09 +02:00
go.sum UV-267 - Update WS service to mproxy (#1917) 2023-10-23 16:22:09 +02:00
health.go NOISSUE - Add instance ID (#1776) 2023-07-07 11:14:55 +02:00
uuid.go NOISSUE - Rename package aliases uuidProvider into uuid (#1323) 2021-01-17 23:12:45 +01:00

README.md

Mainflux

Build Status go report card coverage license chat

banner

Mainflux is modern, scalable, secure, open-source, and patent-free IoT cloud platform written in Go.

It accepts user and thing (sensor, actuator, application) connections over various network protocols (i.e. HTTP, MQTT, WebSocket, CoAP), thus making a seamless bridge between them. It is used as the IoT middleware for building complex IoT solutions.

For more details, check out the official documentation.

Features

  • Multi-protocol connectivity and bridging (HTTP, MQTT, WebSocket and CoAP)
  • Device management and provisioning (Zero Touch provisioning)
  • Mutual TLS Authentication (mTLS) using X.509 Certificates
  • Fine-grained access control (policies, ABAC/RBAC)
  • Message persistence (Cassandra, InfluxDB, MongoDB and PostgresSQL)
  • Platform logging and instrumentation support (Prometheus and OpenTelemetry)
  • Event sourcing
  • Container-based deployment using Docker and Kubernetes
  • LoRaWAN network integration
  • OPC UA integration
  • Edge Agent and Export services for remote IoT gateway management and edge computing
  • SDK
  • CLI
  • Small memory footprint and fast execution
  • Domain-driven design architecture, high-quality code and test coverage

Prerequisites

The following are needed to run Mainflux:

Developing Mainflux will also require:

Install

Once the prerequisites are installed, execute the following commands from the project's root:

docker-compose -f docker/docker-compose.yml up

This will bring up the Mainflux docker services and interconnect them. This command can also be executed using the project's included Makefile:

make run

If you want to run services from specific release checkout code from github and make sure that MF_RELEASE_TAG in .env is being set to match the release version

git checkout tags/<release_number> -b <release_number>
# e.g. `git checkout tags/0.13.0 -b 0.13.0`

Check that .env file contains:

MF_RELEASE_TAG=<release_number>

docker-compose should be used for development and testing deployments. For production we suggest using Kubernetes.

Usage

The quickest way to start using Mainflux is via the CLI. The latest version can be downloaded from the official releases page.

It can also be built and used from the project's root directory:

make cli
./build/mainflux-cli version

Additional details on using the CLI can be found in the CLI documentation.

Documentation

Official documentation is hosted at Mainflux official docs page. Documentation is auto-generated, checkout the instructions on official docs repository:

If you spot an error or a need for corrections, please let us know - or even better: send us a PR.

Authors

Main architect and BDFL of Mainflux project is @drasko.

Additionally, @nmarcetic and @janko-isidorovic assured overall architecture and design, while @manuio and @darkodraskovic helped with crafting initial implementation and continuously worked on the project evolutions.

Besides them, Mainflux is constantly improved and actively developed by @anovakovic01, @dusanb94, @srados, @gsaleh, @blokovi, @chombium, @mteodor and a large set of contributors.

Maintainers are listed in MAINTAINERS file.

The Mainflux team would like to give special thanks to @mijicd for his monumental work on designing and implementing a highly improved and optimized version of the platform, and @malidukica for his effort on implementing the initial user interface.

Professional Support

There are many companies offering professional support for the Mainflux system.

If you need this kind of support, best is to reach out to @drasko directly, and he will point you out to the best-matching support team.

Contributing

Thank you for your interest in Mainflux and the desire to contribute!

  1. Take a look at our open issues. The good-first-issue label is specifically for issues that are great for getting started.
  2. Checkout the contribution guide to learn more about our style and conventions.
  3. Make your changes compatible to our workflow.

We're Hiring

You like Mainflux and you would like to make it your day job? We're always looking for talented engineers interested in open-source, IoT and distributed systems. If you recognize yourself, reach out to @drasko - he will contact you back.

The best way to grab our attention is, of course, by sending PRs 😎.

Community

License

Apache-2.0

FOSSA Status

Data Collection for Mainflux

Mainflux is committed to continuously improving its services and ensuring a seamless experience for its users. To achieve this, we collect certain data from your deployments. Rest assured, this data is collected solely for the purpose of enhancing Mainflux and is not used with any malicious intent. The deployment summary can be found on our website.

The collected data includes:

  • IP Address - Used for approximate location information on deployments.
  • Services Used - To understand which features are popular and prioritize future developments.
  • Last Seen Time - To ensure the stability and availability of Mainflux.
  • Mainflux Version - To track the software version and deliver relevant updates.

We take your privacy and data security seriously. All data collected is handled in accordance with our stringent privacy policies and industry best practices.

Data collection is on by default and can be disabled by setting the env variable: MF_SEND_TELEMETRY=false

By utilizing Mainflux, you actively contribute to its improvement. Together, we can build a more robust and efficient IoT platform. Thank you for your trust in Mainflux!