Commit Graph

22 Commits

Author SHA1 Message Date
Manuel Imperiale ce46723a8f
NOISSUE - Unify MF_INFLUX_READER_DB_HOST and MF_INFLUX_WRITER_DB_HOST envars (#1585)
Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2022-03-29 11:53:27 +02:00
Manuel Imperiale b18c9e79dc
NOISSUE - Add timestamp transformation rules for specifc JSON fields (#1514)
* NOISSUE - Add timestamp transformation rules for specifc JSON fields

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Add tests and defaults

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix Created

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix READMEs

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix Config file

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Improve the timestamp Config structure

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix typos

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix smpp

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Rm duplicated Unmarshal

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Extract the error var

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix review

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Return error if time field transformation fails

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2021-12-14 11:17:23 +01:00
Dušan Borovčanin 516c02bebe
MF-1378 - Update dependencies (#1379)
* Update dependencies

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix compose files and configs

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Upgrade image versions

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Update Postgres version

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Update test dependencies

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix fkey error handling

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>
2021-05-20 20:53:56 +02:00
Manuel Imperiale ca5ff63dfb
NOISSUE - Upgrade Mongo, Cassandra and Influx docker images (#1354)
* NOISSUE - Upgrade Mongo, Cassandra and Influx docker images

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use cassandra version 3.11.9

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Add MF_INFLUXDB_HTTP_AUTH_ENABLED envar

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use influxdb 1.8.4-alpine

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use influxdb 1.8.4

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2021-02-05 12:32:21 +01:00
Mirko Teodorovic 39a649c1bb
MF-1342 - Use environment variables in docker-compose to use tagged version of image (#1343)
* add MF_RELEASE_TAG

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* update readme file for MF_RELEASE_TAG

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* update readme file for MF_RELEASE_TAG

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* fix explanation, space

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* fix bad search/replace

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* fix bad search/replace

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* minor changes

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* fix readme

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* improve explanation for tag

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* replace snippet with link to code section

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* improve explanation for tag

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* replace snippet with link to code section

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* remove MF_RELEASE_TAG from service readme

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>
2021-02-02 16:26:02 +01:00
Manuel Imperiale 1bf485b71b
NOISSUE - Upgrade influxdb and postgres docker images (#1341)
* NOISSUE - Upgrade influxdb and postgres docker images

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Upgrade grafana version

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Upgrade influxdb to 1.8.3-alpine

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2021-01-29 12:37:50 +01:00
Dušan Borovčanin 5cfc9305e5
Make Transformer type configurable (#1331)
Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>
2021-01-20 13:10:40 +01:00
Dušan Borovčanin e326494166
MF-1264 - Add support for JSON readers (#1295)
* MF-1254 - Create universal JSON writer (#1260)

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Add JSON support to Readers

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix Influx Reader tests

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix messages format query

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix Postgres reader

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix Cassandra Readers and writers

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix Mongo reader

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Extract utility method to the JSON transformer

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix Influx and Postgres count

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Update JSON transformer

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix Influxdb Reader total count

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Refactor init.go for Cassandra writer

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Create a Payload type

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Add comments for defaults

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix variable declarations

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Replace interface{} with a new type

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Don't set channel just to overwrite it later

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Fix range search

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

* Rename Messages field

Signed-off-by: dusanb94 <dusan.borovcanin@mainflux.com>

Co-authored-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2020-12-30 15:43:04 +01:00
Manuel Imperiale 6c39c85cc8
MF-1096 - Fix AuthN and Things Auth ENVARS (#1066)
* NOISSUE - Fix AuthN and Things Auth ENVARS

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Add DB envars to env.go

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix envars

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Add DefLogLLevelError

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix DB names

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix DB names

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix DB names and HTTP ports

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix .env

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Revert writers DB names

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Rm unused Twins envars

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix typo

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Remove definitions from env.go

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Revert HTTP gRPC ports

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use default NATS URL as string

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Revert default ports

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix typo

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix cassandra ENVARS

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix cassandra reader ENVARS

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix readers and writers envars

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix readers and writers .env

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2020-04-07 12:02:17 +02:00
Manuel Imperiale 8325c1caf6
MF-1091 - Use channels. as broker prefix (#1098)
Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2020-04-06 12:30:26 +02:00
Jonathan Dreyer 46aadcfd9d
NOISSUE - Adding subtopics filtering in writer services (#1072)
* Add feature of filtering by subtopics in writer

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Fix mistake

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Refactoring writer sevices

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Rename variables related to filter (channels & subtopics)

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Set default value of filtering when configuration file doesn't exist

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Add a blank line at the end of the file

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Refactor loading filters configuration (moving into writer package, merge both loading methods & returning error)

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Remove useless log

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Change type of variables (channels & subtopics) and simplify loading method

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Add logging error when loading filters

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Simplify return configuration in loading method

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Merge both filter files into one file

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Move loading subjects into writer package

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Add subscribe to selected subjects

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Edit README of writer services

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Keep only subscribe loop

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Use full NATS subjects

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

* Edit comment in subjects files

Signed-off-by: Jonathan Dreyer <jonathan.dreyer@cleanenergie.ch>

Co-authored-by: Drasko DRASKOVIC <drasko.draskovic@gmail.com>
2020-03-30 18:24:27 +02:00
Dušan Borovčanin 8be2516321 MF-780 - Use Normalizer as a lib (#915)
* Use Normalizer as a lib

To normalize messages on the consumer side, Normalizer is moved
to the internal pkgs. Writers being message consumers are modified to
do message normalization instead of subscribing to normalized messages
subject.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix logging middleware for readers and writers

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove normalizer interface

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Use Normalizer in writers

As we agreed on #919, we'll use normalizer as an interface and provide
the default SenML implementation. Because of that, Normalizer is removed
from `internal` and we'll use the project structure proposed in the
aforementioned issue.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix tests

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove unused batch settings from influxDB reader

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update docs

Move Normalizer service to `addons`.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Rename channels input topic

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update Noramlizer docs

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove commented code

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update readers logging

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update addons docker-compose files

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update topcis explanations

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2019-10-31 14:04:47 +01:00
Manuel Imperiale e45e0ab307 NOISSUE - Fix InfluxDB env vars (#908)
Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2019-10-24 18:52:53 +02:00
Aleksandar Novaković 19f8922a9e
Update docker-compose version for addons (#874)
Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>
2019-10-02 11:57:12 +02:00
Manuel Imperiale f5b6c63cca NOISSUE - Use .env vars in docker-compose (#770)
* NOISSUE - Use .enr vars in docker-compose

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Add README and dupllicated .env in docker repo

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use .env for Bootstrap Docker Composition

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use .env for Cassandra (Reader-Writer) Docker Composition

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use .env for InfluxDB (Reader-Writer) Docker Composition

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use .env for MongoDB (Reader-Writer) Docker Composition

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use .env for Postgres (Reader-Writer) Docker Composition

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Use .env for LoRa adapter Docker Composition

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix Grafana port in .env file

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Update README.md

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>

* remove docker/.env and update README.md

Signed-off-by: Mirko Teodorovic <mirko.teodorovic@gmail.com>
2019-07-02 14:22:12 +02:00
Aleksandar Novaković 5e5fe88927 MF-708 - Assign Writer(s) to a channel (#737)
* Add support for channel filtering using config

Add support for channel filtering using yaml configuration files
for writers.

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update writers documentation with new env var

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Add info comment to configuration file

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Fix configuration and update yaml dep

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update config from yaml to toml

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>
2019-05-10 14:36:27 +02:00
Jovan Kostovski a30666d999 MF-655 Proper usage of docker volumes (#657)
* added filter and a switch for volume deletion

Signed-off-by: Jovan Kostovski <chombium@gmail.com>

* added persistent volume configuration

Signed-off-by: Jovan Kostovski <chombium@gmail.com>
2019-04-18 17:31:24 +02:00
Aleksandar Novaković ba4b166c96 Update docker-compose so that every service has debug log level (#453)
Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>
2018-11-08 21:22:59 +01:00
Dušan Borovčanin ec0a148ad4 MF-443 Update project dependencies (#444)
* Update project dependencies

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update InfluxDB and Cassandra versions

Update InfluxDB and Cassandra versions in docker-compose file as well as
images version in tests.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update gRPC version

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Make name first value of constraints

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>
2018-11-07 10:18:57 +01:00
Jovan Kostovski 8c0c994385 Expose the InfluxDB and Cassandra ports to host (#441)
* Expose the InfluxDB and Cassandra ports to host

Expose the InfluxDB's HTTP API port 8086 and  Cassandra's CQL port 9042 from the containers
to the host machine, in order to enable easier exploration of the data stored in the databases
from the host machine and development of apps which connect and use the data stored in the databases.

Signed-off-by: Jovan Kostovski <chombium@gmail.com>

* Port exposing cleanup in docker-compose.yml files

There were some doubled configurations for port exposing with the expose and ports directives in the docker-compose.yml files which had to be cleaned.

Signed-off-by: Jovan Kostovski <chombium@gmail.com>
2018-11-02 09:58:27 +01:00
Dušan Borovčanin 3de34062db NOISSUE - Influxdb batch add (#394)
* Add batch of streams

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Add env variables for batch point setup

Fix InfluxDB reader tests.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update Compose and Kubernetes config

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update env variables

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update docs

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove unused const

Refactor code and docs.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Handle NewBatchPoints error

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Reduce wait time in  batch save test

Fix typos.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update tests

Use single test with multiple cases to test single point save as well as
a batch of points.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Add an explanation for not resetting ticker

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2018-09-23 01:53:03 +02:00
Dušan Borovčanin 507bd2ee84 MF-311 - Implement basic InfluxDB reader (#365)
* Add InfluxDB reader

Summary:
- Add basic reader features
- Update Makefile

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Raise test coverage

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update README.md and docker composition

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix docker-compose.yml

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove exposed ports

Ports are already exposed by mapping, so no need to explicity expose them.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2018-08-25 12:48:03 +02:00