Commit Graph

19 Commits

Author SHA1 Message Date
Dušan Borovčanin 8c4da85039 MF-950 - Runtime error in normalizer - CBOR SenML (#974)
* Add Transformer tests

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix readers and writers

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix README typo

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Tidy vendor

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove link field from docs

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2019-11-29 20:47:28 +01:00
Aleksandar Novaković 263108cc85 Migrate from dep to go modules (#971)
Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>
2019-11-27 15:29:34 +01:00
Manuel Imperiale 87dd91e328
NOISSUE - Remove messages limit in influxdb-reader (#968)
* NOISSUE - Remove messages limit in influxdb-reader

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Rm too large limit test

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2019-11-26 18:58:45 +01:00
Dušan Borovčanin f50335ab33 MF-919 - Mainflux message updates (#924)
* Remove RawMessage

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove Normalizer

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update tests

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Replace normalizer with senml-transformer

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Rename Transformer interface and package

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update docs

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove SenML transformer service

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove SenML Protobuf support

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix readers

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix writers tests

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Refactor tests and remove normalizer

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update docs

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Reanme Service interface to Transformer

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Use msg instead of rawmsg

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove rawMsg from Aedes code

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix VerneMQ files

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove RawMessage code

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix missing subtopic return

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove remaining RawMessage reference

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix formatting

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix readers and writers tests

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Rename SenML transformer variables

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix readers and writers tests constants

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2019-11-05 11:57:16 +01:00
Dušan Borovčanin 8be2516321 MF-780 - Use Normalizer as a lib (#915)
* Use Normalizer as a lib

To normalize messages on the consumer side, Normalizer is moved
to the internal pkgs. Writers being message consumers are modified to
do message normalization instead of subscribing to normalized messages
subject.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix logging middleware for readers and writers

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove normalizer interface

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Use Normalizer in writers

As we agreed on #919, we'll use normalizer as an interface and provide
the default SenML implementation. Because of that, Normalizer is removed
from `internal` and we'll use the project structure proposed in the
aforementioned issue.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix tests

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove unused batch settings from influxDB reader

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update docs

Move Normalizer service to `addons`.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Rename channels input topic

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update Noramlizer docs

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove commented code

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update readers logging

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update addons docker-compose files

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update topcis explanations

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2019-10-31 14:04:47 +01:00
Aleksandar Novaković 61f1c2e379 MF-399 - Add open tracing support (#782)
* Add open tracing dependencies

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Add open tracing to users service

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Add open tracing to the things service

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Add open tracing to the http adapter

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Add open tracing to the ws adapter

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Add open tracing to the CoAP adapter

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Update LoRa adapter in accordance with changes

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Update SDK tests in accordance with changes

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Update bootstrap service in accordance with changes

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Update reader services with accordance with changes

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Update .env and docker-compose file

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Add jaeger and timeout env vars

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Fix broken test for can access by id endpoint

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>

* Update deps with proto empty package

Signed-off-by: Aleksandar Novakovic <anovakovic01@gmail.com>
2019-07-18 15:01:09 +02:00
Manuel Imperiale 138dae9277 NOISSUE - Fix Readers logs (#735)
* NOISSUE - Fix Readers logs

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix reviews

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2019-05-07 15:59:18 +02:00
Aleksandar Novaković aba7d909ed MF-525 - Add pagination response to the readers (#729)
* Add pagination response to the readers

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Remove println from influx reader

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>
2019-04-25 00:18:43 +02:00
beres 61b2d6b87b MF-596 - Add subtopic to RawMessage (#642)
* Commit for mainflux-596
Modified and tested:
- cli
- http
- mqtt
- normalizer
- all readers
- sdk messages
- all writers
- ws
Missing:
- coap
- lora

Signed-off-by: ale <ale@metaverso.org>

* - fix subtopic name in, when starting with dot, http/ws/mqtt
- add some test on readers

Signed-off-by: ale <ale@metaverso.org>

* - fix string concatenation
- update http/transport regexp to match subtopic names with only \w-
- update ws/transport regexp like http ones with also the wildcard * and >

Signed-off-by: ale <ale@metaverso.org>

* added subtopic support to coap adapter

Signed-off-by: ale <ale@metaverso.org>

* - update replace functions with replaceall when needed
- renamed getDestChannel to fmtSubject
- update api/transport and ws/transport route to be more readable
- fix mqtt syntax
- renamed func andQuery to query as suggested by @anovakovic01
- have a nice we :)

Signed-off-by: ale <ale@metaverso.org>

* - fix error declaration on ws/nat/publisher
- fix regexp added missing allowed chars - and _ on coap/api/transport
- fix subtopic clean suffix / if present on coap/api/transport
- improve regexp on http and ws /api/transport, now does not accept url that do not strictly match
- add some ws subtopic tests

Signed-off-by: ale <ale@metaverso.org>

* - enabled wildcard chars on coap/api/transport
- allow use special chars on http and ws api/transport

Signed-off-by: ale <ale@metaverso.org>

* - use strings.Replace() insted ReplaceAll()

Signed-off-by: ale <ale@metaverso.org>

* - allow every chars on subtopics
- fix replace error on mqtt

Signed-off-by: ale <ale@metaverso.org>

* fix cassandra test

Signed-off-by: ale <ale@metaverso.org>

* fix ws test with invalid subtopic

Signed-off-by: ale <ale@metaverso.org>

* fix invalid GOCACHE in go1.12, replaced by -count 1, see https://golang.org/doc/go1.10#test

Signed-off-by: ale <ale@metaverso.org>

* - improve regexp on http/ws api/transport
- minor changes

Signed-off-by: ale <ale@metaverso.org>

* - add generic function parseSubtopic on ws/http adapters

Signed-off-by: ale <ale@metaverso.org>

* - add generic function fmtSubtopic on coap adapter

Signed-off-by: ale <ale@metaverso.org>
2019-03-15 18:38:07 +01:00
Manuel Imperiale c8069827bb MF-506 - Fix influx & mongo readers queries to return most recent messages (#510)
* MF-506 - Fix fluxdb-reader query to return recent messages

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* MF-506 - Fix mongodb-reader query to return messages sorted by descendent time

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>

* Fix readers tests

Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com>
2018-12-18 15:17:55 +01:00
Aleksandar Novaković b9bf63e377 MF-475 - Replace increment ID with UUID (#490)
* Update increment ID to UUID in things service

Update increment ID to UUID for things and channels in things
service and proto files. Also, update ID type from uint to string.

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update increment ID to UUID in http adapter

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update increment ID to UUID in ws adapter

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update increment ID to UUID in CoAP adapter

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update increment ID to UUID in normalizer service

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update increment ID to UUID in writer services

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update increment ID to UUID in reader services

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update increment ID to UUID in SDK

Update increment ID to UUID in SDK. Update id type to string.
Update tests.

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update increment ID to UUID in mqtt adapter

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Remove unnecessary case from influxdb reader

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update tests in order to increase code coverage

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update lora adapter to use string ID instead of unsigned int

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>
2018-12-05 13:09:25 +01:00
Aleksandar Novaković 8ddd78c818 Add TLS support to CoAP adapter and all readers (#459)
* Add TLS support to CoAP adapter and all readers

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Add new env vars to README files

Add tls env vars to README files for CoAP adapter, WS adapter
and all readers.

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>

* Update all services to exit on invalid TLS env var value

Signed-off-by: Aleksandar Novakovic <aleksandar.novakovic@mainflux.com>
2018-11-20 12:52:28 +01:00
Dušan Borovčanin 0379eb6c0f MF-454 - Use message Time field as a time for InfluxDB points (#455)
* Use message time as Point time in InfluxDB

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Use actual message time

Update all Reader and Writer services to use time from the message
instead of time given from the corrseponding Writer service.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Remove message time check

Messages time less than 2**28 represent time relative to the current time so Writer used to convert this to the correct value, i.e. msg.Time += time.Now(). However, this step is optional and should really be a part of the app on top of Mainflux or could be introduced with minor changes in Normalizer, Reader or Writer services, so there is no need for this to be supported out of box.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Use channel and publisher as tag keys

Move all the other Message fields to the field keys.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>
2018-11-18 16:42:39 +01:00
Dušan Borovčanin ec0a148ad4 MF-443 Update project dependencies (#444)
* Update project dependencies

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update InfluxDB and Cassandra versions

Update InfluxDB and Cassandra versions in docker-compose file as well as
images version in tests.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update gRPC version

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Make name first value of constraints

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>
2018-11-07 10:18:57 +01:00
Dušan Borovčanin dfa766e013 MF-407 - Values of zero are being omitted (#434)
* Fix empty protobuf values

Update Normalizer service and .proto files. Reader and Writer services
needs to be updated due to message format change.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update HTTP adapter to use gogo protobuf

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update Reader services API

Update API or Reader services to match Message changes due to switching to gogo/proto.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update InfluxDB services

Update InfluxDB Reader and Writer services to match new Message format.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update MongoDB services

Update MongoDB Reader and Writer services to match new message format.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update Cassandra services

Update Cassandra Reader and Writer service to match new Message format.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Fix InfluxDB Reader test

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update Makefile and docs accordingly

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Fix possible data race in InfluxDB writer

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update InfluxDB Writer tests

Raise test coverage.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Fix InfluxDB reader

Fix wrong ValueSum readings. Upadete tests and raise coverage.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update MongoDB services tests

Raise test coverage for MongoDB Reader and Writer services.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update Readers API tests

Raise test coverage.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Raise test coverage

Update Cassandra Reader and Writer services tests.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Use gogo/protobuf in CoAP adapter

Add gogo/protobuf to Gopkg.toml and update dependencies.
Update Dockerfile to run `make proto`.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update Cassandra Reader tests

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Refactor code

Improve code style and comments to improve readability.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Rename Sum to SumValue

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Rename Values to Value

Since message contains only single value (or possibly no value at all),
`Values` name could be misleading. Rename simple double value from
`Value` to `FloatValue` accordingly.

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Fix InfluxDB Reader logging

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Replace exclusive if statements with switch-case

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>

* Update Cassandra services tests

Signed-off-by: Dusan Borovcanin <dusan.borovcanin@mainflux.com>
2018-11-05 19:18:51 +01:00
Michael Finley 6600d26ef1 MF-379 - Log Level Option (#402)
* Adding an option to select log level

Signed-off-by: Michael Finley <Michael.Finley@target.com>

* making error default log level

Signed-off-by: Michael Finley <Michael.Finley@target.com>

* removing go-kit/levels and expanding levels wrapper

Signed-off-by: Michael Finley <Michael.Finley@target.com>

* refactoring test cases and using log.fatal

Signed-off-by: Michael Finley <Michael.Finley@target.com>

* logger.new no longer accpets enum and now accepts string for level

Signed-off-by: Michael Finley <Michael.Finley@target.com>

* level_test.go refactor to compare error

Signed-off-by: Michael Finley <Michael.Finley@target.com>

* Updating the ws README

Signed-off-by: Michael Finley <Michael.Finley@target.com>

* Adding log level for mqtt Adapter

Signed-off-by: Michael Finley <Michael.Finley@target.com>
2018-09-26 18:58:51 +02:00
Dušan Borovčanin 3de34062db NOISSUE - Influxdb batch add (#394)
* Add batch of streams

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Add env variables for batch point setup

Fix InfluxDB reader tests.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update Compose and Kubernetes config

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update env variables

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update docs

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove unused const

Refactor code and docs.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Handle NewBatchPoints error

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Reduce wait time in  batch save test

Fix typos.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update tests

Use single test with multiple cases to test single point save as well as
a batch of points.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Add an explanation for not resetting ticker

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2018-09-23 01:53:03 +02:00
Ivan Milošević 8e9d7b4fdb MF-389 - Add documentation for Writers and Readers (#398)
* MF-389 - Add documentation for Writers and Readers

Signed-off-by: Ivan Milošević <iva@blokovi.com>

* MF-389 - Update readme files for readers and writers

Signed-off-by: Ivan Milošević <iva@blokovi.com>

* MF-389 - Update mongodb-reader docker-compose.yml

Signed-off-by: Ivan Milošević <iva@blokovi.com>

* Fix links formatting in docs

Signed-off-by: Ivan Milošević <iva@blokovi.com>

* Updated reader and writer docs

Signed-off-by: Ivan Milošević <iva@blokovi.com>

* Fix formatting

Signed-off-by: Ivan Milošević <iva@blokovi.com>

* Update Grafana access

Signed-off-by: Ivan Milošević <iva@blokovi.com>
2018-09-20 15:32:50 +02:00
Dušan Borovčanin 507bd2ee84 MF-311 - Implement basic InfluxDB reader (#365)
* Add InfluxDB reader

Summary:
- Add basic reader features
- Update Makefile

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Raise test coverage

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Update README.md and docker composition

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Fix docker-compose.yml

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

* Remove exposed ports

Ports are already exposed by mapping, so no need to explicity expose them.

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>
2018-08-25 12:48:03 +02:00