How to index data from zigbee2mqtt?

swejoos — Wed, 20 Sep 2023 20:04:11 GMT

can't figure out how to indexing my data from zigbee2mgtt. The logs are exported from Home assistance via syslog, as Json.

I have tried various settings in props on the forwarder.

Current setting:

[zigbee2mqtt]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = JSON
category = structured
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = timestamp
LINE_BREAKER = ([\r\n]+)
disabled = false
pulldown_type = true

And on the search:

Current:
[zigbee2mqtt]
KV_MODE = JSON

And this is how the data appears in the log. for me it looks like some kind mix, not just JSON data.

Sep 20 19:13:19 linsrv 1 2023-09-20T17:13:19.941+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P001', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":7.82,"current":0,"device_temperature":25,"energy":7.82,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":3,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":234}'/n

host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt

Sep 20 19:08:13 linsrv06.hemdata.hemdata.se 1 2023-09-20T17:08:13.988+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P002', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":2.58,"current":0,"device_temperature":23,"energy":2.58,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":0,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":229}'/n

host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt

Sep 20 19:08:13 linsrv 1 2023-09-20T17:08:13.968+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P001', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":7.82,"current":0,"device_temperature":25,"energy":7.82,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":3,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":234}'/n

host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.logsourcetype = zigbee2mqtt

Sep 20 19:08:06 linsrv 1 2023-09-20T17:08:06.199+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P002', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":2.58,"current":0,"device_temperature":23,"energy":2.58,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":0,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":229}'/n

host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt

Re: How to index data from zigbee2mqtt?

richgalloway — Thu, 21 Sep 2023 19:26:20 GMT

Those props.conf settings should be on a heavy forwarder and/or an indexer. They do no good on a universal forwarder.

If the event is not pure and correct JSON then the INDEXED_EXTRACTIONS=JSON and KV_MODE=_json settings won't work.

Re: How to index data from zigbee2mqtt?

swejoos — Sun, 24 Sep 2023 18:52:58 GMT

Ok, Thanks.

So I should move all config to the search instead.

I have now tried that and the result seems to be the same, still index.

Re: How to index data from zigbee2mqtt?

richgalloway — Sun, 24 Sep 2023 23:11:40 GMT

As stated in my original reply, the settings go on indexers and/or heavy forwarders. You can put them on search heads, but they won't do any good. Unless, that is, you have a standalone system (combined indexer and search head).

Re: How to index data from zigbee2mqtt?

swejoos — Thu, 28 Sep 2023 13:08:58 GMT

ok. sorry,

But yes I have a combined index/search head, and a separate universal forwarder.

topic Re: How to index data from zigbee2mqtt? in Splunk Search

How to index data from zigbee2mqtt?

Re: How to index data from zigbee2mqtt?

Re: How to index data from zigbee2mqtt?

Re: How to index data from zigbee2mqtt?

Re: How to index data from zigbee2mqtt?