Splunk Search

How to index data from zigbee2mqtt?

swejoos
Observer

can't figure out how to indexing my data from zigbee2mgtt.  The logs are exported from Home assistance via syslog, as Json. 

I have tried various settings in props on the forwarder.

Current setting:

[zigbee2mqtt]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = JSON
category = structured
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = timestamp
LINE_BREAKER = ([\r\n]+)
disabled = false
pulldown_type = true

And on the search:

Current:
[zigbee2mqtt]
KV_MODE = JSON

And this is how the data appears in the log.  for me it looks like some kind mix, not just JSON data.

Sep 20 19:13:19 linsrv 1 2023-09-20T17:13:19.941+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P001', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":7.82,"current":0,"device_temperature":25,"energy":7.82,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":3,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":234}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt
 
Sep 20 19:08:13 linsrv06.hemdata.hemdata.se 1 2023-09-20T17:08:13.988+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P002', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":2.58,"current":0,"device_temperature":23,"energy":2.58,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":0,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":229}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt
 
Sep 20 19:08:13 linsrv 1 2023-09-20T17:08:13.968+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P001', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":7.82,"current":0,"device_temperature":25,"energy":7.82,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":3,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":234}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.logsourcetype = zigbee2mqtt
 
Sep 20 19:08:06 linsrv 1 2023-09-20T17:08:06.199+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P002', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":2.58,"current":0,"device_temperature":23,"energy":2.58,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":0,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":229}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt

 

Labels (1)
Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Those props.conf settings should be on a heavy forwarder and/or an indexer.  They do no good on a universal forwarder.

If the event is not pure and correct JSON then the INDEXED_EXTRACTIONS=JSON and KV_MODE=_json settings won't work.

---
If this reply helps you, Karma would be appreciated.
0 Karma

swejoos
Observer

Ok, Thanks.

So I should move all config to the search instead.

I have now tried that and the result seems to be the same, still index. 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

As stated in my original reply, the settings go on indexers and/or heavy forwarders.  You can put them on search heads, but they won't do any good.  Unless, that is, you have a standalone system (combined indexer and search head).

---
If this reply helps you, Karma would be appreciated.
0 Karma

swejoos
Observer

ok. sorry,

But yes I have a combined index/search head, and a separate universal forwarder.  

0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...