Splunk Search

How to index data from zigbee2mqtt?

swejoos
Observer

can't figure out how to indexing my data from zigbee2mgtt.  The logs are exported from Home assistance via syslog, as Json. 

I have tried various settings in props on the forwarder.

Current setting:

[zigbee2mqtt]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = JSON
category = structured
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = timestamp
LINE_BREAKER = ([\r\n]+)
disabled = false
pulldown_type = true

And on the search:

Current:
[zigbee2mqtt]
KV_MODE = JSON

And this is how the data appears in the log.  for me it looks like some kind mix, not just JSON data.

Sep 20 19:13:19 linsrv 1 2023-09-20T17:13:19.941+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P001', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":7.82,"current":0,"device_temperature":25,"energy":7.82,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":3,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":234}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt
 
Sep 20 19:08:13 linsrv06.hemdata.hemdata.se 1 2023-09-20T17:08:13.988+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P002', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":2.58,"current":0,"device_temperature":23,"energy":2.58,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":0,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":229}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt
 
Sep 20 19:08:13 linsrv 1 2023-09-20T17:08:13.968+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P001', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":7.82,"current":0,"device_temperature":25,"energy":7.82,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":3,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":234}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.logsourcetype = zigbee2mqtt
 
Sep 20 19:08:06 linsrv 1 2023-09-20T17:08:06.199+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P002', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":2.58,"current":0,"device_temperature":23,"energy":2.58,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":0,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":229}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt

 

Labels (1)
Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Those props.conf settings should be on a heavy forwarder and/or an indexer.  They do no good on a universal forwarder.

If the event is not pure and correct JSON then the INDEXED_EXTRACTIONS=JSON and KV_MODE=_json settings won't work.

---
If this reply helps you, Karma would be appreciated.
0 Karma

swejoos
Observer

Ok, Thanks.

So I should move all config to the search instead.

I have now tried that and the result seems to be the same, still index. 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

As stated in my original reply, the settings go on indexers and/or heavy forwarders.  You can put them on search heads, but they won't do any good.  Unless, that is, you have a standalone system (combined indexer and search head).

---
If this reply helps you, Karma would be appreciated.
0 Karma

swejoos
Observer

ok. sorry,

But yes I have a combined index/search head, and a separate universal forwarder.  

0 Karma
Get Updates on the Splunk Community!

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...