Splunk Search

How to index data from zigbee2mqtt?

swejoos
Observer

can't figure out how to indexing my data from zigbee2mgtt.  The logs are exported from Home assistance via syslog, as Json. 

I have tried various settings in props on the forwarder.

Current setting:

[zigbee2mqtt]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = JSON
category = structured
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = timestamp
LINE_BREAKER = ([\r\n]+)
disabled = false
pulldown_type = true

And on the search:

Current:
[zigbee2mqtt]
KV_MODE = JSON

And this is how the data appears in the log.  for me it looks like some kind mix, not just JSON data.

Sep 20 19:13:19 linsrv 1 2023-09-20T17:13:19.941+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P001', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":7.82,"current":0,"device_temperature":25,"energy":7.82,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":3,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":234}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt
 
Sep 20 19:08:13 linsrv06.hemdata.hemdata.se 1 2023-09-20T17:08:13.988+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P002', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":2.58,"current":0,"device_temperature":23,"energy":2.58,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":0,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":229}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt
 
Sep 20 19:08:13 linsrv 1 2023-09-20T17:08:13.968+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P001', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":7.82,"current":0,"device_temperature":25,"energy":7.82,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":3,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":234}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.logsourcetype = zigbee2mqtt
 
Sep 20 19:08:06 linsrv 1 2023-09-20T17:08:06.199+02:00 localhost Zigbee2MQTT - - - MQTT publish: topic 'zigbee2mqtt/P002', payload '{"auto_off":null,"button_lock":null,"consumer_connected":true,"consumption":2.58,"current":0,"device_temperature":23,"energy":2.58,"led_disabled_night":null,"linkquality":255,"overload_protection":null,"power":0,"power_outage_count":0,"power_outage_memory":null,"state":"OFF","update":{"installed_version":41,"latest_version":32,"state":"idle"},"update_available":false,"voltage":229}'/n
host = linsrv index = zigbee source = /disk1/syslog/in/linsrv/2023-09-20/messages.log sourcetype = zigbee2mqtt

 

Labels (1)
Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Those props.conf settings should be on a heavy forwarder and/or an indexer.  They do no good on a universal forwarder.

If the event is not pure and correct JSON then the INDEXED_EXTRACTIONS=JSON and KV_MODE=_json settings won't work.

---
If this reply helps you, Karma would be appreciated.
0 Karma

swejoos
Observer

Ok, Thanks.

So I should move all config to the search instead.

I have now tried that and the result seems to be the same, still index. 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

As stated in my original reply, the settings go on indexers and/or heavy forwarders.  You can put them on search heads, but they won't do any good.  Unless, that is, you have a standalone system (combined indexer and search head).

---
If this reply helps you, Karma would be appreciated.
0 Karma

swejoos
Observer

ok. sorry,

But yes I have a combined index/search head, and a separate universal forwarder.  

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...