I have an requirement to extract a value from an mqtt string before i parse it to json.
Initially i was using MQTT Modular input app to pull each of the topics with their own input.
I found that with more than 3 inputs /topics enabled i am dropping some if not all data.
So i decided to pull all the topics in a single input. This works well except i still need to be able to separate the topics for searches.
I managed to get this working using multiple transforms. i changed something and now i can get it to work again.
Using Transforms i can parse to json with no issues (mqtttojson)
Transforms.conf
[mqtttojson]
REGEX = msg\=(.+)$
FORMAT = $1
DEST_KEY = _raw
[mqtttopic]
CLEAN_KEYS = 0
FORMAT = Topic::"$1"
REGEX = tgw\/data\/0x155f\/(?<Topic>\S*?)\/
Props.conf
In the example below i need the 4th topic level i.e. "TransportContextTracking".
Thu Apr 24 12:42:15 GMT 2025 name=mqtt_msg_received event_id= topic=tgw/data/0x155f/TransportContextTracking/MFC/0278494 msg={"data":{"destination":{"locationAddress":"/UrbanUK/PCOTS13/Exit"},"errorCode":null,"event":"Started","loadCarrierId":"0278494","source":{"locationAddress":"/UrbanUK/PCOTS13/Pick"},"transportId":"f0409b2a-e9d4-407c-bd65-48ccea17b520","transportType":"Transport"},"dbid":8104562815,"ts":1745498528217}
What am i missing ?????
You need to apply the mqtttopic transform before the mqtttojson transform overwrites the _raw field. The order in TRANSFORMS-* matters. Also, adjust the mqtttopic regex and format for correct field extraction.
transforms.conf:
[mqtttojson] REGEX = msg\=(.+) FORMAT = $1 DEST_KEY = _raw [mqtttopic] # Extract from the original _raw field containing 'topic=' REGEX = topic=tgw\/data\/0x155f\/([^\/]+) FORMAT = Topic::$1 WRITE_META = true
props.conf:
[mqtttojson_ubnpfc_all] # Apply mqtttopic first, then mqtttojson TRANSFORMS-topic_then_json = mqtttopic, mqtttojson # The rest of your props.conf settings remain the same DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true TIME_PREFIX = \"ts\": TZ = Europe/London category = Custom pulldown_type = 1 # Ensure KV_MODE=none if you don't want Splunk's default key-value extraction # KV_MODE = none # Ensure JSON extraction runs after transforms if needed # INDEXED_EXTRACTIONS = json
Some useful tips:
🌟 Did this answer help you? If so, please consider:
Your feedback encourages the volunteers in this community to continue contributing
Hi @luminousplumz,
For index-time field extractions, you want something like this (note the order of the transforms in the TRANSFORMS-mqtt setting):
# fields.conf
[sourcetype::mqtttojson_ubnpfc_all::Topic]
INDEXED = true
# props.conf
[mqtttojson_ubnpfc_all]
TRANSFORMS-mqtt = mqtttopic,mqtttojson
# transforms.conf
[mqtttojson]
CLEAN_KEYS = 0
DEST_KEY = _raw
FORMAT = $1
REGEX = msg=(.+)
[mqtttopic]
CLEAN_KEYS = 0
FORMAT = Topic::$1
REGEX = topic=(?:[^/]*/){3}([^/]+)
WRITE_META = true
For search-time field extractions, you want something like this:
[mqtttojson_ubnpfc_all]
EXTRACT-Topic = topic=(?:[^/]*/){3}(?<Topic>[^/]+)
EVAL-_raw = replace(_raw, ".*? msg=", "")
However, in the search-time configuration, you'll need to extract the JSON fields in a search as automatic key-value field extraction happens before calculated fields (EVAL-*):
sourcetype=mqtttojson_ubnpfc_all
| spath
You'll note that the original name, event_id, topic, and msg (value possibly truncated) fields are automatically extracted before the full value of msg is assigned to _raw.