Hi @luminousplumz You need to apply the mqtttopic transform before the mqtttojson transform overwrites the _raw field. The order in TRANSFORMS-* matters. Also, adjust the mqtttopic regex and format...
See more...
Hi @luminousplumz You need to apply the mqtttopic transform before the mqtttojson transform overwrites the _raw field. The order in TRANSFORMS-* matters. Also, adjust the mqtttopic regex and format for correct field extraction. transforms.conf: [mqtttojson]
REGEX = msg\=(.+)
FORMAT = $1
DEST_KEY = _raw
[mqtttopic]
# Extract from the original _raw field containing 'topic='
REGEX = topic=tgw\/data\/0x155f\/([^\/]+)
FORMAT = Topic::$1
WRITE_META = true props.conf: [mqtttojson_ubnpfc_all]
# Apply mqtttopic first, then mqtttojson
TRANSFORMS-topic_then_json = mqtttopic, mqtttojson
# The rest of your props.conf settings remain the same
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIME_PREFIX = \"ts\":
TZ = Europe/London
category = Custom
pulldown_type = 1
# Ensure KV_MODE=none if you don't want Splunk's default key-value extraction
# KV_MODE = none
# Ensure JSON extraction runs after transforms if needed
# INDEXED_EXTRACTIONS = json Transform Order: The TRANSFORMS-topic_then_json line in props.conf should have mqtttopic first. This ensures it runs on the original event data before mqtttojson overwrites _raw with the JSON payload. mqtttopic REGEX: The regex topic=tgw\/data\/0x155f\/([^\/]+) specifically looks for the topic= string, skips the known prefix tgw/data/0x155f/, and captures the next segment of characters that are not a forward slash (/) into capture group 1. mqtttopic FORMAT: FORMAT = Topic::$1 creates a new field named Topic containing the value captured by the regex (the desired topic segment, e.g., "TransportContextTracking"). mqtttopic WRITE_META: WRITE_META = true ensures the extracted field (Topic) is written to the index metadata, making it available for searching even though the original _raw field is later overwritten. mqtttojson: This transform runs second. It extracts the JSON part from the msg= field (which still exists in the original event data at this stage) and overwrites _raw with just the JSON content. Splunk's automatic JSON parsing (or INDEXED_EXTRACTIONS = json) will then parse this new _raw. Some useful tips: Restart the Splunk instance or reload the configuration for changes in props.conf and transforms.conf to take effect. Ensure the sourcetype mqtttojson_ubnpfc_all is correctly assigned to your MQTT data input. Test the regex using Splunk's rex command in search or on regex testing websites against your raw event data to confirm it captures the correct value. If Splunk's automatic key-value extraction interferes before your transforms run, you might need KV_MODE = none in props.conf. If Splunk isn't automatically parsing the final JSON _raw, add INDEXED_EXTRACTIONS = json to your props.conf stanza. Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing