topic Re: How to extract key/value with dynamic key name? in Splunk Search

How to extract key/value with dynamic key name?

oliverja — Wed, 26 Oct 2022 14:09:09 GMT

I found this, but I am unable to replicate it.

I am not understanding where I am messing up here.

Problem:

I feed btool outputs into splunk, and chop them up by stanza.

/opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf [xmlsecurity_eventcode_errorcode_action_lookup] /opt/splunk/etc/system/default/transforms.conf CAN_OPTIMIZE = True /opt/splunk/etc/system/default/transforms.conf CLEAN_KEYS = True /opt/splunk/etc/system/default/transforms.conf DEFAULT_VALUE = /opt/splunk/etc/system/default/transforms.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/system/default/transforms.conf DEST_KEY = /opt/splunk/etc/system/default/transforms.conf FORMAT = /opt/splunk/etc/system/default/transforms.conf KEEP_EMPTY_VALS = False /opt/splunk/etc/system/default/transforms.conf LOOKAHEAD = 4096 /opt/splunk/etc/system/default/transforms.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/transforms.conf MV_ADD = False /opt/splunk/etc/system/default/transforms.conf REGEX = /opt/splunk/etc/system/default/transforms.conf SOURCE_KEY = _raw /opt/splunk/etc/system/default/transforms.conf WRITE_META = False /opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf case_sensitive_match = false /opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf filename = xmlsecurity_eventcode_errorcode_action.csv

I then wanted to extract the fields, for example "SOURCE_KEY = _raw" should be my key/value pair!

I hoped to accomplish this with

(transforms)

[dotheparsething] REGEX = \s([\S-]+)\s=\s([^\/\n]+) LOOKAHEAD = 100000 FORMAT = $1::$2 REPEAT_MATCH = true

(props)

[(?::){0}splunk:config:btool:*] TRUNCATE=10000 MAX_EVENTS=10000 KV_MODE = none BREAK_ONLY_BEFORE = conf[\s]+\[ #SEDCMD-removespaces = s/\ +/\ /g REPORT-dotheparsething = dotheparsething

But I am getting nothing!

Regex101 seems happy with my search.

Re: How to extract key/value with dynamic key name?

richgalloway — Wed, 26 Oct 2022 20:59:03 GMT

Did you know btool-equivalent data is available via REST? Check out

| rest /services/configs/conf-<<conf>>

where <<conf>> is a config file name (props, transforms, etc.).

When you say you are getting nothing, what exactly does that mean? How are you looking for the something? Recall the REPORT settings extract fields at search time as opposed to TRANSFORMS settings which apply at index time.

What sourcetype is specified in your inputs.conf?

Re: How to extract key/value with dynamic key name?

oliverja — Thu, 27 Oct 2022 05:42:52 GMT

Wow. I had no idea that was a thing! It never crossed my mind.

It wont work in this situation, as I use it to keep track of forwarders and stuff rest wouldn't have access to, but that is fascinating. Might be worth writing my own script to push out instead of the btool app...

Here is one of my input stanzas:

[script:///opt/splunk/etc/apps/btool/bin/btool.sh transforms /opt/splunk] _rcvbuf = 1572864 disabled = 0 host = cn02-k index = splunk_insights interval = 43200 sourcetype = splunk:config:btool:transforms

And I know my existing props works, because the rest of my stanza has been working:

[(?::){0}splunk:config:btool:*] TRUNCATE=10000 MAX_EVENTS=10000 KV_MODE = none BREAK_ONLY_BEFORE = conf[\s]+\[ REPORT-dotheparsething = dotheparsething <-----except this critter.

By "getting nothing", I mean that I was expecting my regex to go through and convert the output in the first post to key/value pairs when I search. I have this running on a single instance of splunk, so there's no search head/indexer confusions. So, given:

I am hoping to get

Match - $1 CAN_OPTIMIZE Match - $2 TRUE Match - $1 CLEAN_KEYS Match - $2 TRUE Match - $1 DEPTH_LIMIT Match - $2 1000 Match - $1 KEEP_EMPTY_VALS Match - $2 FALSE Key=Value --> $1::$2 CAN_OPTIMIZE = TRUE CLEAN_KEYS = TRUE DEPTH_LIMIT = 1000 KEEP_EMPTY_VALS = FALSE

Re: How to extract key/value with dynamic key name?

richgalloway — Thu, 27 Oct 2022 15:15:14 GMT

I was able to get it to work by removing the regex from the stanza name in props.conf.

[splunk:config:btool:*] TRUNCATE=10000 MAX_EVENTS=10000 KV_MODE = none BREAK_ONLY_BEFORE = conf[\s]+\[ REPORT-dotheparsething = dotheparsething DATETIME_CONFIG = current

Re: How to extract key/value with dynamic key name?

oliverja — Fri, 28 Oct 2022 06:31:58 GMT

No, changing the regex just kills the whole stanza, because there's no wildcards in sourcetypes.

When my "break only before" is removed, each line is a new event, and splunk auto extracts the single fields.

I want the exact same extractions, but with the whole stanza as a single event

Re: How to extract key/value with dynamic key name?

richgalloway — Fri, 28 Oct 2022 12:45:38 GMT

Perhaps it's a version thing, but it works in 9.0.0.1. Sourcetype splunk:config:btool:transforms was parsed from stanza [splunk:config:btool:*].