How to extract key/value with dynamic key name?

[oliverja]

I found this, but I am unable to replicate it.

I am not understanding where I am messing up here.

Problem:

I feed btool outputs into splunk, and chop them up by stanza.

/opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf                               [xmlsecurity_eventcode_errorcode_action_lookup]
/opt/splunk/etc/system/default/transforms.conf                                               CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf                                               CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf                                               DEFAULT_VALUE = 
/opt/splunk/etc/system/default/transforms.conf                                               DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/transforms.conf                                               DEST_KEY = 
/opt/splunk/etc/system/default/transforms.conf                                               FORMAT = 
/opt/splunk/etc/system/default/transforms.conf                                               KEEP_EMPTY_VALS = False
/opt/splunk/etc/system/default/transforms.conf                                               LOOKAHEAD = 4096
/opt/splunk/etc/system/default/transforms.conf                                               MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/transforms.conf                                               MV_ADD = False
/opt/splunk/etc/system/default/transforms.conf                                               REGEX = 
/opt/splunk/etc/system/default/transforms.conf                                               SOURCE_KEY = _raw
/opt/splunk/etc/system/default/transforms.conf                                               WRITE_META = False
/opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf                               case_sensitive_match = false
/opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf                               filename = xmlsecurity_eventcode_errorcode_action.csv

I then wanted to extract the fields, for example "SOURCE_KEY = _raw" should be my key/value pair!

I hoped to accomplish this with 

(transforms)

[dotheparsething]
REGEX = \s([\S-]+)\s=\s([^\/\n]+)
LOOKAHEAD = 100000
FORMAT = $1::$2
REPEAT_MATCH = true

(props)

[(?::){0}splunk:config:btool:*]
TRUNCATE=10000
MAX_EVENTS=10000
KV_MODE = none
BREAK_ONLY_BEFORE = conf[\s]+\[
#SEDCMD-removespaces = s/\ +/\ /g
REPORT-dotheparsething = dotheparsething

 But I am getting nothing!

Regex101 seems happy with my search.

[richgalloway]

Did you know btool-equivalent data is available via REST?  Check out

| rest /services/configs/conf-<<conf>>

where <<conf>> is a config file name (props, transforms, etc.).

When you say you are getting nothing, what exactly does that mean?  How are you looking for the something?  Recall the REPORT settings extract fields at search time as opposed to TRANSFORMS settings which apply at index time.

What sourcetype is specified in your inputs.conf?

[oliverja]

Wow. I had no idea that was a thing! It never crossed my mind.

It wont work in this situation, as I use it to keep track of forwarders and stuff rest wouldn't have access to, but that is fascinating. Might be worth writing my own script to push out instead of the btool app...  

Here is one of my input stanzas:

[script:///opt/splunk/etc/apps/btool/bin/btool.sh transforms /opt/splunk]                           _rcvbuf = 1572864                          
disabled = 0
host = cn02-k
index = splunk_insights
interval = 43200
sourcetype = splunk:config:btool:transforms

And I know my existing props works, because the rest of my stanza has been working:

[(?::){0}splunk:config:btool:*]
TRUNCATE=10000
MAX_EVENTS=10000
KV_MODE = none
BREAK_ONLY_BEFORE = conf[\s]+\[
REPORT-dotheparsething = dotheparsething    <-----except this critter.

By "getting nothing", I mean that I was expecting my regex to go through and convert the output in the first post to key/value pairs when I search. I have this running on a single instance of splunk, so there's no search head/indexer confusions. So, given:

/opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf                               [xmlsecurity_eventcode_errorcode_action_lookup]
/opt/splunk/etc/system/default/transforms.conf                                               CAN_OPTIMIZE = True
/opt/splunk/etc/system/default/transforms.conf                                               CLEAN_KEYS = True
/opt/splunk/etc/system/default/transforms.conf                                               DEFAULT_VALUE = 
/opt/splunk/etc/system/default/transforms.conf                                               DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/transforms.conf                                               DEST_KEY = 
/opt/splunk/etc/system/default/transforms.conf                                               FORMAT = 
/opt/splunk/etc/system/default/transforms.conf                                               KEEP_EMPTY_VALS = False

I am hoping to get  

Match - $1	CAN_OPTIMIZE	
Match - $2	TRUE	
Match - $1	CLEAN_KEYS	
Match - $2	TRUE	
Match - $1	DEPTH_LIMIT	
Match - $2	1000	
Match - $1	KEEP_EMPTY_VALS	
Match - $2	FALSE	
		
Key=Value -->  $1::$2
		
CAN_OPTIMIZE	 =	TRUE
CLEAN_KEYS	 =	TRUE
DEPTH_LIMIT	 =	1000
KEEP_EMPTY_VALS	 =	FALSE

[richgalloway]

I was able to get it to work by removing the regex from the stanza name in props.conf.

[splunk:config:btool:*]
TRUNCATE=10000
MAX_EVENTS=10000
KV_MODE = none
BREAK_ONLY_BEFORE = conf[\s]+\[
REPORT-dotheparsething = dotheparsething
DATETIME_CONFIG = current

[oliverja]

No, changing the regex just kills the whole stanza, because there's no wildcards in sourcetypes. 

When my "break only before" is removed, each line is a new event, and splunk auto extracts the single fields.

I want the exact same extractions, but with the whole stanza as a single event

[richgalloway]

Perhaps it's a version thing, but it works in 9.0.0.1.  Sourcetype splunk:config:btool:transforms was parsed from stanza [splunk:config:btool:*].