Splunk Enterprise

How to create props and transforms?

scottsavareseat
Path Finder

I have a splunk indexer cluster with a single search head. I'm taking data in via HEC directly to the cluster. The events themselves are not JSON and look like this:

 

host_name audit[86]: key1=val1, key2=val2, key3=val3, key4=[ subkey1: subval1 subkey2: subval2 subkey3: sub val3 subkey4: subval4]

 

Note that the value of "subkey3" does have a space. It is intentional.

Splunk, by default will grab all the = deliminated fields... so key1,2,3,4 are grabbed nicely. BUT, it won't grab the subkeys as they are : deliminated fields and spaces are apparently allowed.

I have a regex that will parse them for me:

 

(?<key>\w+):\s+(?<value>.*?)(?=\s+\w+:|]$)

 

It leverages some look ahead for the next field. I tried putting this in a props/transforms as such:

 

props.conf:
[my_sourcetype]
REPORT-key4 = parse_key4
transforms.conf:
[parse_key4]
REGEX = (\w+):\s+(.*?)(?=\s+\w+:|]$)
FORMAT = $1::$2
REPEAT_MATCH = true

 

I deploy both files in their own app to the cluster master (and then apply cluster bundle) and the search head (and either use the debug/refresh or restart splunk). But, it is not extracting the fields.

Any ideas on why it isn't doing the extraction? Note... the goal is that this should be a search time extraction. I don't need/want it being index time.

0 Karma
1 Solution

scottsavareseat
Path Finder

So, I was able to get this to work finally...I was missing an app.conf (created a blank one) and a local.meta permissions file in my app.

Some notes for troubleshooting:

  • I added an "EVAL-test" line to my props.conf. Turns out, the app is ignored on the indexers completely so I don't need it there as the line was never read.
  • Using the EVAL line on the search head let me know that the props.conf wasn't being read by Splunk and wasn't being sent to the indexers. Adding the local.meta and app.conf files allowed the app to be consumed by the search head. Then I could put my transforms back in and it worked.

 

View solution in original post

0 Karma

scottsavareseat
Path Finder

I would like to understand why putting it on the indexer cluster didn't work. I know that the search bundles being sent to indexers can get quite large so having it in the indexer cluster config would be nice.

0 Karma

scottsavareseat
Path Finder

So, I was able to get this to work finally...I was missing an app.conf (created a blank one) and a local.meta permissions file in my app.

Some notes for troubleshooting:

  • I added an "EVAL-test" line to my props.conf. Turns out, the app is ignored on the indexers completely so I don't need it there as the line was never read.
  • Using the EVAL line on the search head let me know that the props.conf wasn't being read by Splunk and wasn't being sent to the indexers. Adding the local.meta and app.conf files allowed the app to be consumed by the search head. Then I could put my transforms back in and it worked.

 

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...