How to filter the data in my log before indexing?

[Tejkumar451]

HI I have a scenario where I need to filter the data present in a log. I need to index only the last line from that log into splunk & ignore the rest of the lines. I am using a universal forwarder here.

Example file: abc 
---------------------------------------------------------------
abcdefgh
bda
cdb
---
---
---
Splunk000012345
--------------------------------------------------------------

So, I have to ingest only the last line "Splunk000012345" into Splunk ignore the above lines. And just an fyi, I am even fine if I can ingest only the number 000012345 too.

I did the below changes on the forwarder side, but it didn't work.

Edit props.conf and add the following:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

Edit transforms.conf and add the following:

[setnull]
REGEX = .
DEST_KEY = queue

[setparsing]
REGEX = Splunk(?[^\w])
DEST_KEY = queue
FORMAT = indexQueue

[Tejkumar451]

Forgot to add one thing, I did all this on the forwarder side & all the lines are getting ingested which shouldn't be the case.

[DalJeanis]

You can test regex's out over at regex.com and see what they do or don't pull. Here's your problem -

Splunk(?[^\w])

The above regex translates as the letters "Splunk" followed by a single character that is NOT a "word" character. That is, the first character after Splunk must be neither a digit nor an upper or lower case letter. Might exclude underscores or hyphens or something as well, I'd have to look it up. In any case, your zero will fail to match "not being a word character" so the line will not be ingested.

Try this -

Splunk(?\d+)

[Tejkumar451]

Thanks for the response, I have tested this regex Splunk(\d+) in regex101.com and its working fine over there. So, now my concern is should these props & Transforms.conf be done in the Universal Forwarder end? or towards the indexer side? As I am not exactly sure if parsing can be done on the Universal forwarder side.

Thanks