Getting Data In

Assistance in Nulling Events before Indexing

willadams
Contributor

I have the following props and confs which works fine and does what I need it to do.

 

PROPS

 

 

[mydata_logs]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
TRANSFORMS-set = setnull,keptevents

 

 

 

TRANSFORMS

 

 

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keptevents]
REGEX = ^.+INFO:
DEST_KEY = queue
FORMAT = myindex

 

 

 

This leaves me with events in my log such as follows

 

2020-10-02 17:01:32,360 INFO:
User.val (value, value2, value3, value4): User not found. Parameters: myid: 1; orig: userKO: userId: 1234567

2020-10-02 17:09:48,123 INFO:

Helper.loadObjects(): Username does not exist. mystique

2020-10-02 18:01:48,546 INFO:

CleanupProcess.executeHelper(): Running cleanup process for Silly 1.2.3.4000 ...

 

What I am trying to do from the left over logs is to remove the lines that are system events.  In the above example I want to remove the events that read

2020-10-02 17:09:48,123 INFO:

Helper.loadObjects(): Username does not exist. mystique

 

2020-10-02 18:01:48,546 INFO:

CleanupProcess.executeHelper(): Running cleanup process for Silly 1.2.3.4000 ...

 

This should leave me with the following event making it to my index

2020-10-02 17:01:32,360 INFO:
User.val (value, value2, value3, value4): User not found. Parameters: myid: 1; orig: userKO: userId: 1234567

 

Through REGEX (and using the CleanupProcess.executeHelper as an example) - ^CleanupProcess.+ would target that line so I could regex it out but I need help on how I construct the props / transforms to do this.  This is the props and transforms adjusted (I tried changing order so that "keptevents" was directly after the first null (i.e. setnull) or at the end (ensuring that props reflected the order).

UPDATED PROPS

 

 

[mydata_logs]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
TRANSFORMS-set = setnull,keptevents,cleanupprocess_filter,helper_filter

 

 

 

UPDATED TRANSFORMS

 

 

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[cleanupprocess_filter]
REGEX = ^CleanupProcess.+
DEST_KEY = queue
FORMAT = nullQueue

[helper_filter]
REGEX = ^Helper.+
DEST_KEY = queue
FORMAT = nullQueue

[keptevents]
REGEX = ^.+INFO:
DEST_KEY = queue
FORMAT = myindex

 

 

 

The above is an example as there are more filters I need to apply as I work through my data set.  Unfortunately I have no way on the syslog instance to isolate these logs at the source.  

 

Thanks in advance

Labels (2)
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Mile High Learning with Splunk University, Denver, Colorado

If Denver is known for its mile-high elevation, Splunk University is about to raise the bar on technical ...

IT Service Intelligence 5.0 Series: Your Guide to the June Launch

We are excited to announce the June release of Splunk IT Service Intelligence (ITSI) 5.0. This update ...

Agent Mode Engaged! Enchaining Agentic Operations with Splunk AI Assistant 2.0

    Are you ready to transform how your team handles complex data requests? We invite you to our upcoming ...