Getting Data In

Assistance in Nulling Events before Indexing

willadams
Contributor

I have the following props and confs which works fine and does what I need it to do.

 

PROPS

 

 

[mydata_logs]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
TRANSFORMS-set = setnull,keptevents

 

 

 

TRANSFORMS

 

 

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keptevents]
REGEX = ^.+INFO:
DEST_KEY = queue
FORMAT = myindex

 

 

 

This leaves me with events in my log such as follows

 

2020-10-02 17:01:32,360 INFO:
User.val (value, value2, value3, value4): User not found. Parameters: myid: 1; orig: userKO: userId: 1234567

2020-10-02 17:09:48,123 INFO:

Helper.loadObjects(): Username does not exist. mystique

2020-10-02 18:01:48,546 INFO:

CleanupProcess.executeHelper(): Running cleanup process for Silly 1.2.3.4000 ...

 

What I am trying to do from the left over logs is to remove the lines that are system events.  In the above example I want to remove the events that read

2020-10-02 17:09:48,123 INFO:

Helper.loadObjects(): Username does not exist. mystique

 

2020-10-02 18:01:48,546 INFO:

CleanupProcess.executeHelper(): Running cleanup process for Silly 1.2.3.4000 ...

 

This should leave me with the following event making it to my index

2020-10-02 17:01:32,360 INFO:
User.val (value, value2, value3, value4): User not found. Parameters: myid: 1; orig: userKO: userId: 1234567

 

Through REGEX (and using the CleanupProcess.executeHelper as an example) - ^CleanupProcess.+ would target that line so I could regex it out but I need help on how I construct the props / transforms to do this.  This is the props and transforms adjusted (I tried changing order so that "keptevents" was directly after the first null (i.e. setnull) or at the end (ensuring that props reflected the order).

UPDATED PROPS

 

 

[mydata_logs]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
TRANSFORMS-set = setnull,keptevents,cleanupprocess_filter,helper_filter

 

 

 

UPDATED TRANSFORMS

 

 

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[cleanupprocess_filter]
REGEX = ^CleanupProcess.+
DEST_KEY = queue
FORMAT = nullQueue

[helper_filter]
REGEX = ^Helper.+
DEST_KEY = queue
FORMAT = nullQueue

[keptevents]
REGEX = ^.+INFO:
DEST_KEY = queue
FORMAT = myindex

 

 

 

The above is an example as there are more filters I need to apply as I work through my data set.  Unfortunately I have no way on the syslog instance to isolate these logs at the source.  

 

Thanks in advance

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...