Getting Data In

Assistance in Nulling Events before Indexing

willadams
Contributor

I have the following props and confs which works fine and does what I need it to do.

 

PROPS

 

 

[mydata_logs]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
TRANSFORMS-set = setnull,keptevents

 

 

 

TRANSFORMS

 

 

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keptevents]
REGEX = ^.+INFO:
DEST_KEY = queue
FORMAT = myindex

 

 

 

This leaves me with events in my log such as follows

 

2020-10-02 17:01:32,360 INFO:
User.val (value, value2, value3, value4): User not found. Parameters: myid: 1; orig: userKO: userId: 1234567

2020-10-02 17:09:48,123 INFO:

Helper.loadObjects(): Username does not exist. mystique

2020-10-02 18:01:48,546 INFO:

CleanupProcess.executeHelper(): Running cleanup process for Silly 1.2.3.4000 ...

 

What I am trying to do from the left over logs is to remove the lines that are system events.  In the above example I want to remove the events that read

2020-10-02 17:09:48,123 INFO:

Helper.loadObjects(): Username does not exist. mystique

 

2020-10-02 18:01:48,546 INFO:

CleanupProcess.executeHelper(): Running cleanup process for Silly 1.2.3.4000 ...

 

This should leave me with the following event making it to my index

2020-10-02 17:01:32,360 INFO:
User.val (value, value2, value3, value4): User not found. Parameters: myid: 1; orig: userKO: userId: 1234567

 

Through REGEX (and using the CleanupProcess.executeHelper as an example) - ^CleanupProcess.+ would target that line so I could regex it out but I need help on how I construct the props / transforms to do this.  This is the props and transforms adjusted (I tried changing order so that "keptevents" was directly after the first null (i.e. setnull) or at the end (ensuring that props reflected the order).

UPDATED PROPS

 

 

[mydata_logs]
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
TRANSFORMS-set = setnull,keptevents,cleanupprocess_filter,helper_filter

 

 

 

UPDATED TRANSFORMS

 

 

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[cleanupprocess_filter]
REGEX = ^CleanupProcess.+
DEST_KEY = queue
FORMAT = nullQueue

[helper_filter]
REGEX = ^Helper.+
DEST_KEY = queue
FORMAT = nullQueue

[keptevents]
REGEX = ^.+INFO:
DEST_KEY = queue
FORMAT = myindex

 

 

 

The above is an example as there are more filters I need to apply as I work through my data set.  Unfortunately I have no way on the syslog instance to isolate these logs at the source.  

 

Thanks in advance

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Unlock Database Monitoring with Splunk Observability Cloud

  In today’s fast-paced digital landscape, even minor database slowdowns can disrupt user experiences and ...

Purpose in Action: How Splunk Is Helping Power an Inclusive Future for All

At Cisco, purpose isn’t a tagline—it’s a commitment. Cisco’s FY25 Purpose Report outlines how the company is ...

[Upcoming Webinar] Demo Day: Transforming IT Operations with Splunk

Join us for a live Demo Day at the Cisco Store on January 21st 10:00am - 11:00am PST In the fast-paced world ...