Assistance in Nulling Events before Indexing

willadams — Tue, 20 Oct 2020 10:37:12 GMT

I have the following props and confs which works fine and does what I need it to do.

PROPS

[mydata_logs] LINE_BREAKER = ([\r\n]+) SHOULD_LINEMERGE = true NO_BINARY_CHECK = true TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 20 TRANSFORMS-set = setnull,keptevents

TRANSFORMS

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [keptevents] REGEX = ^.+INFO: DEST_KEY = queue FORMAT = myindex

This leaves me with events in my log such as follows

2020-10-02 17:01:32,360 INFO:
User.val (value, value2, value3, value4): User not found. Parameters: myid: 1; orig: userKO: userId: 1234567

2020-10-02 17:09:48,123 INFO:

Helper.loadObjects(): Username does not exist. mystique

2020-10-02 18:01:48,546 INFO:

CleanupProcess.executeHelper(): Running cleanup process for Silly 1.2.3.4000 ...

What I am trying to do from the left over logs is to remove the lines that are system events. In the above example I want to remove the events that read

2020-10-02 17:09:48,123 INFO:

Helper.loadObjects(): Username does not exist. mystique

2020-10-02 18:01:48,546 INFO:

CleanupProcess.executeHelper(): Running cleanup process for Silly 1.2.3.4000 ...

This should leave me with the following event making it to my index

2020-10-02 17:01:32,360 INFO:
User.val (value, value2, value3, value4): User not found. Parameters: myid: 1; orig: userKO: userId: 1234567

Through REGEX (and using the CleanupProcess.executeHelper as an example) - ^CleanupProcess.+ would target that line so I could regex it out but I need help on how I construct the props / transforms to do this. This is the props and transforms adjusted (I tried changing order so that "keptevents" was directly after the first null (i.e. setnull) or at the end (ensuring that props reflected the order).

UPDATED PROPS

[mydata_logs] LINE_BREAKER = ([\r\n]+) SHOULD_LINEMERGE = true NO_BINARY_CHECK = true TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 20 TRANSFORMS-set = setnull,keptevents,cleanupprocess_filter,helper_filter

UPDATED TRANSFORMS

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [cleanupprocess_filter] REGEX = ^CleanupProcess.+ DEST_KEY = queue FORMAT = nullQueue [helper_filter] REGEX = ^Helper.+ DEST_KEY = queue FORMAT = nullQueue [keptevents] REGEX = ^.+INFO: DEST_KEY = queue FORMAT = myindex

The above is an example as there are more filters I need to apply as I work through my data set. Unfortunately I have no way on the syslog instance to isolate these logs at the source.

Thanks in advance

topic Assistance in Nulling Events before Indexing in Getting Data In

Assistance in Nulling Events before Indexing