topic Re: How to filter the index by using mulitple stanzas with different sources in props and transforms in Getting Data In

How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Mon, 28 Sep 2020 13:47:53 GMT

Hi,
I would like to filter out some event logs coming from different forwarders.
I have been able to filter out some specifics winevents coming from security log.
Now I would like to optimize my indexing and filter other events from other sources like application events.
It is not working when I am adding a new stanza in transforms.conf. Only winevent security is filtered out.

Would you please have an idea ?
Here is below my files:

--------props-----

[WinEventLog:Security]
TRANSFORMS-Security=setnull,setparsing

[WinEventLog:Application]
TRANSFORMS-Application=setnull2,setparsing2

-----------transforms-----------
[setnull]
REGEX = (?m).
DEST_KEY = queue
FORMAT = nullQueue

[setnull2]
REGEX = (?m)^Database_copy=(MailDB_14|MailDB_04)
DEST_KEY = queue
FORMAT = nullqueue

[setparsing]
REGEX = (?m)^EventCode=(5157|4625|4624|7036|1102|1033)
DEST_KEY = queue
FORMAT = indexQueue

[setparsing2]
REGEX = (?m).
DEST_KEY = queue
FORMAT = indexQueue

Thank you
Joffrey

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

kristian_kolb — Mon, 29 Apr 2013 08:51:25 GMT

The reason is that all events will pass through both transforms.

So, first an event will go through setnull2 and some will be changed to nullQueue, but it will also be going through the next transform, setparsing2 which promptly restores its destination to indexQueue.

That's why the order is most important.

Probably you should just remove the call for the setparsing2 transform in props.conf stanza.

http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

UPDATE:

So that was it all along. The regex will work on the actual text in the event, not on the (search-time) parsed fields. Thus, your regex in setnull2 should be;

REGEX = (?m)Database\scopy:\s+(MailDB_04|MailDB_14)

Hope this helps,

Kristian Kolb

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

Ayn — Mon, 29 Apr 2013 09:29:49 GMT

Also setnull2 has a typo - "nullqueue" instead of "nullQueue".

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

kristian_kolb — Mon, 29 Apr 2013 09:42:57 GMT

good spotting

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Mon, 29 Apr 2013 14:27:25 GMT

Thank you for your help. Unfortunately, none of your answers are solving my issue. THe order is important, that's why I first filtered the events to nullQueue in my transforms.

I still tried to remove setparsing2 but no success. Good spotting indeed on the Q in capital but unfortunately, that did not change anything.

Thanks anyway Any other ideas ?

Joffrey

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Mon, 28 Sep 2020 13:48:06 GMT

I am still having the event Database_copy=(MailDB_14|MailDB_04) in my index.
Thank you
Joffrey

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

kristian_kolb — Mon, 29 Apr 2013 16:38:55 GMT

I think you're having regex issues. You do realize that any event that matches the regex in the second transform ((?m). and setparsing2, respectively in your case) will be indexed?

So if you have a REGEX = . (or something to that effect) in the second transform, ALL events will match.

So either you have both transforms, which will behave as described above, OR the setnull2 regex is NOT matching the events you want to filter out.

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Tue, 30 Apr 2013 09:22:44 GMT

Thank you kristian. You oriented me on the good way as actually my regex is not working.
I removed all the stanzas and kept only setnull2. My database_copy events are still not redirected to the queue.

I dont know what am I doing wrong..

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Tue, 30 Apr 2013 09:23:38 GMT

I meant, still NOT redirected to the nullqueue sorry

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Tue, 30 Apr 2013 09:34:59 GMT

I have seen that the item "Database_copy" is a categorical value. Can it explain this issue ? Can I filter any type of values?

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

kristian_kolb — Tue, 30 Apr 2013 09:53:39 GMT

'Categorical value', what is that?

One thing I would try in your case is to remove the caret (i.e. start of line) before the Database_copy..

Also, perhaps more important, is that you need to ensure that the regex matches the actual string in the event, and NOT a sanitized Splunk field, e.g. Account Name in the log, will be called Account_Name as a Splunk field. Could that be what you referred to as a categorical value?

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Tue, 30 Apr 2013 12:10:27 GMT

I meant that the "database_copy" field value is "categorical". Just as "eventcode" value is numeric. But maybe it's not related to my issue at all.

I have paid attention to the field name in splunk and it's named this way: database_copy.

I removed the caret but the events are still indexed. I am surprised because it works well with eventcode field.

Thank you kristian for your help

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

Ayn — Tue, 30 Apr 2013 12:50:50 GMT

Note that the regexes here are applied to the raw event data, thus whatever fields Splunk makes of it has nothing to do with this. You should post some sample data so we can help you find a valid filter string - I suspect you've got some concepts mixed up here...

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

kristian_kolb — Tue, 30 Apr 2013 13:19:17 GMT

Joffrey. As I said before, and as Ayn suggests, I suspect that you're trying to use a field name called Database_copy in your regex, but in the raw event text, it says Database copy.

That is a world of difference. Please post an actual event.

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Mon, 28 Sep 2020 13:48:30 GMT

Ok here is an event that i want to filter out.
Thank you both.

04/30/2013 03:16:38 PM
LogName=Application
SourceName=MSExchangeRepl
EventCode=4113
EventType=2
Type=Error
ComputerName=GCHEEX02
TaskCategory=Service
OpCode=None
RecordNumber=2127847
Keywords=Classic
Message=Database redundancy health check failed.
Database copy: MailDB_14
Redundancy count: 1

Error: The number of configured copies for database 'MailDB_14' (1) is less than the required redundancy count (2).

Name Status RealCopyQueu InspectorQue ReplayQueue CIState
e ue

MailDB_14\GCH Mounted 0 0 0 Healthy
EEX02

===============

Full Status

Identity : MailDB_14\GCHEEX02
Name : MailDB_14\GCHEEX02
DatabaseName : MailDB_14
Status : Mounted
MailboxServer : GCHEEX02
ActiveDatabaseCopy : gcheex02
ActivationSuspended : False
ActionInitiator : Unknown
ErrorMessage :
ErrorEventId :
ExtendedErrorInfo :
SuspendComment :
SinglePageRestore : 0
ContentIndexState : Healthy
ContentIndexErrorMessage :
CopyQueueLength : 0
ReplayQueueLength : 0
LatestAvailableLogTime :
LastCopyNotificationedLogTime :
LastCopiedLogTime :
LastInspectedLogTime :
LastReplayedLogTime :
LastLogGenerated : 0
LastLogCopyNotified : 0
LastLogCopied : 0
LastLogInspected : 0
LastLogReplayed : 0
LogsReplayedSinceInstanceStart : 0
LogsCopiedSinceInstanceStart : 0
LatestFullBackupTime :
LatestIncrementalBackupTime :
LatestDifferentialBackupTime :
LatestCopyBackupTime :
SnapshotBackup :
SnapshotLatestFullBackup :
SnapshotLatestIncrementalBackup :
SnapshotLatestDifferentialBackup :
SnapshotLatestCopyBackup :
LogReplayQueueIncreasing : False
LogCopyQueueIncreasing : False
OutstandingDumpsterRequests : {}
OutgoingConnections :
IncomingLogCopyingNetwork :
SeedingNetwork :
ActiveCopy : True

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Tue, 30 Apr 2013 15:27:14 GMT

I have posted an event in a new answer. I was limited in characters number.
Thanks again
Jo

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

kristian_kolb — Tue, 30 Apr 2013 19:39:18 GMT

see comment in my edited answer above.

Re: How to filter the index by using mulitple stanzas with different sources in props and transforms

joffrey_braban — Wed, 01 May 2013 09:49:26 GMT

Thanks a million Kristian. I works perfectly. I had just to remove the stanza setparsing2.
Thank you again
Joffrey