Filtration of Logs Before Ingestion For Few Sourcetypes

[anandhalagaras1]

Hi All,

I want to filter out few of the lines from the events for different sourcetypes but for the same index.So that i can save some licenses.

index=abc and Where "x" denotes numbers.

Case 1:
From sourecetype=def I want to filter out the lines from the event if it comes in sequence like this.
SourceType = def

(xx:xx:xxx): Version: x.x.x.x, Inside thread x. . MessageQueueException Timeout for the requested operation has expired.
(xx:xx:xxx): Version: x.x.x.x, Inside thread x. . Timeout for the requested operation has expired.

--------------------------------------------------------------------------------------------------------------------------------------------------------
Case 2:

Similarly for sourecetype=ghi I want to filter out the lines from the event if it comes in sequence like this.
SourceType = ghi

(xx:xx:xxx): Version: x.x.xx.x, Thread x,CmdID na,Timeout for the requested operation has expired.
(xx:xx:xxx): Version: x.x.xx.x, Thread x,CmdID na,Finished execution.
----------------------------------------------------------------------------------------------------------------------------------------------------------
Case 3:

Similarly for sourecetype=jkl I want to filter out the lines from the event if it comes in sequence like this.
SourceType = jkl

12/08/2020-12:00:00.2246074| Version: x.x.x.xxxxx| Information: exception type: System.Exception| message: System.Exception: Testingmaterialin::TestinExecutionThread() - Running - Begin| thread: 8
12/08/2020-12:00:01.2896317| Version: x.x.x.xxxxx| Information: exception type: System.Exception| message: System.Exception: Testingmaterialin::TestinExecutionThread() - Message queue has no messages, will try again.| thread: 8
12/08/2020-12:00:01.2896317| Version: x.x.x.xxxxx| Information: exception type: System.Exception| message: System.Exception: Testingmaterialin::TestinExecutionThread() - Running - End| thread: 8

So kindly help with the props and transforms so that I can filter those logs before ingestion.Thanks.

[scelikok]

Hi @anandhalagaras1,

You can filter entire event that contains something with regex , please see below sample for your Case 1;

In props.conf, set the TRANSFORMS-null attribute:
[source::/var/log/messages]
TRANSFORMS-null= setnull_case1

Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":
[setnull_case1]
REGEX = Timeout\for\sthe\srequested\soperation\shas\sexpired\.
DEST_KEY = queue
FORMAT = nullQueue

Please check below document;

https://docs.splunk.com/Documentation/Splunk/8.1.2/Forwarding/Routeandfilterdatad#Discard_specific_e... 

[scelikok]

Hi @anandhalagaras1,

I think you want to filter events that exists in logs in a sequential. Since Splunk process every event as separate, this kind of filtering is not supported. You can filter some events with regex matching, but this regex should match only that particular event.

https://docs.splunk.com/Documentation/Splunk/8.1.2/Forwarding/Routeandfilterdatad#Filter_event_data_... 

[anandhalagaras1]

@scelikok ,

Thank you for your prompt response.

So you meant to say that we cant filter out those lines from the event if it is sequential.

So is it possible to filter out the entire line from the event if it contains the keyword something like this Or if exactly the version details and everything is matched then can we filter out those lines from the events.

Case 1:
MessageQueueException Timeout for the requested operation has expired.
Timeout for the requested operation has expired.

----------------------------------------------------------------------------------------------------------------------------------------------------------

Case 2:
Thread 1,CmdID na,Timeout for the requested operation has expired.
Thread 1,CmdID na,Finished execution.

----------------------------------------------------------------------------------------------------------------------------------------------------------

Case 3:
Information: exception type: System.Exception| message: System.Exception: Testingmaterialin::TestinExecutionThread() - Running - Begin| thread: 8
Information: exception type: System.Exception| message: System.Exception: Testingmaterialin::TestinExecutionThread() - Message queue has no messages, will try again.| thread: 8
Information: exception type: System.Exception| message: System.Exception: Testingmaterialin::TestinExecutionThread() - Running - End| thread: 8

===========================================================================================================================================================
OR
============================================================================================================================================================
Case 1:
From sourecetype=def I want to filter out the lines from the event if it comes in sequence like this.
SourceType = def

(12:34:567): Version: 1.2.3.4, Inside thread 5. . MessageQueueException Timeout for the requested operation has expired.
(12:34:567): Version: 1.2.3.4, Inside thread 5. . Timeout for the requested operation has expired.

--------------------------------------------------------------------------------------------------------------------------------------------------------
Case 2:

Similarly for sourecetype=ghi I want to filter out the lines from the event if it comes in sequence like this.
SourceType = ghi

(89:75:432): Version: 5.4.32.1, Thread 1,CmdID na,Timeout for the requested operation has expired.
(89:75:432): Version: 5.4.32.1, Thread 1,CmdID na,Finished execution.
----------------------------------------------------------------------------------------------------------------------------------------------------------
Case 3:

Similarly for sourecetype=jkl I want to filter out the lines from the event if it comes in sequence like this.
SourceType = jkl

12/08/2020-12:00:00.2246074| Version: 1.2.3.65433| Information: exception type: System.Exception| message: System.Exception: Testingmaterialin::TestinExecutionThread() - Running - Begin| thread: 8
12/08/2020-12:00:01.2896317| Version: 1.2.3.65433| Information: exception type: System.Exception| message: System.Exception: Testingmaterialin::TestinExecutionThread() - Message queue has no messages, will try again.| thread: 8
12/08/2020-12:00:01.2896317| Version: 1.2.3.65433| Information: exception type: System.Exception| message: System.Except

Also i just surfed the internet and i came to see that we can able to filter out the lines from an event if we have in wineventlog ( Under Saving License section)

https://hurricanelabs.com/splunk-tutorials/leveraging-windows-event-log-filtering-and-design-techniq...

So kindly help on the request.

[anandhalagaras1]

Can anyone help on my request.