Getting Data In

I have a log file. I want to ignore lines from that log file before indexing it. can I do this? Please help me on this matter as soon as possible

saibal6
Path Finder

In the log file I have below mentioned a line :

EVENT_SESH;0;04/01/2018 06:30:23:5000;1;;1;0;;;END OF IMPORT PROCESS FOR THE MASTER STORE - PENDING_TXT(0).

I want to ignore all the lines after above mentioned line before indexing.

Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi saibal6,
let me understand:

  • you have files with many logs to take,
  • when there's the above event, do you want to discard alla events until the end of the file,

is it correct?

In this case, you could use in props.conf

SEDCMD-delete = s/(?ms)EVENT_SESH.*//g

Or a pre-parsing script to run before indexing

If instead you want to discard only the event with EVENT_SESH, you can follow http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad

Edit props.conf and add the following:

[your_sourcetype]
TRANSFORMS-set= setparsing,setnull

Edit transforms.conf and add the following:

[setnull]
REGEX = EVENT_SESH
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi saibal6,
let me understand:

  • you have files with many logs to take,
  • when there's the above event, do you want to discard alla events until the end of the file,

is it correct?

In this case, you could use in props.conf

SEDCMD-delete = s/(?ms)EVENT_SESH.*//g

Or a pre-parsing script to run before indexing

If instead you want to discard only the event with EVENT_SESH, you can follow http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad

Edit props.conf and add the following:

[your_sourcetype]
TRANSFORMS-set= setparsing,setnull

Edit transforms.conf and add the following:

[setnull]
REGEX = EVENT_SESH
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

0 Karma

saibal6
Path Finder

Thanks Giuseppe for the answer.

In our case your first suggestion is useful.
we added the line in " SEDCMD-delete = s/(?ms)EVENT_SESH.*//g" in props.conf but it didn't work. It's picking up all the lines from my Log file.

Could you please suggest the next step that we have to do.

0 Karma

gcusello
SplunkTrust
SplunkTrust

one additional information:
let me understand: do you have one line for each event or one event in the file?
do you want to maintain "EVENT_SESH" event and discard the others?

this regex discards all the events where there's "EVENT_SESH", it doesn't take more events, so if you want to discard all the events from the one with "EVENT_SESH" to the end of file, you have to find a different solution: for this reason I suggested also the second solution.

If instead you want to maintain "EVENT_SESH" event and discard the others, you have to try with a different regex, to test because I don't know if the event format is always the same:

SEDCMD-delete = s/(?ms)EVENT_SESH.*/EVENT_SESH;(.*;){8}.*\./g

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...