Getting Data In

I have a log file. I want to ignore lines from that log file before indexing it. can I do this? Please help me on this matter as soon as possible

saibal6
Path Finder

In the log file I have below mentioned a line :

EVENT_SESH;0;04/01/2018 06:30:23:5000;1;;1;0;;;END OF IMPORT PROCESS FOR THE MASTER STORE - PENDING_TXT(0).

I want to ignore all the lines after above mentioned line before indexing.

Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi saibal6,
let me understand:

  • you have files with many logs to take,
  • when there's the above event, do you want to discard alla events until the end of the file,

is it correct?

In this case, you could use in props.conf

SEDCMD-delete = s/(?ms)EVENT_SESH.*//g

Or a pre-parsing script to run before indexing

If instead you want to discard only the event with EVENT_SESH, you can follow http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad

Edit props.conf and add the following:

[your_sourcetype]
TRANSFORMS-set= setparsing,setnull

Edit transforms.conf and add the following:

[setnull]
REGEX = EVENT_SESH
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi saibal6,
let me understand:

  • you have files with many logs to take,
  • when there's the above event, do you want to discard alla events until the end of the file,

is it correct?

In this case, you could use in props.conf

SEDCMD-delete = s/(?ms)EVENT_SESH.*//g

Or a pre-parsing script to run before indexing

If instead you want to discard only the event with EVENT_SESH, you can follow http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad

Edit props.conf and add the following:

[your_sourcetype]
TRANSFORMS-set= setparsing,setnull

Edit transforms.conf and add the following:

[setnull]
REGEX = EVENT_SESH
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Bye.
Giuseppe

0 Karma

saibal6
Path Finder

Thanks Giuseppe for the answer.

In our case your first suggestion is useful.
we added the line in " SEDCMD-delete = s/(?ms)EVENT_SESH.*//g" in props.conf but it didn't work. It's picking up all the lines from my Log file.

Could you please suggest the next step that we have to do.

0 Karma

gcusello
SplunkTrust
SplunkTrust

one additional information:
let me understand: do you have one line for each event or one event in the file?
do you want to maintain "EVENT_SESH" event and discard the others?

this regex discards all the events where there's "EVENT_SESH", it doesn't take more events, so if you want to discard all the events from the one with "EVENT_SESH" to the end of file, you have to find a different solution: for this reason I suggested also the second solution.

If instead you want to maintain "EVENT_SESH" event and discard the others, you have to try with a different regex, to test because I don't know if the event format is always the same:

SEDCMD-delete = s/(?ms)EVENT_SESH.*/EVENT_SESH;(.*;){8}.*\./g

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Hi Splunky people! We are excited to share the newest updates in Splunk Enterprise 9.3!Admins and Analyst can ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...