topic Re: How to ignore first three line of my log in Getting Data In

How to ignore first three line of my log

jangid — Thu, 31 May 2012 11:40:14 GMT

Is there any way to ignore first three line from my text format log?

Re: How to ignore first three line of my log

sdaniels — Thu, 31 May 2012 11:57:07 GMT

What type of log file is it? Do the lines start with a "#" or something else? If so you can route them to the null queue to ignore them.

http://docs.splunk.com/Documentation/Splunk/4.0.9/Admin/Routeeventstospecificqueues

When skipping the first few lines, in inputs.conf you'll want to set crcSalt=<SOURCE>.

Re: How to ignore first three line of my log

jangid — Thu, 31 May 2012 12:21:47 GMT

my three line are

Log file created at: 2012/05/17 11:47:18
Running on machine: TEST-W2K
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg

Re: How to ignore first three line of my log

jangid — Thu, 31 May 2012 12:35:29 GMT

there is common pattern for my log first char is always I or W or F or E.
I think this is better way to skip three line, search only based regex _raw="^[I|W|E|F]"
Is it good idea?

Re: How to ignore first three line of my log

jangid — Thu, 31 May 2012 12:41:46 GMT

There is common pattern in my log file, every log message contain a character it could be I or W or E or F.

do you think its good idea to use regex _raw="^[I|W|E|F]" for all search result?

if it good then How can I create sourcetype based on above regular expression?

Re: How to ignore first three line of my log

lguinn2 — Fri, 01 Jun 2012 08:44:35 GMT

On the forwarder or indexer (wherever the file is), you can set the sourcetype for the file:

# inputs.conf
[monitor://yourfilename]
sourcetype=your_sourcetype

On the indexer, you can get rid of the first lines like this:

# props.conf
[your_sourcetype]
TRANSFORMS-t1=eliminate_header

# transforms.conf
[eliminate_header]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

You should not need to set crcSalt with this technique. This eliminates any lines beginning with the word "Log" or "Running".

Re: How to ignore first three line of my log

jangid — Mon, 28 Sep 2020 11:53:36 GMT

Its not working 😞 here is my conf settings from splunk forwarder

[root@hv-centos local]# cat inputs.conf
[default]
host = hv-centos

[monitor:/home/manoj/rels/PATCH/log/default]
sourcetype = TAFC_LOG_LINE

[root@hv-centos local]# cat props.conf
[TAFC_LOG_LINE]
TRANSFORMS-t1=eliminate_first_three_line

[root@hv-centos local]# cat transforms.conf
[eliminate_first_three_line]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

I restarted splunk after the changes
I can still see the first three line form logs in splunk main instance.

Re: How to ignore first three line of my log

lguinn2 — Fri, 01 Jun 2012 12:58:31 GMT

This will not eliminate data that has already been indexed. Remember that Splunk is essentially a "write once" datastore.

If you want to eliminate data that has already been indexed, you will need to clean eventdata or use the delete command. Restarting Splunk won't do the job.

http://docs.splunk.com/Documentation/Splunk/latest/admin/RemovedatafromSplunk

Re: How to ignore first three line of my log

jangid — Wed, 06 Jun 2012 10:53:23 GMT

Any Update regarding my question?

Re: How to ignore first three line of my log

jangid — Thu, 07 Jun 2012 10:16:56 GMT

I was changing *.conf files from Splunkforwarder instead of main instance, now these lines are not displaying in search result.

Re: How to ignore first three line of my log

lguinn2 — Thu, 07 Jun 2012 22:19:20 GMT

The inputs.conf file belongs on the forwarder. The props.conf file and the transforms.conf file must go on the indexer (or wherever the data is parsed).