How to ignore first three line of my log

[jangid]

Is there any way to ignore first three line from my text format log?

[lguinn2 · Accepted solution]

On the forwarder or indexer (wherever the file is), you can set the sourcetype for the file:

# inputs.conf
[monitor://yourfilename]
sourcetype=your_sourcetype

On the indexer, you can get rid of the first lines like this:

# props.conf
[your_sourcetype]
TRANSFORMS-t1=eliminate_header

# transforms.conf
[eliminate_header]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

You should not need to set crcSalt with this technique. This eliminates any lines beginning with the word "Log" or "Running".

[lguinn2]

The inputs.conf file belongs on the forwarder. The props.conf file and the transforms.conf file must go on the indexer (or wherever the data is parsed).

[jangid]

I was changing *.conf files from Splunkforwarder instead of main instance, now these lines are not displaying in search result.

[jangid]

Any Update regarding my question?

[lguinn2]

This will not eliminate data that has already been indexed. Remember that Splunk is essentially a "write once" datastore.

If you want to eliminate data that has already been indexed, you will need to clean eventdata or use the delete command. Restarting Splunk won't do the job.

http://docs.splunk.com/Documentation/Splunk/latest/admin/RemovedatafromSplunk

[jangid]

Its not working 😞 here is my conf settings from splunk forwarder

[root@hv-centos local]# cat inputs.conf
[default]
host = hv-centos

[monitor:/home/manoj/rels/PATCH/log/default]
sourcetype = TAFC_LOG_LINE

[root@hv-centos local]# cat props.conf
[TAFC_LOG_LINE]
TRANSFORMS-t1=eliminate_first_three_line

[root@hv-centos local]# cat transforms.conf
[eliminate_first_three_line]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

I restarted splunk after the changes
I can still see the first three line form logs in splunk main instance.

[jangid]

There is common pattern in my log file, every log message contain a character it could be I or W or E or F.

do you think its good idea to use regex _raw="^[I|W|E|F]" for all search result?

if it good then How can I create sourcetype based on above regular expression?

[jangid]

my three line are

Log file created at: 2012/05/17 11:47:18
Running on machine: TEST-W2K
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg

[sdaniels]

What type of log file is it? Do the lines start with a "#" or something else? If so you can route them to the null queue to ignore them.

http://docs.splunk.com/Documentation/Splunk/4.0.9/Admin/Routeeventstospecificqueues

When skipping the first few lines, in inputs.conf you'll want to set crcSalt=<SOURCE>.

[jangid]

there is common pattern for my log first char is always I or W or F or E.
I think this is better way to skip three line, search only based regex _raw="^[I|W|E|F]"
Is it good idea?