Getting Data In

How to ignore first three line of my log

Builder

Is there any way to ignore first three line from my text format log?

Tags (2)
1 Solution

Legend

On the forwarder or indexer (wherever the file is), you can set the sourcetype for the file:

# inputs.conf
[monitor://yourfilename]
sourcetype=your_sourcetype

On the indexer, you can get rid of the first lines like this:

# props.conf
[your_sourcetype]
TRANSFORMS-t1=eliminate_header

# transforms.conf
[eliminate_header]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

You should not need to set crcSalt with this technique. This eliminates any lines beginning with the word "Log" or "Running".

View solution in original post

Legend

On the forwarder or indexer (wherever the file is), you can set the sourcetype for the file:

# inputs.conf
[monitor://yourfilename]
sourcetype=your_sourcetype

On the indexer, you can get rid of the first lines like this:

# props.conf
[your_sourcetype]
TRANSFORMS-t1=eliminate_header

# transforms.conf
[eliminate_header]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

You should not need to set crcSalt with this technique. This eliminates any lines beginning with the word "Log" or "Running".

View solution in original post

Legend

The inputs.conf file belongs on the forwarder. The props.conf file and the transforms.conf file must go on the indexer (or wherever the data is parsed).

Builder

I was changing *.conf files from Splunkforwarder instead of main instance, now these lines are not displaying in search result.

0 Karma

Builder

Any Update regarding my question?

0 Karma

Legend

This will not eliminate data that has already been indexed. Remember that Splunk is essentially a "write once" datastore.

If you want to eliminate data that has already been indexed, you will need to clean eventdata or use the delete command. Restarting Splunk won't do the job.

http://docs.splunk.com/Documentation/Splunk/latest/admin/RemovedatafromSplunk

0 Karma

Builder

Its not working 😞 here is my conf settings from splunk forwarder

[root@hv-centos local]# cat inputs.conf
[default]
host = hv-centos

[monitor:/home/manoj/rels/PATCH/log/default]
sourcetype = TAFCLOGLINE

[root@hv-centos local]# cat props.conf
[TAFCLOGLINE]
TRANSFORMS-t1=eliminatefirstthree_line

[root@hv-centos local]# cat transforms.conf
[eliminatefirstthreeline]
REGEX=^(?:Log|Running)\s
DEST
KEY=queue
FORMAT=nullQueue

I restarted splunk after the changes
I can still see the first three line form logs in splunk main instance.

0 Karma

Builder

There is common pattern in my log file, every log message contain a character it could be I or W or E or F.

do you think its good idea to use regex _raw="^[I|W|E|F]" for all search result?

if it good then How can I create sourcetype based on above regular expression?

0 Karma

Builder

my three line are

Log file created at: 2012/05/17 11:47:18
Running on machine: TEST-W2K
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg

0 Karma

Splunk Employee
Splunk Employee

What type of log file is it? Do the lines start with a "#" or something else? If so you can route them to the null queue to ignore them.

http://docs.splunk.com/Documentation/Splunk/4.0.9/Admin/Routeeventstospecificqueues

When skipping the first few lines, in inputs.conf you'll want to set crcSalt=<SOURCE>.

0 Karma

Builder

there is common pattern for my log first char is always I or W or F or E.
I think this is better way to skip three line, search only based regex _raw="^[I|W|E|F]"
Is it good idea?

0 Karma