Getting Data In

How to ignore first three line of my log

jangid
Builder

Is there any way to ignore first three line from my text format log?

Tags (2)
1 Solution

lguinn2
Legend

On the forwarder or indexer (wherever the file is), you can set the sourcetype for the file:

# inputs.conf
[monitor://yourfilename]
sourcetype=your_sourcetype

On the indexer, you can get rid of the first lines like this:

# props.conf
[your_sourcetype]
TRANSFORMS-t1=eliminate_header

# transforms.conf
[eliminate_header]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

You should not need to set crcSalt with this technique. This eliminates any lines beginning with the word "Log" or "Running".

View solution in original post

lguinn2
Legend

On the forwarder or indexer (wherever the file is), you can set the sourcetype for the file:

# inputs.conf
[monitor://yourfilename]
sourcetype=your_sourcetype

On the indexer, you can get rid of the first lines like this:

# props.conf
[your_sourcetype]
TRANSFORMS-t1=eliminate_header

# transforms.conf
[eliminate_header]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

You should not need to set crcSalt with this technique. This eliminates any lines beginning with the word "Log" or "Running".

lguinn2
Legend

The inputs.conf file belongs on the forwarder. The props.conf file and the transforms.conf file must go on the indexer (or wherever the data is parsed).

jangid
Builder

I was changing *.conf files from Splunkforwarder instead of main instance, now these lines are not displaying in search result.

0 Karma

jangid
Builder

Any Update regarding my question?

0 Karma

lguinn2
Legend

This will not eliminate data that has already been indexed. Remember that Splunk is essentially a "write once" datastore.

If you want to eliminate data that has already been indexed, you will need to clean eventdata or use the delete command. Restarting Splunk won't do the job.

http://docs.splunk.com/Documentation/Splunk/latest/admin/RemovedatafromSplunk

0 Karma

jangid
Builder

Its not working 😞 here is my conf settings from splunk forwarder

[root@hv-centos local]# cat inputs.conf
[default]
host = hv-centos

[monitor:/home/manoj/rels/PATCH/log/default]
sourcetype = TAFC_LOG_LINE

[root@hv-centos local]# cat props.conf
[TAFC_LOG_LINE]
TRANSFORMS-t1=eliminate_first_three_line

[root@hv-centos local]# cat transforms.conf
[eliminate_first_three_line]
REGEX=^(?:Log|Running)\s
DEST_KEY=queue
FORMAT=nullQueue

I restarted splunk after the changes
I can still see the first three line form logs in splunk main instance.

0 Karma

jangid
Builder

There is common pattern in my log file, every log message contain a character it could be I or W or E or F.

do you think its good idea to use regex _raw="^[I|W|E|F]" for all search result?

if it good then How can I create sourcetype based on above regular expression?

0 Karma

jangid
Builder

my three line are

Log file created at: 2012/05/17 11:47:18
Running on machine: TEST-W2K
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg

0 Karma

sdaniels
Splunk Employee
Splunk Employee

What type of log file is it? Do the lines start with a "#" or something else? If so you can route them to the null queue to ignore them.

http://docs.splunk.com/Documentation/Splunk/4.0.9/Admin/Routeeventstospecificqueues

When skipping the first few lines, in inputs.conf you'll want to set crcSalt=<SOURCE>.

0 Karma

jangid
Builder

there is common pattern for my log first char is always I or W or F or E.
I think this is better way to skip three line, search only based regex _raw="^[I|W|E|F]"
Is it good idea?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...