topic How to remove header of a log? in Getting Data In

How to remove header of a log?

sabaKhadivi — Tue, 12 Mar 2019 09:30:35 GMT

as I edit props.conf & transforms.conf to remove header of log , but it didn't work
here is my config:

props.conf

[sourcetype]
TRANSFORMS-skiphdr= setnull

transforms.conf

[setnull]
REGEX = 
DEST_KEY = queue
FORMAT = nullQueue

Is there any idea or suggestion?

Re: How to remove header of a log?

whrg — Tue, 12 Mar 2019 09:50:29 GMT

The REGEX line does not show anything. Is this correct? If not, use the Code Sample formatting for displaying special characters.
You will need a proper regular expression.
It will help us if you post the log header (anonymized).

Re: How to remove header of a log?

nickhills — Tue, 12 Mar 2019 09:55:24 GMT

Your REGEX = does not contain anything.

If there is a header string you can identify, add this to the regex.

For example, if the first line of your log was:
-------Start of Log------
you might set REGEX = \-+Start of Log\-+

Re: How to remove header of a log?

sabaKhadivi — Tue, 12 Mar 2019 10:18:54 GMT

yes ,I add the regex of unused part of log

Re: How to remove header of a log?

sabaKhadivi — Tue, 12 Mar 2019 10:22:08 GMT

@whrg Mar 12 13:44:04 10.10.10.5 1
this is the useless part of my log which I want to remove, I put regex of it infront of Regex =

Re: How to remove header of a log?

nickhills — Tue, 12 Mar 2019 10:26:15 GMT

When you post code (or regex) use the code tool to make sure it’s is formatted/displayed.

The code tool is the icon which looks like 101010

Re: How to remove header of a log?

whrg — Tue, 12 Mar 2019 10:29:30 GMT

I'm assuming you put the correct regex in REGEX. See @nickhillscpl answer.

Here are some more ideas:

Remember to restart Splunk after making changes to configuration fies.

Also, you must put these settings on your Heavy Forwarder / Indexer. I will not work on a Universal Forwarder.

Re: How to remove header of a log?

sabaKhadivi — Tue, 12 Mar 2019 10:33:20 GMT

@whrg yes,It's heavy forwarder , and I restart splunk service after changes.

Re: How to remove header of a log?

nickhills — Tue, 12 Mar 2019 10:40:25 GMT

Can you post a copy of the log header and your regex - please use the code formatter which looks like 101010

Re: How to remove header of a log?

sabaKhadivi — Tue, 12 Mar 2019 11:27:25 GMT

Mar 12 14:52:42 x.x.x.x 1 2019-03-12T14:52:42Z x.x.x.x s1 ;

this is the header that I need to remove from Mar to 1 and this is my regex (x are octet of IP Add)

^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s([0-2]\d|3[0-1])\s[0-2]\d:[0-5]\d:[0-5]\d x.x.x.x\s1\s

Re: How to remove header of a log?

sabaKhadivi — Tue, 12 Mar 2019 11:42:36 GMT

REGEX = ^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s([0-2]\d|3[0-1])\s[0-2]\d:[0-5]\d:[0-5]\d 10.10.10.5\s1\s

useless part of my log is:
Mar 12 15:11:57 10.10.10.5 1

Re: How to remove header of a log?

sabaKhadivi — Wed, 13 Mar 2019 05:31:59 GMT

@nickhillscpl

Re: How to remove header of a log?

whrg — Wed, 13 Mar 2019 09:54:04 GMT

Your REGEX looks too complicated. Try to simplify/shorten it.
Use regex101.com for testing. I noticed that your regex does not match because of the \s at the end.

Re: How to remove header of a log?

nickhills — Wed, 13 Mar 2019 09:59:34 GMT

Try this regex: ^\w{3}\s\d+\s\d{2}:\d{2}:\d{2}\s\d+\.\d+\.\d+\.\d+\s\d
https://regex101.com/r/TwH2pp/1

Re: How to remove header of a log?

sabaKhadivi — Sat, 16 Mar 2019 12:12:00 GMT

@nickhillscpl
tnx for your answer, I give the result with SEDCMD in props.conf