Getting Data In

How to remove header of a log?

sabaKhadivi
Path Finder

as I edit props.conf & transforms.conf to remove header of log , but it didn't work
here is my config:

props.conf

[sourcetype]
TRANSFORMS-skiphdr= setnull

transforms.conf

[setnull]
REGEX = 
DEST_KEY = queue
FORMAT = nullQueue

Is there any idea or suggestion?

0 Karma

whrg
Motivator

I'm assuming you put the correct regex in REGEX. See @nickhillscpl answer.

Here are some more ideas:

Remember to restart Splunk after making changes to configuration fies.

Also, you must put these settings on your Heavy Forwarder / Indexer. I will not work on a Universal Forwarder.

0 Karma

sabaKhadivi
Path Finder

@whrg yes,It's heavy forwarder , and I restart splunk service after changes.

0 Karma

nickhills
Ultra Champion

Your REGEX = does not contain anything.

If there is a header string you can identify, add this to the regex.

For example, if the first line of your log was:
-------Start of Log------
you might set REGEX = \-+Start of Log\-+

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

yes ,I add the regex of unused part of log

0 Karma

nickhills
Ultra Champion

Can you post a copy of the log header and your regex - please use the code formatter which looks like 101010

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

REGEX = ^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s([0-2]\d|3[0-1])\s[0-2]\d:[0-5]\d:[0-5]\d 10.10.10.5\s1\s

useless part of my log is:
Mar 12 15:11:57 10.10.10.5 1

0 Karma

whrg
Motivator

Your REGEX looks too complicated. Try to simplify/shorten it.
Use regex101.com for testing. I noticed that your regex does not match because of the \s at the end.

0 Karma

nickhills
Ultra Champion

Try this regex: ^\w{3}\s\d+\s\d{2}:\d{2}:\d{2}\s\d+\.\d+\.\d+\.\d+\s\d
https://regex101.com/r/TwH2pp/1

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

@nickhillscpl
tnx for your answer, I give the result with SEDCMD in props.conf

0 Karma

sabaKhadivi
Path Finder

@nickhillscpl

0 Karma

sabaKhadivi
Path Finder

Mar 12 14:52:42 x.x.x.x 1 2019-03-12T14:52:42Z x.x.x.x s1 ;

this is the header that I need to remove from Mar to 1 and this is my regex (x are octet of IP Add)

^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s([0-2]\d|3[0-1])\s[0-2]\d:[0-5]\d:[0-5]\d x.x.x.x\s1\s

0 Karma

whrg
Motivator

The REGEX line does not show anything. Is this correct? If not, use the Code Sample formatting for displaying special characters.
You will need a proper regular expression.
It will help us if you post the log header (anonymized).

0 Karma

sabaKhadivi
Path Finder

@whrg Mar 12 13:44:04 10.10.10.5 1
this is the useless part of my log which I want to remove, I put regex of it infront of Regex =

0 Karma

nickhills
Ultra Champion

When you post code (or regex) use the code tool to make sure it’s is formatted/displayed.

The code tool is the icon which looks like 101010

If my comment helps, please give it a thumbs up!
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...