Getting Data In

How to remove header of a log?

sabaKhadivi
Path Finder

as I edit props.conf & transforms.conf to remove header of log , but it didn't work
here is my config:

props.conf

[sourcetype]
TRANSFORMS-skiphdr= setnull

transforms.conf

[setnull]
REGEX = 
DEST_KEY = queue
FORMAT = nullQueue

Is there any idea or suggestion?

0 Karma

whrg
Motivator

I'm assuming you put the correct regex in REGEX. See @nickhillscpl answer.

Here are some more ideas:

Remember to restart Splunk after making changes to configuration fies.

Also, you must put these settings on your Heavy Forwarder / Indexer. I will not work on a Universal Forwarder.

0 Karma

sabaKhadivi
Path Finder

@whrg yes,It's heavy forwarder , and I restart splunk service after changes.

0 Karma

nickhills
Ultra Champion

Your REGEX = does not contain anything.

If there is a header string you can identify, add this to the regex.

For example, if the first line of your log was:
-------Start of Log------
you might set REGEX = \-+Start of Log\-+

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

yes ,I add the regex of unused part of log

0 Karma

nickhills
Ultra Champion

Can you post a copy of the log header and your regex - please use the code formatter which looks like 101010

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

REGEX = ^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s([0-2]\d|3[0-1])\s[0-2]\d:[0-5]\d:[0-5]\d 10.10.10.5\s1\s

useless part of my log is:
Mar 12 15:11:57 10.10.10.5 1

0 Karma

whrg
Motivator

Your REGEX looks too complicated. Try to simplify/shorten it.
Use regex101.com for testing. I noticed that your regex does not match because of the \s at the end.

0 Karma

nickhills
Ultra Champion

Try this regex: ^\w{3}\s\d+\s\d{2}:\d{2}:\d{2}\s\d+\.\d+\.\d+\.\d+\s\d
https://regex101.com/r/TwH2pp/1

If my comment helps, please give it a thumbs up!
0 Karma

sabaKhadivi
Path Finder

@nickhillscpl
tnx for your answer, I give the result with SEDCMD in props.conf

0 Karma

sabaKhadivi
Path Finder

@nickhillscpl

0 Karma

sabaKhadivi
Path Finder

Mar 12 14:52:42 x.x.x.x 1 2019-03-12T14:52:42Z x.x.x.x s1 ;

this is the header that I need to remove from Mar to 1 and this is my regex (x are octet of IP Add)

^(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s([0-2]\d|3[0-1])\s[0-2]\d:[0-5]\d:[0-5]\d x.x.x.x\s1\s

0 Karma

whrg
Motivator

The REGEX line does not show anything. Is this correct? If not, use the Code Sample formatting for displaying special characters.
You will need a proper regular expression.
It will help us if you post the log header (anonymized).

0 Karma

sabaKhadivi
Path Finder

@whrg Mar 12 13:44:04 10.10.10.5 1
this is the useless part of my log which I want to remove, I put regex of it infront of Regex =

0 Karma

nickhills
Ultra Champion

When you post code (or regex) use the code tool to make sure it’s is formatted/displayed.

The code tool is the icon which looks like 101010

If my comment helps, please give it a thumbs up!
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...