Getting Data In

How to configure my props.conf for a file with header in it?

Dingu
Explorer

Hello ,

I'm trying to configure my props.conf for one of the files in which it has header. I don't have any props.conf configured yet, looking for help in configuring this. Thanks in advance

example logfile:

Field1 Field2 Field3 Field4 Field5 Field6 Field7
------+------+---------------------------+---------------------------+---------------------------
0 1 6/16/20 18:35:23:193 EDT 6/16/20 18:35:23:193 EDT xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1 1 6/16/20 18:35:23:216 EDT 6/16/20 18:35:23:216 EDT yyyyyyyyyyyyyyyyyyyyyy
2 1 6/16/20 18:35:23:285 EDT 6/16/20 18:35:23:285 EDT zzzzzzzzzzzzzzzzz

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust
Will you be ingesting this file in batch mode or monitoring it?
Do you have any control over how it is created? There are 7 fields in the header, but 9 or more in the data. Changing it to comma or pipe-delimited will make for easier onboarding.
Also, see if you can get rid of the second line. If you can't, we can use SEDCMD to avoid indexing it.
---
If this reply helps you, Karma would be appreciated.

Dingu
Explorer

Thank you for your response @richgalloway 

The file will be monitored , sadly I don't have any control over the way it created and the header fields will be matching the no. of field values in the data. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It's unfortunate you can't change how the log is written, but we can work with it.  You'll need to write a transform to parse the file and use props to ignore the header lines.

props.conf:

[mysourcetype]
SEDCMD-header1 = s/Field1.*//
SEDCMD-header2 = s/[-+]+//
TRANSFORMS-fields = mytransform

transforms.conf:

[mytransform]
REGEX = (\d+)\s(?<Field2>\d+)\s(\d+\/\d+\/\d+)\s(\d\d:\d\d:\d\d:\d\d\d \w+)\s(\d+\/\d+\/\d+)\s(\d\d:\d\d:\d\d:\d\d\d \w+)\s(.*)
FORMAT = $1::$2 $3::$4 $5::$6 $7::$8 $9::$10 $11:$12 $13:$14

 

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...