Getting Data In

Why the header from psv file sometimes being indexed and fields are not extracted while in most times it is ok

mlevsh
Builder

One of our teams on-boards psv logs and while the data on-boarded correctly in most case, sometimes the header is not recognized and field extraction is not happening. 
props.conf:

[status_psv]
INDEXED_EXTRACTIONS = PSV
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
HEADER_FIELD_LINE_NUMBER = 1
TIMESTAMP_FIELDS = rqid
TIME_FORMAT = %s%6Q
MAX_DAYS_HENCE = 5


what are the possible reasons for some logs ST being ignored

Labels (3)
Tags (2)
0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Can you be certain that there always is a header line and that it is always line 1? Have you tried to set HEADER_FIELD_LINE_NUMBER to '0' to let Splunk figure it out?

Also, is it possible that the file is being overwritten vs. being rolled while it is read?

0 Karma

mlevsh
Builder

@s2_splunk 

Thank you for you suggestions!

Majority (99.9%) of logs have header set as a 1st line and this Sourcetype works for other logs.
What makes this issue harder to troubleshoot that  the files are deleted after they ingested.
We were not successful to catch the actual log file so far.  the team will try HEADER_FIELD_LINE_NUMBER = 0 (which is default anyway) , but they want to troubleshoot without this change first

 

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...