Getting Data In

Why the header from psv file sometimes being indexed and fields are not extracted while in most times it is ok

mlevsh
Builder

One of our teams on-boards psv logs and while the data on-boarded correctly in most case, sometimes the header is not recognized and field extraction is not happening. 
props.conf:

[status_psv]
INDEXED_EXTRACTIONS = PSV
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
HEADER_FIELD_LINE_NUMBER = 1
TIMESTAMP_FIELDS = rqid
TIME_FORMAT = %s%6Q
MAX_DAYS_HENCE = 5


what are the possible reasons for some logs ST being ignored

Labels (3)
Tags (2)
0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Can you be certain that there always is a header line and that it is always line 1? Have you tried to set HEADER_FIELD_LINE_NUMBER to '0' to let Splunk figure it out?

Also, is it possible that the file is being overwritten vs. being rolled while it is read?

0 Karma

mlevsh
Builder

@s2_splunk 

Thank you for you suggestions!

Majority (99.9%) of logs have header set as a 1st line and this Sourcetype works for other logs.
What makes this issue harder to troubleshoot that  the files are deleted after they ingested.
We were not successful to catch the actual log file so far.  the team will try HEADER_FIELD_LINE_NUMBER = 0 (which is default anyway) , but they want to troubleshoot without this change first

 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...