topic Re: Why the header from psv file sometimes being indexed and fields are not extracted while in most times it is ok in Getting Data In

Why the header from psv file sometimes being indexed and fields are not extracted while in most times it is ok

mlevsh — Mon, 03 May 2021 19:01:10 GMT

One of our teams on-boards psv logs and while the data on-boarded correctly in most case, sometimes the header is not recognized and field extraction is not happening.
props.conf:

[status_psv]
INDEXED_EXTRACTIONS = PSV
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
HEADER_FIELD_LINE_NUMBER = 1
TIMESTAMP_FIELDS = rqid
TIME_FORMAT = %s%6Q
MAX_DAYS_HENCE = 5

what are the possible reasons for some logs ST being ignored

Re: Why the header from psv file sometimes being indexed and fields are not extracted while in most times it is ok

s2_splunk — Mon, 03 May 2021 21:57:21 GMT

Can you be certain that there always is a header line and that it is always line 1? Have you tried to set HEADER_FIELD_LINE_NUMBER to '0' to let Splunk figure it out?

Also, is it possible that the file is being overwritten vs. being rolled while it is read?

Re: Why the header from psv file sometimes being indexed and fields are not extracted while in most times it is ok

mlevsh — Tue, 04 May 2021 15:59:10 GMT

@s2_splunk

Thank you for you suggestions!

Majority (99.9%) of logs have header set as a 1st line and this Sourcetype works for other logs.
What makes this issue harder to troubleshoot that the files are deleted after they ingested.
We were not successful to catch the actual log file so far. the team will try HEADER_FIELD_LINE_NUMBER = 0 (which is default anyway) , but they want to troubleshoot without this change first