topic cvs log is not getting ingested when it has only one line (in addition to the header) in Getting Data In

cvs log is not getting ingested when it has only one line (in addition to the header)

mlevsh — Thu, 01 Feb 2024 15:04:09 GMT

Hi,

We came across strange issue:
cvs logs are not getting ingested when it only has only one line (in addition to the header) in a log.
The same logs with two and more lines are ingested successfully

Here are inputs.conf and props.conf we are using

Inputs.conf [monitor:///apps/ab_cd/resources/abcd/reports_rr/reports/abc/.../*_splunk.csv] sourcetype=source_type_name index=index_name ignoreOlderThan = 2h crcSalt = <SOURCE> props.conf [source_type_name] KV_MODE = none NO_BINARY_CHECK = true SHOULD_LINEMERGE = false PREAMBLE_REGEX = ^Region TIME_PREFIX= ^(?:[^,\n]*,){1} TIME_FORMAT = %Y-%m-%d MAX_TIMESTAMP_LOOKAHEAD=10 MAX_DAYS_HENCE = 5

Appreciate all the ideas

Re: cvs log is not getting ingested when it has only one line (in addition to the header)

Richfez — Thu, 01 Feb 2024 13:57:48 GMT

I was expecting your props.conf to have

INDEXED_EXTRACTIONS = CSV

You are also using a TIME_PREFIX instead of TIMESTAMP_FIELDS...

And you have a PREAMBLE_REGEX set, which looks like it's set to the first fieldname in the header (which would remove the header line) though you don't provide FIELD_NAMES...

Putting that all together, it looks like you aren't really treating those files as CSV files. I'm not sure what's going on, but I wonder if it would work right if you treated them as CSV.

If that doesn't help, it'd be useful to see the contents of a file that doesn't work, and one that does.