Getting Data In

cvs log is not getting ingested when it has only one line (in addition to the header)

mlevsh
Builder

Hi,

We came across strange issue:
cvs logs are not getting ingested when it only has only one line (in addition to the header) in a log.
The same logs with two and more lines are ingested successfully

Here are inputs.conf and  props.conf we are using

Inputs.conf
[monitor:///apps/ab_cd/resources/abcd/reports_rr/reports/abc/.../*_splunk.csv]  
sourcetype=source_type_name
index=index_name
ignoreOlderThan = 2h
crcSalt = <SOURCE>

props.conf

[source_type_name]
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
PREAMBLE_REGEX = ^Region
TIME_PREFIX= ^(?:[^,\n]*,){1}
TIME_FORMAT = %Y-%m-%d
MAX_TIMESTAMP_LOOKAHEAD=10
MAX_DAYS_HENCE = 5


Appreciate all the ideas

0 Karma

Richfez
SplunkTrust
SplunkTrust

I was expecting your props.conf to have

INDEXED_EXTRACTIONS = CSV

You are also using a TIME_PREFIX instead of TIMESTAMP_FIELDS... 

And you have a PREAMBLE_REGEX set, which looks like it's set to the first fieldname in the header (which would remove the header line) though you don't provide FIELD_NAMES...

Putting that all together, it looks like you aren't really treating those files as CSV files.  I'm not sure what's going on, but I wonder if it would work right if you treated them as CSV.

If that doesn't help, it'd be useful to see the contents of a file that doesn't work, and one that does. 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...