topic Re: Inconsistent sourcetype while indexing CSV files in Getting Data In

Inconsistent sourcetype while indexing CSV files

erga00 — Sat, 26 Jun 2010 03:48:42 GMT

I'm indexing a CSV file and I just can't get Splunk to extract any fields or apply the proper sourcetype to the events. The desired sourcetype is acs and what I'm getting is acs-n where is some number which varies, acs-1, acs-2, etc.

I saw another answer about this issue which basically stated that it had something to do with the functionality that automatically parses field names from the header. I don't have CHECK_FOR_HEADER = true and I specify the sourcetype in the inputs.conf stanza so I don't understand what's going on.

I should mention that I have another CSV file being indexed using the same procedure as below and it works perfectly.

Has anyone run into this before?

input file:
Date,Time,Message-Type,User-Name,Group-Name,Caller-ID,NAS-Port,NAS-IP-Address
04/03/2010,00:00:00,Authen OK,c1tr1x_wds,Network Services,0004.235f.4c1c,c1tr1x_wds,10.6.70.

inputs.conf:
[monitor://\server\logs]
sourcetype = acs
_whitelist = Failed Attempts.*.csv

props.conf:
[acs]
REPORT-acs-fail = acs-fail-fields

transforms.conf:
[acs-fail-fields]
FIELDS = "date", "time", "message-type", "user", "user_group", "message", "junk1", "junk2", "src_mac", "nas-port", "src_ip"
DELIMS = ","

Re: Inconsistent sourcetype while indexing CSV files

Lowell — Mon, 28 Sep 2020 09:14:02 GMT

Are you doing any forwarding? I believe that CHECK_FOR_HEADER doesn't work right when forwarders are involved. (I could be wrong about that.) I've always ended up making explicit sourcetypes rather than relying on CHECK_FOR_HEADER, for whatever that's worth.

Re: Inconsistent sourcetype while indexing CSV files

jrodman — Sat, 26 Jun 2010 05:53:10 GMT

This is the fallout from CHECK_FOR_HEADER.

There's a bug where CHECK_FOR_HEADER is applied based on filename pattern, despite not being in your sourcetype. You might be able to work around by declaring CHECK_FOR_HEADER = false explicitly in your sourcetype, or by declaring [source::.../Failed Attempts.*.csv] sourcetype=acs in props.conf, especially if you set a priority value of eg 100.

The bug is due for fixing in 4.1.4, which will be a few weeks -- we're verifying bugs as fast as we can.

Re: Inconsistent sourcetype while indexing CSV files

erga00 — Mon, 28 Jun 2010 22:08:29 GMT

Yes. This is on a forwarder.

I thought I was setting an explicit sourcetype. Am I not understanding what that means?

Re: Inconsistent sourcetype while indexing CSV files

erga00 — Mon, 28 Sep 2020 09:14:08 GMT

Neither explicitly setting CHECK_FOR_HEADER=false or declaring [source::.../Failed Attempts.*.csv] with priority = 100 worked.

Thanks for the help. I'll open a support case and see if they can track it down.

Re: Inconsistent sourcetype while indexing CSV files

jrodman — Mon, 28 Sep 2020 09:14:10 GMT

In the dumbest possible case, you can either disable the [source::....csv] sourcetype=csv stanza, or turn off CHECK_FOR_HEADER for the [csv] sourcetype stanza. Obviously these approaches are a bit more intrusive.

Re: Inconsistent sourcetype while indexing CSV files

sideview — Mon, 28 Sep 2020 09:25:52 GMT

fwiw something similar is happening to me on 4.1.6. But I do need CHECK_FOR_HEADER, so maybe I'm out of luck. Are there docs about what the "-2" suffixes are trying to do for me?

Re: Inconsistent sourcetype while indexing CSV files

jrodman — Sat, 05 Mar 2011 07:47:51 GMT

The expansion of the sourcetype into multiple sourcetypes allows the system to store the list of fields on a per-file basis, since there's no guarantee it doesn't change. Why the first one is -2, instead of nothing or -1, is some minor internal misbehavior that doesn't really change the situation, since you'll get multiple sourcetypes eventually. This behavior forces you to account for multiple sourcetypes in your searches manually, by eg folding them back together using eval and such. It's a pain.