topic Re: Splunk insisting on Auto-finding CSV fields? in Getting Data In

Splunk insisting on Auto-finding CSV fields?

travispowell — Thu, 28 Apr 2011 18:20:01 GMT

Hello,

Splunk is insisting on trying to auto-find headers in a tab-delimited CSV file for which I have manually defined headers in a CONF file. I thought that putting this information in /etc/system/local would override the /etc/apps/learned/ but it doesn't look like that's the case...

Here are my CONF files for system-local:

inputs.conf

[monitor:///logs/strauss_splunk/bulksession]
sourcetype=csv
source=strauss_sessions
index=strauss_sessions
host=WHITNEY

[monitor:///logs/strauss_splunk/bulkurl]
sourcetype=csv
source=strauss_url
index=strauss_url
host=WHITNEY

[monitor:///inetpub/strauss_splunk/bulkhit]
sourcetype=csv
source=strauss_hits
index=strauss_hits
host=WHITNEY

props.conf

[source::strauss_url]
SHOULD_LINEMERGE=false
CHECK_FOR_HEADER=false
TRANSFORMS-STRAUSSTSV=STRAUSSTSV-1

[source::strauss_sessions]
SHOULD_LINEMERGE=false
CHECK_FOR_HEADER=false
TRANSFORMS-STRAUSSTSV=STRAUSSTSV-2

[source::strauss_hits]
SHOULD_LINEMERGE=false
CHECK_FOR_HEADER=false
TRANSFORMS-STRAUSSTSV=STRAUSSTSV-3

transforms.conf

[STRAUSSTSV-3]
DELIMS = "  "
FIELDS = "SESSION_KEY", "HIT_KEY", "ID", "SECURE"

[STRAUSSTSV-2]
DELIMS = "  "
FIELDS = "SESSION_KEY", "ADDRESS", "CANISTER"

[STRAUSSTSV-1]
DELIMS = "  "
FIELDS = "SESSION_KEY", "HIT_KEY", "NAME", "VALUE", "TIMESTAMP"

* * * * * * * * * * * * * * * * * * * * * * * *

...and this all looks good, right? But... this is what the /system/learned/ CONF files populate as afterwards:

* * * * * * * * * * * * * * * * * * * * * * * *

props.conf

[csv-2]
KV_MODE = none
REPORT-AutoHeader = AutoHeader-1
SHOULD_LINEMERGE = False
given_type = csv
pulldown_type = true

[csv-3]
KV_MODE = none
REPORT-AutoHeader = AutoHeader-2
SHOULD_LINEMERGE = False
given_type = csv
pulldown_type = true

[csv-4]
KV_MODE = none
REPORT-AutoHeader = AutoHeader-3
SHOULD_LINEMERGE = False
given_type = csv
pulldown_type = true

[csv-5]
KV_MODE = none
REPORT-AutoHeader = AutoHeader-4
SHOULD_LINEMERGE = False
given_type = csv
pulldown_type = true

[csv-6]
KV_MODE = none
REPORT-AutoHeader = AutoHeader-5
SHOULD_LINEMERGE = False
given_type = csv
pulldown_type = true95

transforms.conf

[AutoHeader-1]
DELIMS = "  "
FIELDS = "58b0c3f3c517dd9ee90cf256800dae98", "27eec7a8d949b8afe03a11b47604633b", "B99004EA29E4FA783416FAC3F7AB87A5", "N"

[AutoHeader-2]
DELIMS = "  "
FIELDS = "0f4c4f0c76bb2898ccbcfa816cfbe49b", "cbb2438acf31acf8acefacb3ff2b59a9", "EA9AE62469C9E2DCE926B32A545675EA", "N"

[AutoHeader-3]
DELIMS = "  "
FIELDS = "8ed97b717ce4b20e561a9c5a033f925c", "44bdc568a0b0052fd2232239257ebc6c", "897FFA5C1C1F078517B1FF8DB392AC54", "Y"

[AutoHeader-4]
DELIMS = "  "
FIELDS = "58b0c3f3c517dd9ee90cf256800dae98", "63.194.158.158", "LSSN_20110419_WHITNEY.dat"

[AutoHeader-5]
DELIMS = "  "
FIELDS = "becf1cd6433bd8ddf2a3f4e9da3fe133", "20c184e3ce2396fbda9d5071c8b3344d", "login_username", "XXXXXXXXXXXXXXXX", "2011-04-19 07:00:20.000"

Any ideas??

Thank you so much

Re: Splunk insisting on Auto-finding CSV fields?

JSapienza — Mon, 28 Sep 2020 09:30:32 GMT

You might want to add this to your props.conf

http://www.splunk.com/base/Documentation/latest/admin/Propsconf

LEARN_SOURCETYPE = [true|false]

Determines whether learning of known or unknown sourcetypes is enabled. * For known sourcetypes, refer to LEARN_MODEL. * For unknown sourcetypes, refer to the rule:: and delayedrule:: configuration (see below).
Setting this field to false disables CHECK_FOR_HEADER as well (see above).
Defaults to true.

Re: Splunk insisting on Auto-finding CSV fields?

travispowell — Thu, 28 Apr 2011 19:44:32 GMT

Thanks, haven't got it to work yet, but I'll keep investigating. I think I might have to change some other things around relating to the source types.

Re: Splunk insisting on Auto-finding CSV fields?

JSapienza — Thu, 28 Apr 2011 19:59:26 GMT

Light bulb went off when I re-read your question. You will need to use DELIMS = "\t" for tab and not " "

Re: Splunk insisting on Auto-finding CSV fields?

travispowell — Thu, 28 Apr 2011 20:21:16 GMT

Thanks for the suggestions, but neither of those actually fixed it. The learn_sourcetype modifier did stop Splunk from trying to auto-define fields, but it didn't let my CONF files take over...

Re: Splunk insisting on Auto-finding CSV fields?

JSapienza — Thu, 28 Apr 2011 20:48:00 GMT

I think changes at this point would only apply to new event's and not events already indexed.

Re: Splunk insisting on Auto-finding CSV fields?

travispowell — Thu, 28 Apr 2011 20:59:40 GMT

This is a test box, because I have about 10+ GB /day of this stuff to index, (short halflife) and I'm cleaning the index every time. (> splunk clean eventdata) I got it to work with auto field extraction by inserting my own header line, but the issue there is that the header line is included in the count, and if I have 27,804 events I don't want it to say 27,805.

Re: Splunk insisting on Auto-finding CSV fields?

travispowell — Thu, 28 Apr 2011 22:39:35 GMT

Solved: I capitulated. Don't fight the beast. I ended up saying screw-it, splunk, you can auto-extract field names for me. But I wrote a REGEX rule that pointed to nullQueue to remove the first line.

See here:

http://www.splunk.com/support/forum:SplunkAdministration/4081

Re: Splunk insisting on Auto-finding CSV fields?

gkanapathy — Fri, 29 Apr 2011 02:48:06 GMT

You would need to clean out the etc/apps/learned/props.conf file, and reindex the data.