Getting Data In

Why are interesting fields missing after importing a large CSV file with 100 fields?

Explorer

I am importing a CSV with around 100 fields. When importing, I see the review screen and it shows all of the fields and values in the file in a nicely formatted table. Once I submit and go to the search are with the new data source select, I only see around 20 fields in the "interesting fields" list and another 11 in the "11 more fields" link. Also, when I am typing in the search bar, it is autocompleting with fields that are missing, but when I submit the search, it finds no records.

How can I choose what fields I see in that search list? is there a limit to the number of fields the CSV can have?

1 Solution

SplunkTrust
SplunkTrust

Might be a restriction with the new INDEXED_EXTRACTIONS=csv feature. Do file a case with Splunk Support to be certain.

0 Karma

Explorer

After more testing, there appears to be a 50 field limit. Does that sound right? With the testing data, I number the fields 1-100 and it had every field from 1-50 and then stopped.

0 Karma

SplunkTrust
SplunkTrust

There is a bug related to a large number of fields at http://answers.splunk.com/answers/129773/advice-for-when-you-have-more-than-100-automatically-extrac... but it shouldn't lead to you only seeing 30ish fields.

0 Karma

Explorer

I can't paste in any sample data here because the limit is to short, but I just generated a bunch of sample data through Mockaroo with 100 fields and 3000 rows with only colors as the content and I still had the same problem. It only listed some of the fields in the interesting fields column.

0 Karma

SplunkTrust
SplunkTrust

If most fields are very rare then check they're not being filtered from that view. Click "All Fields" or the "11 more fields" link and see if changing the Coverage changes the number of fields you see.

If that doesn't change anything, post some sample data - maybe there's some oddities in there that stop fields from being extracted.

0 Karma

Explorer

No, all fields are not present in all events, they are only present part of the time. My props.conf looks like this:
(#)your settings
NO_BINARY_CHECK=1

(#)set by detected source type
INDEXED_EXTRACTIONS=csv
KV_MODE=none
SHOULD_LINEMERGE=false
pulldown_type=true

0 Karma

SplunkTrust
SplunkTrust

If you're only seeing around 30 fields in total then you're not hitting any potential field count limits.

Are all fields present in all events?
What's in props.conf under that sourcetype's stanza?

0 Karma