Getting Data In
Highlighted

Why are interesting fields missing after importing a large CSV file with 100 fields?

Explorer

I am importing a CSV with around 100 fields. When importing, I see the review screen and it shows all of the fields and values in the file in a nicely formatted table. Once I submit and go to the search are with the new data source select, I only see around 20 fields in the "interesting fields" list and another 11 in the "11 more fields" link. Also, when I am typing in the search bar, it is autocompleting with fields that are missing, but when I submit the search, it finds no records.

How can I choose what fields I see in that search list? is there a limit to the number of fields the CSV can have?

Highlighted

Re: Why are interesting fields missing after importing a large CSV file with 100 fields?

SplunkTrust
SplunkTrust

If you're only seeing around 30 fields in total then you're not hitting any potential field count limits.

Are all fields present in all events?
What's in props.conf under that sourcetype's stanza?

0 Karma
Highlighted

Re: Why are interesting fields missing after importing a large CSV file with 100 fields?

Explorer

No, all fields are not present in all events, they are only present part of the time. My props.conf looks like this:
(#)your settings
NOBINARYCHECK=1

(#)set by detected source type
INDEXEDEXTRACTIONS=csv
KV
MODE=none
SHOULDLINEMERGE=false
pulldown
type=true

0 Karma
Highlighted

Re: Why are interesting fields missing after importing a large CSV file with 100 fields?

SplunkTrust
SplunkTrust

If most fields are very rare then check they're not being filtered from that view. Click "All Fields" or the "11 more fields" link and see if changing the Coverage changes the number of fields you see.

If that doesn't change anything, post some sample data - maybe there's some oddities in there that stop fields from being extracted.

0 Karma
Highlighted

Re: Why are interesting fields missing after importing a large CSV file with 100 fields?

Explorer

I can't paste in any sample data here because the limit is to short, but I just generated a bunch of sample data through Mockaroo with 100 fields and 3000 rows with only colors as the content and I still had the same problem. It only listed some of the fields in the interesting fields column.

0 Karma
Highlighted

Re: Why are interesting fields missing after importing a large CSV file with 100 fields?

SplunkTrust
SplunkTrust

There is a bug related to a large number of fields at http://answers.splunk.com/answers/129773/advice-for-when-you-have-more-than-100-automatically-extrac... but it shouldn't lead to you only seeing 30ish fields.

0 Karma
Highlighted

Re: Why are interesting fields missing after importing a large CSV file with 100 fields?

Explorer

After more testing, there appears to be a 50 field limit. Does that sound right? With the testing data, I number the fields 1-100 and it had every field from 1-50 and then stopped.

0 Karma
Highlighted

Re: Why are interesting fields missing after importing a large CSV file with 100 fields?

SplunkTrust
SplunkTrust

Might be a restriction with the new INDEXED_EXTRACTIONS=csv feature. Do file a case with Splunk Support to be certain.

0 Karma
Highlighted

Re: Why are interesting fields missing after importing a large CSV file with 100 fields?

Explorer