Getting Data In

Why are only some (91 out of 1300) JSON files from the same folder location being indexed?

jrodsecurity
New Member

Splunk Version 6.1.2 and Splunk 6.2.0
I have created a Data Inputs folder with roughly 1300 small JSON files in it. When Splunk Indexes the data it only grabs 91 of the files to index then no other files get indexed
I have created the proper Index for this with the proper Data Input.
I put the folder with the files locally on the same Splunk Instance to take the Forwarder out of the equation. At this point im at a bit of a loss as to why some not all the files get indexed.

Path:/Applications/Splunk/outgoing

sourcetype: _json
index: default

Number of Files: 1377

Tags (2)
0 Karma

david_rundle_fi
Explorer

I've also just encountered this same issue - 838 JSON files in target dir, less than 16 MB on disk, but only 108 indexed. Also happens from a forwarder to search head.

0 Karma

emiller42
Motivator

Is this just a group of files that need to be indexed once, or are they being updated and must be tailed? If its the former and you're using typical 'monitor' stanzas, you might be running into some bottlenecks with the number of open file handles.

If this is just historical data you need to pull in, you might want to look into using a 'batch' input. (details in inputs.conf) This reads the file once, then deletes it, instead of constantly watching it for updates.

I've seen similar behavior when attempting to index IIS servers which have no log retention policies, and Splunk tries to watch multiple years worth of iis log files.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>