Getting Data In

Why is splunk not indexing all the data?

eylonronen
Explorer

Hi, lately we've been checking how many files our splunk is indexing, and we noticed that it "skips" some files... We checked by searching:

index=our_index | stats dc(source)
index = _internal group = per_so* series=*.ourfile | stats count

And both ways we got the same results, which are not all the files we indexed.

0 Karma

ddrillic
Ultra Champion

A cheerful place to start at I can't find my data!

0 Karma

eylonronen
Explorer

I didn't find any help in this page....

0 Karma

somesoni2
Revered Legend

Also, check internal logs from yoru forwarder(s) to see if there are any warnings/error for your files. ( index=_itnernal sourcetype=splunkd host=yourFwd *filename.ext* )

0 Karma

eylonronen
Explorer

I've already looked there... Zero warnings or errors....
Also we've tried both monitor and batch input. Both had the same problem...

0 Karma

somesoni2
Revered Legend

Try running following on your forwarder instance. See if Splunk is monitoring all the files you've configured for monitoring (will prompt for admin credentials for that Splunk instance)

$SPLUNK_HOME/bin/splunk list monitor
0 Karma

somesoni2
Revered Legend

Even if there are no errors/warnings, do you see any entry for your log file that's missing?

index=_internal sourcetype=splunkd host=yourFwd  adding watch
0 Karma

eylonronen
Explorer

well the forwarder doesnt write log when it monitors, only with batch input for some reason. Today we indexed some logs, and we saw one of the files in the forwarders log, but we could not search it...
I wonder if it has something to do with the fact that we added a few indexers recently.
Is there something i should update in the search head when i add indexers?

0 Karma