topic Re: Data Not Showing Up In Distributed Search Environment in Getting Data In

Data Not Showing Up In Distributed Search Environment

sk049q — Tue, 29 Sep 2020 22:52:37 GMT

Environment has one search head and one search peer. Data is sent to a directory [item (1)] configured to be monitored and indexed by the search peer. Both the search head and search peer have the same "indexes.conf" entry for the index [see item 21)], and the index is showing up in the search head GUI. Search peer has entry in "inputs.conf" to monitor the directory where data is being sent [see item (3)]. When a file is copied into the directory, the expected behavior is for the file to be ingested into Splunk and consequently be searchable; however this behavior is not occurring.

We have other indexes on this environment that do work as intended, but for some reason this particular setup is not working. Any and all help would be appreciated.

Item (1)*
/my/file/dir_ectory

Item (2)
[MY_in_dex]
homePath = $SPLUNK_DB/MY_in_dex
thawedPath = $SPLUNK_DB/thawedpath/MY_in_dex
coldPath = $SPLUNK_DB/coldpath/MY_in_dex

Item (3)
[monitor:///my/FILE/dir_ectory]
index = My_in_dex

*[NOTE: this traversal does start from "/" on a *nix machine]

Re: Data Not Showing Up In Distributed Search Environment

dkeck — Sat, 19 Jan 2019 18:11:32 GMT

HI,

did you check the splunkd log on forwarder and Indexer??

You should have logs from the forwarder which stats that its trying to read the file which should be monitored.
If not you should check if it picked up the inputs.conf at all with btool. Maybe you did not restart the forwarder after deploying the inputs.conf to it?

In which order did you create the input and the index? somethimes if you deploy the input before creating the index the forwarder tries to send logs and its not getting it written to the index, so in the fishbucket its saved that it read it but it wasn´t indexed.

Maybe a permission issue? Check what indexes you are allowed to search

Or your indexer is not excepting the data being sent by the forwarder? You could check the _internal for tcpin connections from the forwarder to the indexer

Re: Data Not Showing Up In Distributed Search Environment

dkeck — Sat, 19 Jan 2019 18:41:31 GMT

You can also elevated the log level:

On the Splunk instance that is monitoring the files, navigate to the $SPLUNK_HOME/etc directory and edit the file:
log.cfg

modify the following settings and change INFO to DEBUG

category.TailingProcessor=INFO
category.WatchedFile=INFO
category.ArchiveProcessor=INFO
category.TailReader=INFO

save the file.

Restart the Splunk instance.

Take a look at the log: $SPLUNK_HOME/var/log/splunk/splunkd.log

Look for the names of the files you were monitoring, the debug information should tell you why they were skipped.
Set the values back to INFO after you figure out your problem.
source: https://answers.splunk.com/answers/439546/how-to-troubleshoot-why-monitored-files-in-a-direc.html

Re: Data Not Showing Up In Distributed Search Environment

Vijeta — Sun, 20 Jan 2019 18:03:24 GMT

What role do you have, you can check the indexes under your role to which you have access. Probably this is not listed.

Re: Data Not Showing Up In Distributed Search Environment

woodcock — Mon, 21 Jan 2019 19:50:06 GMT

Your indexes.conf is highly unusual (but I don't see why it shouldn't work); why is it not like this:

homePath = $SPLUNK_DB/MY_in_dex/db
coldPath = $SPLUNK_DB/MY_in_dex/colddb
thawedPath = $SPLUNK_DB/MY_in_dex/thaweddb

Re: Data Not Showing Up In Distributed Search Environment

sk049q — Thu, 07 Feb 2019 17:59:10 GMT

Thanks to everyone for their help!

It ended up being a latency issue, where the data too up to an hour to be ingested.