topic Re: Events from new index are not showing up in Splunk Search

Events from new index are not showing up

kjohnsonzenimax — Thu, 04 Oct 2012 12:33:12 GMT

I have inherited a fairly undocumented splunk deployment which looks as follows (splunk 4.3.2):

Forwarders -> 2x Heavy Forwarders -> 3x Indexers -> Search Head

I have added an index to the Search Head via the web interface and installed two forwarders with an inputs.conf as below:

[monitor:////opt/tld/glassfish/domains/tldcs/logs/feedback.log]
sourcetype = tld_gameplay
index = tld_gameplay

[monitor:////home/tldcs/web/apps/cstools/log/production.log]
index = customer_service

[monitor:////opt/tld/glassfish/domains/tldcs/logs/server.log]
index = customer_service

[monitor:////opt/tld/glassfish/domains/tldcs/logs/services.log]
index = customer_service

The issue is that I am not seeing any events in the web interface.

How can I debug this? What information do you need from me, so that I can help you? How can I verify that data is, or is not, even being received by the heavy forwarder, and then the indexers?

I am unclear whether I need to "add" the index somewhere else other than via the web interface.

Thanks,

Re: Events from new index are not showing up

sowings — Thu, 04 Oct 2012 20:06:58 GMT

Adding the index on the web interface adds it to the search head's local filesystem. It doesn't add it on the indexers themselves. In a lot of cases, the web interface of the indexers is turned off, to save memory as it commonly isn't used in this kind of distributed environment. I'd suggest first going to the indexers (command line is OK) and issuing splunk list indexes to see if your indexers have this new one.

You could then add them directly to the indexes.conf, or temporarily spin up the Splunk UI on the indexers to be able to use the UI to add the index.

Re: Events from new index are not showing up

kjohnsonzenimax — Wed, 10 Oct 2012 17:24:08 GMT

Hey, thanks for your answer. I have just checked the three indexers, running the command 'splunk list index' and have verified that the new index is listed on all three indexers.

Re: Events from new index are not showing up

kjohnsonzenimax — Wed, 10 Oct 2012 17:27:41 GMT

All that I had to do was restart the individual indexers, which the heavy forwarder was reporting to. After doing this, events began showing up in the search.