topic Re: Splunk not indexing new hosts in Getting Data In

Splunk not indexing new hosts

ptierney — Fri, 08 Nov 2013 17:34:13 GMT

I've added new servers that are pushing their syslogs to my splunk host. The messages are getting properly routed to /var/log/messages. If I tail the file, I see my new hosts appearing in the messages file, but they don't appear in a splunk search. Existing hosts from the same file are getting updated with new messages. What am I missing here?

Re: Splunk not indexing new hosts

jtrucks — Sat, 09 Nov 2013 00:36:00 GMT

How is that source configured in Splunk? Have you restarted Splunk to ensure it's operating correctly?

Re: Splunk not indexing new hosts

rsennett_splunk — Sat, 09 Nov 2013 00:51:13 GMT

I would imagine that if you have added another host to the file but it isn't appearing in Splunk, that the Splunk config has some very specific Routing and Filtering somewhere that is handling the existing hosts one at a time.

And your new hostname isn't matching any of the existing patterns,so it isn't getting sent anywhere. Probably in order to route them to particular sourcetypes to then have fields extracted, etc. $SPLUNK_HOME/etc/system/local/props.conf and trasnsforms.conf

Re: Splunk not indexing new hosts

yuvalba — Sat, 09 Nov 2013 18:02:38 GMT

What search are you using?

It could be that the messages format from this new host are different and the host name is not prased correctly by Splunk.

Do you see the messages if you search only based on the source:
source=/var/log/messages

Re: Splunk not indexing new hosts

rsennett_splunk — Sat, 09 Nov 2013 18:10:07 GMT

Oh, very good yuvalba... I should have said that to. 🙂

Re: Splunk not indexing new hosts

yannK — Sun, 10 Nov 2013 18:41:38 GMT

If your sourcetype is syslog, then the host field will be extracted from the event (not from the splunk forwarder host name)

verify with
index=/var/log/* | stats values(sourcetype) count by host

Re: Splunk not indexing new hosts

rsennett_splunk — Sun, 10 Nov 2013 19:11:05 GMT

And I reiterate... if you are filtering the sourcetype syslog with specific routing by host you are going to have to account for the new ones...
But you should see them with yannK's suggestion, unless there is a very strict routing and filtering happening where non matching things get tossed. Just something to consider if all logical attempts to see them fail... start looking for the possibility that there are other steps required to add new hosts. Especially if there is more than one admin at your installation...