topic Re: Splunk not recording events on hosts in Getting Data In

Splunk not recording events on hosts

pfcnetwork — Tue, 25 Feb 2014 20:33:58 GMT

Hello,

I am running the free version of Splunk for syslog (v 5.0.3) and after a server reboot last week, it is no longer recording events for any of my hosts that I can see in the search index (I have 10 or so).

I have tried restarting the splunkd and splunkweb services, but no luck.

Any suggestions would be approeciated

Cheers

Re: Splunk not recording events on hosts

lukejadamec — Tue, 25 Feb 2014 20:42:22 GMT

Here is a little Splunk magic that might work....

Put this stanza in the etc/system/local/inputs.conf file on the Indexer, and restart splunkd.

[splunktcp://9997] Connection_host = none

Re: Splunk not recording events on hosts

linu1988 — Tue, 25 Feb 2014 20:43:02 GMT

Free version how many days have passed? is it like you are now not able to search them?

Re: Splunk not recording events on hosts

pfcnetwork — Tue, 25 Feb 2014 20:55:33 GMT

Thanks lukejadamec

I now have the inputs.conf looking like this:

[default]
[splunktcp://9997] Connection_host = none
host = MGTNMS100

I will let you know how it goes

Re: Splunk not recording events on hosts

lukejadamec — Tue, 25 Feb 2014 20:57:43 GMT

Actually, let me check the last line....

I believe it should read:

`[default]
host = MGTNMS100

[splunktcp://9997]
Connection_host = none`

Re: Splunk not recording events on hosts

pfcnetwork — Tue, 25 Feb 2014 21:31:16 GMT

Corrections made, thanks

Re: Splunk not recording events on hosts

lukejadamec — Tue, 25 Feb 2014 22:43:59 GMT

did it work?

Re: Splunk not recording events on hosts

pfcnetwork — Wed, 26 Feb 2014 16:01:16 GMT

Unfortunately not. Any further suggestions?

Re: Splunk not recording events on hosts

lukejadamec — Wed, 26 Feb 2014 16:11:00 GMT

Are you sure the data is not getting indexed?
What kind of search are you running?
Are there errors in the splunkd log?

Re: Splunk not recording events on hosts

pfcnetwork — Wed, 26 Feb 2014 16:31:23 GMT

On the main Summary page where you can select a host from the 'Hosts' list. Or doing a 'host=name/IP' search yields nothing beyond Feb 18th.

In the Splunkd log file, what exactly should I be looking for? There's nothing that explictly says 'error'

Re: Splunk not recording events on hosts

linu1988 — Wed, 26 Feb 2014 17:46:00 GMT

please go and check in forwarder splunkd.log why it's not forwarding rather than the search head

Re: Splunk not recording events on hosts

lukejadamec — Wed, 26 Feb 2014 17:49:23 GMT

This is a syslog input? Is the port in use by something else?

Re: Splunk not recording events on hosts

pfcnetwork — Mon, 28 Sep 2020 15:59:00 GMT

Sorry - I'm rather new to Splunk and not sure what you are referring to. There's only a few lines in the log file with 'forwarder' in them and they all read like this:

02-25-2014 14:37:19.985 -0700 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder

Re: Splunk not recording events on hosts

lukejadamec — Wed, 26 Feb 2014 17:56:56 GMT

If these are syslog inputs, then on the indexer you might see errors in the splunkd log. If they are syslog inputs then you probably don't have forwarders installed on the hosts that are not reporting. If all of the hosts went offline at the same time, there are no errors in the splunkd log, and you still have the syslog input active, then it sounds like a problem with the syslog port on your indexer UDP 514. From a command line run netstat and look for port 514 to see if it is in use.

Re: Splunk not recording events on hosts

pfcnetwork — Wed, 26 Feb 2014 18:01:19 GMT

UDP port 514 is not in use.

Re: Splunk not recording events on hosts

lukejadamec — Wed, 26 Feb 2014 18:11:27 GMT

Do you have a firewall blocking traffic on UDP514?
Is the input still configured for the syslog input?

Re: Splunk not recording events on hosts

pfcnetwork — Wed, 26 Feb 2014 20:19:34 GMT

Our hosts are either on the same LAN or connected via site-to-site IPSec VPN which forwards all traffic through the FW; it's not checked against the access lists.