Splunk not recording events on hosts

[pfcnetwork]

Hello,

I am running the free version of Splunk for syslog (v 5.0.3) and after a server reboot last week, it is no longer recording events for any of my hosts that I can see in the search index (I have 10 or so).

I have tried restarting the splunkd and splunkweb services, but no luck.

Any suggestions would be approeciated

Cheers

[linu1988]

Free version how many days have passed? is it like you are now not able to search them?

[lukejadamec]

Here is a little Splunk magic that might work....

Put this stanza in the etc/system/local/inputs.conf file on the Indexer, and restart splunkd.

[splunktcp://9997]
Connection_host = none

[pfcnetwork]

Our hosts are either on the same LAN or connected via site-to-site IPSec VPN which forwards all traffic through the FW; it's not checked against the access lists.

[lukejadamec]

Do you have a firewall blocking traffic on UDP514?
Is the input still configured for the syslog input?

[pfcnetwork]

UDP port 514 is not in use.

[lukejadamec]

If these are syslog inputs, then on the indexer you might see errors in the splunkd log. If they are syslog inputs then you probably don't have forwarders installed on the hosts that are not reporting. If all of the hosts went offline at the same time, there are no errors in the splunkd log, and you still have the syslog input active, then it sounds like a problem with the syslog port on your indexer UDP 514. From a command line run netstat and look for port 514 to see if it is in use.

[pfcnetwork]

Sorry - I'm rather new to Splunk and not sure what you are referring to. There's only a few lines in the log file with 'forwarder' in them and they all read like this:

02-25-2014 14:37:19.985 -0700 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder

[lukejadamec]

This is a syslog input? Is the port in use by something else?

[linu1988]

please go and check in forwarder splunkd.log why it's not forwarding rather than the search head

[pfcnetwork]

On the main Summary page where you can select a host from the 'Hosts' list. Or doing a 'host=name/IP' search yields nothing beyond Feb 18th.

In the Splunkd log file, what exactly should I be looking for? There's nothing that explictly says 'error'

[lukejadamec]

Are you sure the data is not getting indexed?
What kind of search are you running?
Are there errors in the splunkd log?

[pfcnetwork]

Unfortunately not. Any further suggestions?

[lukejadamec]

did it work?

[pfcnetwork]

Corrections made, thanks

[lukejadamec]

Actually, let me check the last line....

I believe it should read:

`[default]
host = MGTNMS100

[splunktcp://9997]
Connection_host = none`

[pfcnetwork]

Thanks lukejadamec

I now have the inputs.conf looking like this:

[default]
[splunktcp://9997] Connection_host = none
host = MGTNMS100

I will let you know how it goes