topic Re: LightForwarder, Not sending updated log entries in Getting Data In

LightForwarder, Not sending updated log entries

drewbfl — Wed, 15 Sep 2010 00:57:37 GMT

Hi,
Have a lightforwarder configured to send updated entries from /mnt/nagios/nagios.log on 10.1.1.1. It looks like there was an initial load into the search app (42k events) and it hasn't updated in 5 days. Also interesting is that on stop/start it shows parsing configuration for the file, but never states "Will begin reading". The log itself is being updated every couple minutes and shows an updated timestamp on 10.1.1.1. Permissions are open to 755. Syslog is being sent and properly updated to our splunk instance. I also have nagios events logged to syslog and those are appearing (just in-case this sorta thing were to happen). but, I would really like to disable that and have the log with the sep. index and sourcetype be logged from the proper log.

FORWARDER:
./splunk list monitor
Monitored Files:
/mnt/nagios/nagios.log

inputs.conf in search/local:
[monitor:///mnt/nagios/nagios.log]
disabled = false
host = nagios.blah.blah.com
sourcetype = nagios
index = nagios

outputs.conf in search/local:
[tcpout]
defaultGroup = 10.1.1.1_514
disabled = false

[tcpout:10.1.1.1_514]
server = 10.1.1.1:514

[tcpout-server://10.1.1.1:514]

stop/start log:
9-14-2010 17:50:32.263 INFO loader - Server supporting SSL v2/v3
09-14-2010 17:50:32.263 INFO loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
09-14-2010 17:50:32.272 INFO TPool - initializing BatchReaderTPool with 1 workers
09-14-2010 17:50:32.272 INFO TcpOutputProc - attempting to connect to 10.1.1.1:514...
09-14-2010 17:50:32.273 INFO TcpOutputProc - Connected to 10.1.1.1:514
09-14-2010 17:50:33.513 INFO TailingProcessor - TailWatcher initializing...
09-14-2010 17:50:33.543 INFO TailingProcessor - Parsing configuration stanza: monitor:///mnt/nagios/nagios.log.
09-14-2010 17:50:33.544 INFO WatchedFile - Will begin reading at offset=7600309 for file='/mnt/nagios/nagios.log'.
09-14-2010 17:50:53.056 INFO timeinvertedIndex - starting loggerPipe eloop
09-14-2010 17:50:53.056 INFO timeinvertedIndex - running loggerPipe eloop

INDEXER:
inputs.conf in search/local:
[splunktcp://514]

inputs.conf in system/local:
[default] host = splunk.blah.blah.com

Re: LightForwarder, Not sending updated log entries

hulahoop — Wed, 15 Sep 2010 01:08:11 GMT

Did you happen to enable LWF in the last 5 days/since setting up the forwarder? The index parameter in inputs.conf on a LWF is not honored. It needs to be a regular forwarding if you want to perform routing to an index other than the default.

Thanks for updating your description. Can you try adding this to the inputs.conf on the indexer?

[monitor:///mnt/nagios/nagios.log]
index = nagios

Also, did you try enabling "index and forward" on the forwarder to ensure that data is indeed getting indexed and to the correct index? Then we can rule out any input config issues.

Re: LightForwarder, Not sending updated log entries

drewbfl — Wed, 15 Sep 2010 01:52:07 GMT

enabled SplunkForwarder.
stoppped.
started.

still no luck.

Re: LightForwarder, Not sending updated log entries

hulahoop — Wed, 15 Sep 2010 02:12:57 GMT

is index=nagios created on the indexer?

Re: LightForwarder, Not sending updated log entries

hulahoop — Wed, 15 Sep 2010 02:22:03 GMT

if it is, then try enabling local indexing on the forwarder to ensure there is nothing wrong with the input config. you'll probably have to create the nagios index temporarily on the forwarder.

Re: LightForwarder, Not sending updated log entries

hulahoop — Wed, 15 Sep 2010 02:26:26 GMT

i should also note, if you want to use the LWF, then i believe you can put the index=nagios setting on the indexer.

Re: LightForwarder, Not sending updated log entries

drewbfl — Wed, 15 Sep 2010 03:40:13 GMT

it is on the indexer. interestingly, the latest event in the nagios index is accurate. it must be pulling that from the syslog source. the source and sourcetype on the main search app still have the stale numbers.

Re: LightForwarder, Not sending updated log entries

hulahoop — Wed, 15 Sep 2010 04:41:25 GMT

would you please update your question with inputs.conf from forwarder and indexer?

Re: LightForwarder, Not sending updated log entries

drewbfl — Wed, 15 Sep 2010 05:24:12 GMT

i added it above. thanks

Re: LightForwarder, Not sending updated log entries

drewbfl — Thu, 16 Sep 2010 04:37:01 GMT

Didn't help. I tried adding it to both system/local and search/local inputs.confs and it didn't help.

Re: LightForwarder, Not sending updated log entries

hulahoop — Thu, 16 Sep 2010 07:17:23 GMT

I'm sorry these steps haven't produced any different results for you. Have you tried enabling "index and forward" on the forwarder? If that does not produce the correct result, then I would recommend opening a ticket with the Splunk support team to have your configuration files reviewed in detail.

Re: LightForwarder, Not sending updated log entries

drewbfl — Fri, 17 Sep 2010 21:15:16 GMT

I really don't want the forwarder to do any indexing, it doesn't have the cycles nor should it need to. Isn't this a common thing everyone does with the product?

Re: LightForwarder, Not sending updated log entries

hulahoop — Tue, 21 Sep 2010 00:04:12 GMT

I just mean to enable it for debugging purposes.

Re: LightForwarder, Not sending updated log entries

gkanapathy — Thu, 23 Dec 2010 07:29:01 GMT

using index= in inputs.conf on LWF does work, and should work, and is the preferred way to set an index when using a LWF. What does not work is routing to an index via transforms.