Hi,
Have a lightforwarder configured to send updated entries from /mnt/nagios/nagios.log on 10.1.1.1. It looks like there was an initial load into the search app (42k events) and it hasn't updated in 5 days. Also interesting is that on stop/start it shows parsing configuration for the file, but never states "Will begin reading". The log itself is being updated every couple minutes and shows an updated timestamp on 10.1.1.1. Permissions are open to 755. Syslog is being sent and properly updated to our splunk instance. I also have nagios events logged to syslog and those are appearing (just in-case this sorta thing were to happen). but, I would really like to disable that and have the log with the sep. index and sourcetype be logged from the proper log.
FORWARDER:
./splunk list monitor
Monitored Files:
/mnt/nagios/nagios.log
inputs.conf in search/local:
[monitor:///mnt/nagios/nagios.log]
disabled = false
host = nagios.blah.blah.com
sourcetype = nagios
index = nagios
outputs.conf in search/local:
[tcpout]
defaultGroup = 10.1.1.1_514
disabled = false
[tcpout:10.1.1.1_514]
server = 10.1.1.1:514
[tcpout-server://10.1.1.1:514]
stop/start log:
9-14-2010 17:50:32.263 INFO loader - Server supporting SSL v2/v3
09-14-2010 17:50:32.263 INFO loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
09-14-2010 17:50:32.272 INFO TPool - initializing BatchReaderTPool with 1 workers
09-14-2010 17:50:32.272 INFO TcpOutputProc - attempting to connect to 10.1.1.1:514...
09-14-2010 17:50:32.273 INFO TcpOutputProc - Connected to 10.1.1.1:514
09-14-2010 17:50:33.513 INFO TailingProcessor - TailWatcher initializing...
09-14-2010 17:50:33.543 INFO TailingProcessor - Parsing configuration stanza: monitor:///mnt/nagios/nagios.log.
09-14-2010 17:50:33.544 INFO WatchedFile - Will begin reading at offset=7600309 for file='/mnt/nagios/nagios.log'.
09-14-2010 17:50:53.056 INFO timeinvertedIndex - starting loggerPipe eloop
09-14-2010 17:50:53.056 INFO timeinvertedIndex - running loggerPipe eloop
INDEXER:
inputs.conf in search/local:
[splunktcp://514]
inputs.conf in system/local:
[default]
host = splunk.blah.blah.com
Did you happen to enable LWF in the last 5 days/since setting up the forwarder? The index parameter in inputs.conf on a LWF is not honored. It needs to be a regular forwarding if you want to perform routing to an index other than the default.
Thanks for updating your description. Can you try adding this to the inputs.conf on the indexer?
[monitor:///mnt/nagios/nagios.log]
index = nagios
Also, did you try enabling "index and forward" on the forwarder to ensure that data is indeed getting indexed and to the correct index? Then we can rule out any input config issues.
using index=
in inputs.conf on LWF does work, and should work, and is the preferred way to set an index when using a LWF. What does not work is routing to an index via transforms.
I just mean to enable it for debugging purposes.
I really don't want the forwarder to do any indexing, it doesn't have the cycles nor should it need to. Isn't this a common thing everyone does with the product?
I'm sorry these steps haven't produced any different results for you. Have you tried enabling "index and forward" on the forwarder? If that does not produce the correct result, then I would recommend opening a ticket with the Splunk support team to have your configuration files reviewed in detail.
Didn't help. I tried adding it to both system/local and search/local inputs.confs and it didn't help.
i added it above. thanks
would you please update your question with inputs.conf from forwarder and indexer?
it is on the indexer. interestingly, the latest event in the nagios index is accurate. it must be pulling that from the syslog source. the source and sourcetype on the main search app still have the stale numbers.
i should also note, if you want to use the LWF, then i believe you can put the index=nagios setting on the indexer.
if it is, then try enabling local indexing on the forwarder to ensure there is nothing wrong with the input config. you'll probably have to create the nagios index temporarily on the forwarder.
is index=nagios created on the indexer?
enabled SplunkForwarder.
stoppped.
started.
still no luck.