Getting Data In

LightForwarder, Not sending updated log entries

Path Finder

Hi,
Have a lightforwarder configured to send updated entries from /mnt/nagios/nagios.log on 10.1.1.1. It looks like there was an initial load into the search app (42k events) and it hasn't updated in 5 days. Also interesting is that on stop/start it shows parsing configuration for the file, but never states "Will begin reading". The log itself is being updated every couple minutes and shows an updated timestamp on 10.1.1.1. Permissions are open to 755. Syslog is being sent and properly updated to our splunk instance. I also have nagios events logged to syslog and those are appearing (just in-case this sorta thing were to happen). but, I would really like to disable that and have the log with the sep. index and sourcetype be logged from the proper log.

FORWARDER:
./splunk list monitor
Monitored Files:
/mnt/nagios/nagios.log

inputs.conf in search/local:
[monitor:///mnt/nagios/nagios.log]
disabled = false
host = nagios.blah.blah.com
sourcetype = nagios
index = nagios

outputs.conf in search/local:
[tcpout]
defaultGroup = 10.1.1.1_514
disabled = false

[tcpout:10.1.1.1_514]
server = 10.1.1.1:514

[tcpout-server://10.1.1.1:514]

stop/start log:
9-14-2010 17:50:32.263 INFO loader - Server supporting SSL v2/v3
09-14-2010 17:50:32.263 INFO loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
09-14-2010 17:50:32.272 INFO TPool - initializing BatchReaderTPool with 1 workers
09-14-2010 17:50:32.272 INFO TcpOutputProc - attempting to connect to 10.1.1.1:514...
09-14-2010 17:50:32.273 INFO TcpOutputProc - Connected to 10.1.1.1:514
09-14-2010 17:50:33.513 INFO TailingProcessor - TailWatcher initializing...
09-14-2010 17:50:33.543 INFO TailingProcessor - Parsing configuration stanza: monitor:///mnt/nagios/nagios.log.
09-14-2010 17:50:33.544 INFO WatchedFile - Will begin reading at offset=7600309 for file='/mnt/nagios/nagios.log'.
09-14-2010 17:50:53.056 INFO timeinvertedIndex - starting loggerPipe eloop
09-14-2010 17:50:53.056 INFO timeinvertedIndex - running loggerPipe eloop

INDEXER:
inputs.conf in search/local:
[splunktcp://514]

inputs.conf in system/local:
[default] host = splunk.blah.blah.com

0 Karma

Splunk Employee
Splunk Employee

Did you happen to enable LWF in the last 5 days/since setting up the forwarder? The index parameter in inputs.conf on a LWF is not honored. It needs to be a regular forwarding if you want to perform routing to an index other than the default.

Thanks for updating your description. Can you try adding this to the inputs.conf on the indexer?

[monitor:///mnt/nagios/nagios.log]
index = nagios

Also, did you try enabling "index and forward" on the forwarder to ensure that data is indeed getting indexed and to the correct index? Then we can rule out any input config issues.

0 Karma

Splunk Employee
Splunk Employee

using index= in inputs.conf on LWF does work, and should work, and is the preferred way to set an index when using a LWF. What does not work is routing to an index via transforms.

0 Karma

Splunk Employee
Splunk Employee

I just mean to enable it for debugging purposes.

0 Karma

Path Finder

I really don't want the forwarder to do any indexing, it doesn't have the cycles nor should it need to. Isn't this a common thing everyone does with the product?

0 Karma

Splunk Employee
Splunk Employee

I'm sorry these steps haven't produced any different results for you. Have you tried enabling "index and forward" on the forwarder? If that does not produce the correct result, then I would recommend opening a ticket with the Splunk support team to have your configuration files reviewed in detail.

0 Karma

Path Finder

Didn't help. I tried adding it to both system/local and search/local inputs.confs and it didn't help.

0 Karma

Path Finder

i added it above. thanks

0 Karma

Splunk Employee
Splunk Employee

would you please update your question with inputs.conf from forwarder and indexer?

0 Karma

Path Finder

it is on the indexer. interestingly, the latest event in the nagios index is accurate. it must be pulling that from the syslog source. the source and sourcetype on the main search app still have the stale numbers.

0 Karma

Splunk Employee
Splunk Employee

i should also note, if you want to use the LWF, then i believe you can put the index=nagios setting on the indexer.

0 Karma

Splunk Employee
Splunk Employee

if it is, then try enabling local indexing on the forwarder to ensure there is nothing wrong with the input config. you'll probably have to create the nagios index temporarily on the forwarder.

0 Karma

Splunk Employee
Splunk Employee

is index=nagios created on the indexer?

0 Karma

Path Finder

enabled SplunkForwarder.
stoppped.
started.

still no luck.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!