topic Re: Forwarder stops forwarding in Getting Data In

Forwarder stops forwarding

parth_jec — Mon, 25 Jun 2012 17:10:00 GMT

One of our forwarders is monitoring three logs. Few hours back the forwarder stopped forwarding one of the three logs to the indexer, the others two logs monitored by the same forwarder are forwarded to the indexer properly. I have checked that the log file which is not forwarded is being updated continuously in real time.

I am using Universal forwarder (splunkforwarder-4.3.2-123586-x64-release) to forward logs to indexer (version 4.2.4, build 110225 ). The forwarder is installe on windows server and the indexer on linux.

At times forwarder stops forwarding the logs when I try to make some change the monitored log file path in the forwarder and restart it. Afterwards even if I revert the monitored log file path to the previously working path and restart the forwarder it does not forward the logs and after many hours the logs are suddenly forwarded to the indexer.

Can the version mismatch between forwarder and indexer be the problem ? or is there anything else which I should be taknig care of ?

Thanks,

Re: Forwarder stops forwarding

kristian_kolb — Mon, 25 Jun 2012 18:25:45 GMT

You can always poll the REST API on the forwarder to see what's happening when it fails to send you files.

https://your-forwarder-ip:8089/services/admin/inputstatus/TailingProcessor:FileStatus

You need to authenticate to get access. Unless you've changed the admin password on the forwarder, you should be able to log on with admin/changeme.

Apart from the files that Splunk monitors about itself, you should see your own three files if you scroll down a bit (with some status like '100 % read' or 'file not found' or 'permission denied').

Other things to check include:

How do you know it is not sending you data all the time? It sounds like a stupid question, but if Splunk misinterprets your timestamps the events will be indexed in the "wrong time" (and will not turn up when you search 'last 15 minutes'). Make a search for 'all time' or run a metadata search when the problem occurs; if the lastTime value differs from the recentTime this might indicate that the parsing of timestamps are wrong. Read more on metadata here and on troubleshooting inputs in general here.

Could this have anything to do with file rotation?
Take a look at the inputs.conf documentation and see if you might want to add alwaysOpenFile or crcSalt.

And of course, look in the splunkd.log on both forwarder and indexer for any interesting error messages.

I do not think that the mismatch in version numbers is significant here.

Hope this helps,

Kristian

Re: Forwarder stops forwarding

parth_jec — Mon, 25 Jun 2012 20:21:12 GMT

I havent changed the password but I cannot login with the default one. I tried resetting the default admin password by following http://splunk-base.splunk.com/answers/834/how-could-i-reset-the-admin-password but I am still not able to login.
I am using the command '%SPLUNK_HOME%\bin\splunk login -auth admin:changeme' to login.

Re: Forwarder stops forwarding

parth_jec — Mon, 25 Jun 2012 20:32:38 GMT

The problem occurs when I change the log file path in the forwarder. If splunk is misinterpreting my timestamps than shouldn't no events be forwarded at all ?
Also, I am using crcSalt= in my inputs.conf.

Re: Forwarder stops forwarding

kristian_kolb — Tue, 26 Jun 2012 07:01:55 GMT

Ok,
1) login. Point your browser to the splunkd port on the forwarder (see URL above). You'll get a login dialog box. Type admin/changeme. Beware of any firewalls that might block your access.

2) crcSalt. I believe it should be , not .

3). Post your inputs.conf (from your forwarder). And perhaps some more info on the files your trying to monitor.

Re: Forwarder stops forwarding

parth_jec — Tue, 26 Jun 2012 13:56:43 GMT

This is the fromat of my inputs.conf:

[default]
host =

[monitor://]
whitelist=$
index=
crcSalt=
1). I checked there seems to be no issue with firewall. I unable to login to the forwarder with the URL you mentioned.

2). I am not sure what do you mean by ", not ."

3).The inputs.conf of the forwarder looks like:

[monitor://]
whitelist=$
index=
crcSalt=

[monitor://path3]
index=index3
crcSalt=

Re: Forwarder stops forwarding

kristian_kolb — Tue, 26 Jun 2012 14:18:26 GMT

Did you actually edit the link to use your IP-address, or just click the link?
It should be uppercase: 'SOURCE' not 'source'

Re: Forwarder stops forwarding

parth_jec — Tue, 26 Jun 2012 14:28:13 GMT

Yes, I edited the link first then I got a prompt "The server at /splunk requires a username and password." I tried logging in with admin/changename, but it failed.

Also, I am using 'SOURCE' not 'source' in my inputs.config.

Thanks,

Re: Forwarder stops forwarding

kristian_kolb — Tue, 26 Jun 2012 14:31:45 GMT

admin/changeme

NOT

admin/changename

Re: Forwarder stops forwarding

parth_jec — Tue, 26 Jun 2012 14:34:45 GMT

yes tried that. not working.

Re: Forwarder stops forwarding

kristian_kolb — Tue, 26 Jun 2012 14:40:36 GMT

Then you either have changed the default password from changeme to something else. Ohh.. btw, you may not be able to do that remotely...come to think of it. If you go sit at the machine running the forwarder and do the same thing but use localhost as the ip (127.0.0.1) you should be able to do it.

Re: Forwarder stops forwarding

kristian_kolb — Tue, 26 Jun 2012 14:41:53 GMT

please use the comments instead of posting new answers as well.

Re: Forwarder stops forwarding

parth_jec — Tue, 26 Jun 2012 15:07:18 GMT

I havent changed the default password. I tried to reset the password by renaming the password file also.

I tried logging in with this link on the forwarder machine: https://127.0.0.1:8089/services/admin/inputstatus/TailingProcessor:FileStatus, didn't work either.

Can there be a port other than 8089 or this port is fixed ?

sorry, will use comments from now on.

Re: Forwarder stops forwarding

kristian_kolb — Tue, 26 Jun 2012 15:18:10 GMT

Ok - how does it not work? Do you get an error message of some sort? In that case - which?

Re: Forwarder stops forwarding

parth_jec — Tue, 26 Jun 2012 17:04:20 GMT

After multiple unsuccessful attempts I get the following message on the brower:
"401 Unauthorized."

Is there any log file in the forwarder which I can look into for this?

Can there be a port other than 8089 in the link or this port is fixed ?

Re: Forwarder stops forwarding

parth_jec — Tue, 26 Jun 2012 17:24:46 GMT

Also, I have installed multiple forwarders each on a different machine, but I cannot login to any of them.

Re: Forwarder stops forwarding

kristian_kolb — Thu, 28 Jun 2012 07:18:52 GMT

Ok, well, let's leave the REST api aside.

Have you looked at the splunkd.log file that is generated on the forwarder? It should be located in c:\program files\splunkuniversalforwarder\var\log\splunk.

Restart the forwarder and make note of the time. Look for any interesting errors after the restart, containing the filename you are looking for.

Also try to find the lines looking like:

TailingProcessor - Parsing configuration stanza:

WatchedFile - Will begin reading at offset

Re: Forwarder stops forwarding

parth_jec — Wed, 18 Jul 2012 13:07:44 GMT

The timestamp format was incorrect. The 12 hour format didn't had the AM/PM associated with the time so everyday after 12:59 the logs were not forwarded.

Thanks,