Forwarder stops forwarding

[parth_jec]

One of our forwarders is monitoring three logs. Few hours back the forwarder stopped forwarding one of the three logs to the indexer, the others two logs monitored by the same forwarder are forwarded to the indexer properly. I have checked that the log file which is not forwarded is being updated continuously in real time.

I am using Universal forwarder (splunkforwarder-4.3.2-123586-x64-release) to forward logs to indexer (version 4.2.4, build 110225 ). The forwarder is installe on windows server and the indexer on linux.

At times forwarder stops forwarding the logs when I try to make some change the monitored log file path in the forwarder and restart it. Afterwards even if I revert the monitored log file path to the previously working path and restart the forwarder it does not forward the logs and after many hours the logs are suddenly forwarded to the indexer.

Can the version mismatch between forwarder and indexer be the problem ? or is there anything else which I should be taknig care of ?

Thanks,

[parth_jec · Accepted solution]

The timestamp format was incorrect. The 12 hour format didn't had the AM/PM associated with the time so everyday after 12:59 the logs were not forwarded.

Thanks,

[kristian_kolb]

You can always poll the REST API on the forwarder to see what's happening when it fails to send you files.

https://your-forwarder-ip:8089/services/admin/inputstatus/TailingProcessor:FileStatus

You need to authenticate to get access. Unless you've changed the admin password on the forwarder, you should be able to log on with admin/changeme.

Apart from the files that Splunk monitors about itself, you should see your own three files if you scroll down a bit (with some status like '100 % read' or 'file not found' or 'permission denied').

Other things to check include:

How do you know it is not sending you data all the time? It sounds like a stupid question, but if Splunk misinterprets your timestamps the events will be indexed in the "wrong time" (and will not turn up when you search 'last 15 minutes'). Make a search for 'all time' or run a metadata search when the problem occurs; if the lastTime value differs from the recentTime this might indicate that the parsing of timestamps are wrong. Read more on metadata here and on troubleshooting inputs in general here.

Could this have anything to do with file rotation?
Take a look at the inputs.conf documentation and see if you might want to add alwaysOpenFile or crcSalt.

And of course, look in the splunkd.log on both forwarder and indexer for any interesting error messages.

I do not think that the mismatch in version numbers is significant here.

Hope this helps,

Kristian

[kristian_kolb]

Ok, well, let's leave the REST api aside.

Have you looked at the splunkd.log file that is generated on the forwarder? It should be located in c:\program files\splunkuniversalforwarder\var\log\splunk.

Restart the forwarder and make note of the time. Look for any interesting errors after the restart, containing the filename you are looking for.

Also try to find the lines looking like:

TailingProcessor - Parsing configuration stanza:

OR

WatchedFile - Will begin reading at offset

/k

[parth_jec]

Also, I have installed multiple forwarders each on a different machine, but I cannot login to any of them.

[parth_jec]

After multiple unsuccessful attempts I get the following message on the brower:
"401 Unauthorized."

Is there any log file in the forwarder which I can look into for this?

Can there be a port other than 8089 in the link or this port is fixed ?

[kristian_kolb]

Ok - how does it not work? Do you get an error message of some sort? In that case - which?

/k

[parth_jec]

I havent changed the default password. I tried to reset the password by renaming the password file also.

I tried logging in with this link on the forwarder machine: https://127.0.0.1:8089/services/admin/inputstatus/TailingProcessor:FileStatus, didn't work either.

Can there be a port other than 8089 or this port is fixed ?

sorry, will use comments from now on.

[kristian_kolb]

please use the comments instead of posting new answers as well.

[kristian_kolb]

Then you either have changed the default password from changeme to something else. Ohh.. btw, you may not be able to do that remotely...come to think of it. If you go sit at the machine running the forwarder and do the same thing but use localhost as the ip (127.0.0.1) you should be able to do it.

[parth_jec]

yes tried that. not working.

[kristian_kolb]

admin/changeme

NOT

admin/changename

[parth_jec]

Yes, I edited the link first then I got a prompt "The server at /splunk requires a username and password." I tried logging in with admin/changename, but it failed.

Also, I am using 'SOURCE' not 'source' in my inputs.config.

Thanks,

[kristian_kolb]

1. Did you actually edit the link to use your IP-address, or just click the link?

2. It should be uppercase: 'SOURCE' not 'source'

[parth_jec]

This is the fromat of my inputs.conf:

[default]
host =

[monitor://]
whitelist=$
index=
crcSalt=
1). I checked there seems to be no issue with firewall. I unable to login to the forwarder with the URL you mentioned.

2). I am not sure what do you mean by ", not ."

3).The inputs.conf of the forwarder looks like:

[monitor://]
whitelist=$
index=
crcSalt=

[monitor://path3]
index=index3
crcSalt=

[kristian_kolb]

Ok,
1) login. Point your browser to the splunkd port on the forwarder (see URL above). You'll get a login dialog box. Type admin/changeme. Beware of any firewalls that might block your access.

2) crcSalt. I believe it should be , not .

3). Post your inputs.conf (from your forwarder). And perhaps some more info on the files your trying to monitor.

/k

[parth_jec]

The problem occurs when I change the log file path in the forwarder. If splunk is misinterpreting my timestamps than shouldn't no events be forwarded at all ?
Also, I am using crcSalt= in my inputs.conf.

[parth_jec]

I havent changed the password but I cannot login with the default one. I tried resetting the default admin password by following http://splunk-base.splunk.com/answers/834/how-could-i-reset-the-admin-password but I am still not able to login.
I am using the command '%SPLUNK_HOME%\bin\splunk login -auth admin:changeme' to login.