topic Re: Forwarder not forwarding for 6 hours every day. in Getting Data In

Forwarder not forwarding for 6 hours every day.

ssearwar — Mon, 10 Feb 2014 18:39:29 GMT

I'm running 6.01 server and forwarders. I've got a couple of linux servers that are not forwarding between a 6 hour period of time everyday for a particular log. Can someone point me in the right direction to correct this issue?

Re: Forwarder not forwarding for 6 hours every day.

somesoni2 — Mon, 10 Feb 2014 18:42:02 GMT

Check the _internal logs from forwarder if there is any error at that period.

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Mon, 10 Feb 2014 18:55:47 GMT

No errors for the log that I need:

02-10-2014 11:02:53.021 -0500 INFO WatchedFile - Will begin reading at offset=1236998 for file='....

Re: Forwarder not forwarding for 6 hours every day.

kristian_kolb — Mon, 10 Feb 2014 19:16:25 GMT

that is a message that shows up when the splunk instance starts up - the forwarder in your case. Is the machine being rebooted? or the forwarder service shutdown/restarted?

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Mon, 10 Feb 2014 19:27:29 GMT

Ok. The forwarder was restarted to see if forwarding would kickstart. Found a few errors.

ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=....). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

ERROR TailingProcessor - Ignoring path="/apps/splunk/splunkforwarder/var/log/splunk/.metrics.log.swp" due to: Bug: tried to check/configure STData processing but have no pending metadata.

Re: Forwarder not forwarding for 6 hours every day.

kristian_kolb — Mon, 10 Feb 2014 20:05:41 GMT

No, well, I meant that it could be that someone/something kills your forwarder process.

What type of file is it, what OS? Log rotation scheme?

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Mon, 10 Feb 2014 20:10:38 GMT

No one is killing the process. REL 6 x86_64. Logrotation is on, but doesn't occur until midnight. It's just an application file that is perfectly fine until 9am and starts working again at 2pm.

Re: Forwarder not forwarding for 6 hours every day.

MuS — Tue, 11 Feb 2014 07:02:31 GMT

Splunk tries to recognize the time stamp of events in multiple ways. One is to look in the raw data and search for date and time in it. Do you have some fancy date and time setting in the raw data that could confuse Splunk and makes it think your time is a date? Have you check 'all time' to see if the events come in with a wrong time stamp?

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Tue, 11 Feb 2014 14:55:29 GMT

Nothing fancy. The raw data is "Feb 11 09:47" format. Like I said, data gets indexed correctly then stops at 9am. I've checked "all time" and nothing shows.

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Tue, 11 Feb 2014 15:14:19 GMT

Hmmm, I think their time format in their logs is bad. I don't see a year.

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Tue, 11 Feb 2014 16:23:07 GMT

I know that splunk can grab date from the source file's modification time, so I don't understand why it would work most of the time and then not work for a specific duration.

Re: Forwarder not forwarding for 6 hours every day.

MuS — Wed, 12 Feb 2014 07:09:31 GMT

does 'stops at 9am' mean that also no more _internal logs are forwarded by the universal forwarder or just the log data?

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Wed, 12 Feb 2014 12:35:02 GMT

It seems to be just the log data. Although there is another log in the directory that is being forwarded just fine. It's just 4 logs that stop being forwarded.

Re: Forwarder not forwarding for 6 hours every day.

kristian_kolb — Wed, 12 Feb 2014 19:43:01 GMT

Did you check what the forwarder is really doing at the time of the 'outage'?

https://your_forwarder:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Tue, 18 Feb 2014 18:56:19 GMT

The forwarder appears to be functioning properly. It is forwarding all other logs on the server execpt for a small handfull.

Re: Forwarder not forwarding for 6 hours every day.

linu1988 — Tue, 18 Feb 2014 19:09:58 GMT

i doubt this could actually happen.

just type the in search and see the result.

source="whatever filename"|eval Time=strftime(_indextime,"%d/%m/%y %H:%M:%S %p")|table Time

Run it for last 7 days, and see you find the time series for the missing interval

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Tue, 18 Feb 2014 19:25:49 GMT

Nothing there between 08:59:24 AM and 14:01:01 PM.

Re: Forwarder not forwarding for 6 hours every day.

linu1988 — Tue, 18 Feb 2014 19:31:51 GMT

Could you paste some sample log updated between 9 AM to 2 PM your time when the file is not being read? That would be helpful.

Re: Forwarder not forwarding for 6 hours every day.

ssearwar — Tue, 18 Feb 2014 19:53:49 GMT

Had to edit the entries due to sensitive data.

INFO Feb 18 11:25 com.web.struts.ViewSummary - randomcharacters.stuff : Retrieving information.
INFO Feb 18 11:25 com.web.struts.ViewDetail - randomcharacters.stuff : Getting details for document randomnumbers
INFO Feb 18 11:25 com.web.struts.ViewSummary - randomcharacters.stuff : User Annuities Robot_test owns document: MONITOR

Re: Forwarder not forwarding for 6 hours every day.

linu1988 — Tue, 18 Feb 2014 20:33:49 GMT

make the search

source=xx

in search time criteria : custom time earliest choose 2/18/11 11:00:00.000 to latest "2/18/11 23:00:00.000"

Do you get any event?