Forwarder not forwarding for 6 hours every day.

[ssearwar]

I'm running 6.01 server and forwarders. I've got a couple of linux servers that are not forwarding between a 6 hour period of time everyday for a particular log. Can someone point me in the right direction to correct this issue?

[linu1988 · Accepted solution]

Hello,
i was testing with your sample data and suspected the same when i saw your year field is not available. Splunk understands it's own way. In my case i was able to make it happen for 5 hours instead of 6 hours. The data exists in your splunk instance but it would hard to find the time period for that if you don't dig deep. If you see below.

You could see where your logs are going! So recommendation would be not to have log timestamp for the source rather have option as current i.e. file modification time or setup proper timestamp in the log itself. Thanks

[ssearwar]

You're right. It's every 5 hours. And yes, a proper timestamp in the log would be the best option. Thanks.

[linu1988]

make the search

source=xx

in search time criteria : custom time earliest choose 2/18/11 11:00:00.000 to latest "2/18/11 23:00:00.000"

Do you get any event?

[ssearwar]

Had to edit the entries due to sensitive data.

INFO Feb 18 11:25 com.web.struts.ViewSummary - randomcharacters.stuff : Retrieving information.
INFO Feb 18 11:25 com.web.struts.ViewDetail - randomcharacters.stuff : Getting details for document randomnumbers
INFO Feb 18 11:25 com.web.struts.ViewSummary - randomcharacters.stuff : User Annuities Robot_test owns document: MONITOR

[linu1988]

Could you paste some sample log updated between 9 AM to 2 PM your time when the file is not being read? That would be helpful.

[ssearwar]

Nothing there between 08:59:24 AM and 14:01:01 PM.

[linu1988]

i doubt this could actually happen.

just type the in search and see the result.

source="whatever filename"|eval Time=strftime(_indextime,"%d/%m/%y %H:%M:%S %p")|table Time

Run it for last 7 days, and see you find the time series for the missing interval

[ssearwar]

The forwarder appears to be functioning properly. It is forwarding all other logs on the server execpt for a small handfull.

[kristian_kolb]

Did you check what the forwarder is really doing at the time of the 'outage'?

https://your_forwarder:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Read more here;
http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

/K

[ssearwar]

It seems to be just the log data. Although there is another log in the directory that is being forwarded just fine. It's just 4 logs that stop being forwarded.

[MuS]

does 'stops at 9am' mean that also no more _internal logs are forwarded by the universal forwarder or just the log data?

[ssearwar]

I know that splunk can grab date from the source file's modification time, so I don't understand why it would work most of the time and then not work for a specific duration.

[ssearwar]

Hmmm, I think their time format in their logs is bad. I don't see a year.

[ssearwar]

Nothing fancy. The raw data is "Feb 11 09:47" format. Like I said, data gets indexed correctly then stops at 9am. I've checked "all time" and nothing shows.

[MuS]

Splunk tries to recognize the time stamp of events in multiple ways. One is to look in the raw data and search for date and time in it. Do you have some fancy date and time setting in the raw data that could confuse Splunk and makes it think your time is a date? Have you check 'all time' to see if the events come in with a wrong time stamp?

[ssearwar]

No one is killing the process. REL 6 x86_64. Logrotation is on, but doesn't occur until midnight. It's just an application file that is perfectly fine until 9am and starts working again at 2pm.

[kristian_kolb]

No, well, I meant that it could be that someone/something kills your forwarder process.

What type of file is it, what OS? Log rotation scheme?

[ssearwar]

Ok. The forwarder was restarted to see if forwarding would kickstart. Found a few errors.

ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=....). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

ERROR TailingProcessor - Ignoring path="/apps/splunk/splunkforwarder/var/log/splunk/.metrics.log.swp" due to: Bug: tried to check/configure STData processing but have no pending metadata.

[kristian_kolb]

that is a message that shows up when the splunk instance starts up - the forwarder in your case. Is the machine being rebooted? or the forwarder service shutdown/restarted?

[ssearwar]

No errors for the log that I need:

02-10-2014 11:02:53.021 -0500 INFO WatchedFile - Will begin reading at offset=1236998 for file='....