Why is there a gap in my metrics.log?

[hulahoop]

Why would there be a gap of logged events in metrics.log between 01-21-2010 15:47:39.421 and 01-22-2010 08:53:28.231 (~18 hours)?

01-21-2010 15:47:39.421 INFO Metrics - group=queue, name=typingqueue, blocked!!=true, max_size=1000, filled_count=0, empty_count=0, current_size=1000, largest_size=0, smallest_size=1000
01-21-2010 15:47:39.421 INFO Metrics - group=tcpin_connections, 10.150.166.214:54853:5140, connectionType=raw, sourcePort=54853, sourceHost=itg-mpsc-syslog03.uboc.com, sourceIp=10.150.166.214, destPort=5140, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.37, _tcp_Kprocessed=9763.00, _tcp_eps=0.00
01-21-2010 15:47:39.421 INFO Metrics - group=tcpin_connections, 10.150.28.121:47494:9997, connectionType=cooked, sourcePort=47494, sourceHost=XPD-LKVB494.uboc.com, sourceIp=10.150.28.121, destPort=9997, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=4.76, _tcp_Kprocessed=253798.00, _tcp_eps=0.00
01-21-2010 15:47:39.421 INFO Metrics - group=tcpin_connections, 10.151.254.188:43220:5141, connectionType=raw, sourcePort=43220, sourceHost=ITG-MPSC-LOG01.uboc.com, sourceIp=10.151.254.188, destPort=5141, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=3.42, _tcp_Kprocessed=89758.00, _tcp_eps=0.00
01-22-2010 08:53:28.231 INFO StatusMgr - Registering StatusListener StatusMgrLogger
01-22-2010 08:53:28.231 INFO StatusMgr - Registering StatusListener TCPOutStatusListener
01-22-2010 08:53:33.951 INFO StatusMgr - destPort=5141, eventType=connect_done, rawMode=true, sourceHost=ITG-MPSC-LOG01.uboc.co m, sourceIp=10.151.254.188, sourcePort=55514, statusee=TcpInputProcessor

[natrixia · Accepted solution]

splunkd is running right now. How can I find out if it was running during the gap? splunkd.log does show log entries through out that time frame.

[hulahoop]

Thank you, k8to, for information on the general behavior of the metrics log. On recollection, we were running Splunk 4.0.6 at the time this gap was detected. We have since upgraded to Splunk 4.0.9, and will continue to monitor for this behavior. One question: why k8to? 😉

[jrodman]

A pause in metrics.log while splunkd.log continues is symptomatic of a bug fixed around 4.0.5 or 4.0.6 timeframe. I do not have access to the defect system right now, but the version of splunk you are running is probably informative.

Generally speaking, metrics.log should always continue chatting when splunkd is up, provided that there is disk space available and suchlike.

[hulahoop]

In $SPLUNK_HOME/var/log/splunk, are there any crash logs? Crash logs start with "Crash..."

[Johnvey]

(from V_at_Splunk): Are you sure splunkd was up all this time? On UNIX systems, check with ps -eo comm,args,pid,etime | grep splunkd, or such. (The etime gives elapsed time.)