Syslog data is lost

[imbuto]

Hi,
I saw several posts about this problem, but none with a valid answer.
My problem is that I have a running Splunk 4.2.2 on Win64 that is receiving syslog messages from our switches since several months. It worked fine.
Today I configured some IP phones to log to it, using the same UDP 514 port I use for everything.
I cannot find any data making searches about phones (IP addresses, raw data, ...).
To separate things, I created a new UDP port 8514 listener, and I see in metrics.log that it is receiving data, I see data in Wireshark, but then it is lost.
I also created a specific index bound to this source, and the index contains no data.
No firewall involved in any network path.
No license limits hit. Logging to another syslog works ok.

How can I track where the data is going and have it indexed?

Some added debug

The port is open for listening

C:\Users\bigboss>netstat -an | grep 8514

UDP 0.0.0.0:8514 :

phones is the index I created and bound to the udp 8514 listener:

07-13-2011 12:47:54.346 +0200 INFO Metrics - group=per_index_thruput, series="phones", kbps=0.040134, eps=0.903226, kb=1.244141, ev=28, avg_age=40007.000000, max_age=40007

8514 UDP is my new listener

07-13-2011 12:47:54.346 +0200 INFO Metrics - group=udpin_connections, 8514, sourcePort=8514, _udp_bps=41.55, _udp_kbps=0.04, _udp_avg_thruput=0.04, _udp_kprocessed=80.00, _udp_eps=0.45

UPDATE
I found dozen of this in splunkd.log

137 similar messages suppressed.

First occurred at: Wed Jul 13 12:52:20 2011
07-13-2011 12:57:30.201 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:30.217 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:30.217 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:30.217 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:30.217 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:30.217 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:49.811 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:49.826 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:49.826 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

07-13-2011 12:57:49.826 +0200 WARN DateParserVerbose - Failed to parse timestamp for event. Context="source::Tel3|host::SRV-SYSLOG|syslog|" Text="mac:00-08-5D-27-3F-AB..."

The format of one log is like (yes it is on two lines):

01:58:33.070000 Refresh: (UI) FUNC: UiManager::Refresh, Leave

mac:00-08-5D-27-3F-AB

I'm not interested in parsing that timestamp it is wrong on each reboot of the phone, I could set it to the time Splunk received the message. Is there a way to do it?

[jbsplunk]

It sounds to me like your syslog data is coming into the index, it is just being indexed in a way that you do not expect. Perhaps the wrong year is being applied to the data. I would do a all time(real-time) search against the phone index and see if data is streaming in. If it is, then you'll likely need to configure timestamp recognition in props.conf with TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD.

You may also want to take a look at MAX_DAYS_AGO and MAX_DAYS_HENCE in props.conf to tell splunk not to look past 30 days in the past for an event. Based on the format of your event, I think you should also be using BREAK_ONLY_BEFORE and MUST_BREAK_AFTER in props.conf to tell splunk how you'd like the events to be broken.

I hope this helps!