Why is the timestamp not recognized for one particular event from our Microsoft TMG Proxy Logs?

[btiggemann]

Hi Splunkers,

I have a strange problem with Microsoft TMG, Splunk can't find the time stamp on one particular event. On all other events the time stamp is running fine.

For this event Splunk finds the time stamp automaticly:

172.31.192.191, anonymous, -, -, 11/10/2014, 16:19:36, -, KST032, -,
172.31.209.144, 172.31.209.144, 443, 1, 59, 3391, https, -, GET, http://172.31.209.144/, -, -, 12202,
-, [Enterprise] Standardregel, Req ID: 0b150cc4; Compression: client=No` server=No` compress rate=0% decompress rate=0% ; FBA cookie: exists=no` valid=no` updated=no` logged off=no` client type=unknown` user activity=yes, Internal, Local Host, 0x0, Denied, -, -, -, -, -, -, -, -,
-, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, Web Proxy, 172.31.209.144, 35716, -

For this event Splunk is not able to find the time stamp:

172.21.29.67, VERWALTUNG\user, Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML` like Gecko) Chrome/38.0.2125.111 Safari/537.36, -, 11/10/2014, 16:19:38, -, KST032, http://webradio.ffh.de/top40, 172.31.209.226, 172.31.209.226, 9090, 62, 648, 1003, http, -, POST, http://webradio.ffh.de/custom/getAllSonginfos.php, application/json, Upstream, 200, -, HTTP und HTTPS erlaubt, Req ID: 0b150ca5; Compression: client=No` server=No` compress rate=0% decompress rate=0%, Internal, Internal, 0x580, Allowed, 11/10/2014 15:19:38, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, None, webradio.ffh.de, 56419, -

Because of that issue the line breaking is going wrong and some events are combined together which is wrong.

We tried to use a REGEX that takes out the time stamp after the 4th comma, but it was not successful because there's also a comma between date and time.

I thought there's a problem with the "`" character in the user agent field. This is not the issue, I have tried to remove this character in one log event, but the time stamp problem is still there.

Does any body see a difference between the logs? What could be the issue?

I would be very thankful if you can help me.

Best regards

[tom_frotscher]

Hi, i think this might be caused by the following configuration default:

MAX_TIMESTAMP_LOOKAHEAD

This is the configuration, how far splunk is seaching in the event to find a timestamp and it defaults to 150 characters. If you count the characters to the end of your date in the event where the timestamp is not correctly recognized, you exceed the 150 chars.

Can you try to set this value up, maybe to 250? You should add this to the stanza of your sourcetype in the props.conf.

Greetings

Tom

[btiggemann]

OH yes, we will try this. I have seen this value but I haven't counted the characters... Thanks a lot