Getting Data In

Why is the timestamp not recognized for one particular event from our Microsoft TMG Proxy Logs?

btiggemann
Path Finder

Hi Splunkers,

I have a strange problem with Microsoft TMG, Splunk can't find the time stamp on one particular event. On all other events the time stamp is running fine.

For this event Splunk finds the time stamp automaticly:

172.31.192.191, anonymous, -, -, 11/10/2014, 16:19:36, -, KST032, -,
172.31.209.144, 172.31.209.144, 443, 1, 59, 3391, https, -, GET, http://172.31.209.144/, -, -, 12202,
-, [Enterprise] Standardregel, Req ID: 0b150cc4; Compression: client=No` server=No` compress rate=0% decompress rate=0% ; FBA cookie: exists=no` valid=no` updated=no` logged off=no` client type=unknown` user activity=yes, Internal, Local Host, 0x0, Denied, -, -, -, -, -, -, -, -,
-, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, Web Proxy, 172.31.209.144, 35716, -

For this event Splunk is not able to find the time stamp:

172.21.29.67, VERWALTUNG\user, Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML` like Gecko) Chrome/38.0.2125.111 Safari/537.36, -, 11/10/2014, 16:19:38, -, KST032, http://webradio.ffh.de/top40, 172.31.209.226, 172.31.209.226, 9090, 62, 648, 1003, http, -, POST, http://webradio.ffh.de/custom/getAllSonginfos.php, application/json, Upstream, 200, -, HTTP und HTTPS erlaubt, Req ID: 0b150ca5; Compression: client=No` server=No` compress rate=0% decompress rate=0%, Internal, Internal, 0x580, Allowed, 11/10/2014 15:19:38, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, None, webradio.ffh.de, 56419, -

Because of that issue the line breaking is going wrong and some events are combined together which is wrong.

We tried to use a REGEX that takes out the time stamp after the 4th comma, but it was not successful because there's also a comma between date and time.

I thought there's a problem with the "`" character in the user agent field. This is not the issue, I have tried to remove this character in one log event, but the time stamp problem is still there.

Does any body see a difference between the logs? What could be the issue?

I would be very thankful if you can help me.

Best regards

Tags (2)
0 Karma

tom_frotscher
Builder

Hi, i think this might be caused by the following configuration default:

MAX_TIMESTAMP_LOOKAHEAD

This is the configuration, how far splunk is seaching in the event to find a timestamp and it defaults to 150 characters. If you count the characters to the end of your date in the event where the timestamp is not correctly recognized, you exceed the 150 chars.

Can you try to set this value up, maybe to 250? You should add this to the stanza of your sourcetype in the props.conf.

Greetings

Tom

btiggemann
Path Finder

OH yes, we will try this. I have seen this value but I haven't counted the characters... Thanks a lot

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...