topic Re: Optimizing custom log formats in Getting Data In

Optimizing custom log formats

travispowell — Tue, 19 Apr 2011 17:39:17 GMT

I read a post on the site describing how an optimum custom log format for Splunk would take the form:

<timestamp> key=val key=val key=val key=val

...and I tried to build a log formatter for our in-house software that would write logs like this. I'm trying out Splunk, and trying to figure out why it doesn't pick up the timestamps for what they are. Here's a single log entry (the first number is a UNIX timestamp):

1303115585 SESSION_KEY=56c2964bce6b36da9e895c5be963584a REMOTE_ADDRESS=65.13.25.203 CANISTER_LSSN=LSSN_20110418_MASTER.dat CANISTER_SESSION_ID=153051 SID=7B019FB669961069023EADEB66C4E2BE UID=6C6838A20A1E100A01139E8210F7048E VID= CANISTER_SERVER=MASTER:19000 DURATION=103 HCOUNT=2 HTTP_USER_AGENT=Windows-RSS-Platform/2.0_(MSIE_8.0;_Windows_NT_5.1) EXTRACTID=1303156352 LINK=http:\/\/MASTER:19000/Session.rfx?canName%3DCANISTER.dbs\LSSN_20110418_MASTER.dat&sessionId%3D153051

I'm wondering if the link at the end if causing me grief, but I even encoded the '=' and replaced the spaces in the HTTP_USER_AGENT field with underscores.

So am I right to assume that I have to teach it how to read my dates with the >splunk train command? Does Splunk not auto-extract UNIX timestamps?

Re: Optimizing custom log formats

dwaddle — Tue, 19 Apr 2011 19:04:18 GMT

There was a bug related to auto-detecting an epoch time format, fixed in 4.2.1. You can see a set of workarounds at http://www.splunk.com/base/Documentation/4.2/ReleaseNotes/Knownissues. My personal preference is to explicitly set TIME_FORMAT for my sourcetypes so there is no guessing as to how the time is parsed.

Re: Optimizing custom log formats

travispowell — Tue, 19 Apr 2011 19:14:46 GMT

Guess that's what I'll have to do. I don't think it's entirely fixed.
Thanks 🙂

Re: Optimizing custom log formats

travispowell — Tue, 19 Apr 2011 23:43:56 GMT

SOLVED: I ended up setting the TIME_FORMAT. Thanks