Getting Data In

Optimizing custom log formats

travispowell
Path Finder

I read a post on the site describing how an optimum custom log format for Splunk would take the form:

<timestamp> key=val key=val key=val key=val

...and I tried to build a log formatter for our in-house software that would write logs like this. I'm trying out Splunk, and trying to figure out why it doesn't pick up the timestamps for what they are. Here's a single log entry (the first number is a UNIX timestamp):

1303115585 SESSION_KEY=56c2964bce6b36da9e895c5be963584a REMOTE_ADDRESS=65.13.25.203 CANISTER_LSSN=LSSN_20110418_MASTER.dat CANISTER_SESSION_ID=153051 SID=7B019FB669961069023EADEB66C4E2BE UID=6C6838A20A1E100A01139E8210F7048E VID= CANISTER_SERVER=MASTER:19000 DURATION=103 HCOUNT=2 HTTP_USER_AGENT=Windows-RSS-Platform/2.0_(MSIE_8.0;_Windows_NT_5.1) EXTRACTID=1303156352 LINK=http:\/\/MASTER:19000/Session.rfx?canName%3DCANISTER.dbs\LSSN_20110418_MASTER.dat&sessionId%3D153051

I'm wondering if the link at the end if causing me grief, but I even encoded the '=' and replaced the spaces in the HTTP_USER_AGENT field with underscores.

So am I right to assume that I have to teach it how to read my dates with the >splunk train command? Does Splunk not auto-extract UNIX timestamps?

0 Karma
1 Solution

dwaddle
SplunkTrust
SplunkTrust

There was a bug related to auto-detecting an epoch time format, fixed in 4.2.1. You can see a set of workarounds at http://www.splunk.com/base/Documentation/4.2/ReleaseNotes/Knownissues. My personal preference is to explicitly set TIME_FORMAT for my sourcetypes so there is no guessing as to how the time is parsed.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

There was a bug related to auto-detecting an epoch time format, fixed in 4.2.1. You can see a set of workarounds at http://www.splunk.com/base/Documentation/4.2/ReleaseNotes/Knownissues. My personal preference is to explicitly set TIME_FORMAT for my sourcetypes so there is no guessing as to how the time is parsed.

travispowell
Path Finder

SOLVED: I ended up setting the TIME_FORMAT. Thanks

0 Karma

travispowell
Path Finder

Guess that's what I'll have to do. I don't think it's entirely fixed.
Thanks 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...