topic Re: How do I configure Splunk to recognize the non-standard timestamp format in my log file? in Getting Data In

How do I configure Splunk to recognize the non-standard timestamp format in my log file?

markwymer — Thu, 20 Aug 2015 15:21:49 GMT

Hi All,

I'm trying to Upload a file/log using the 'Add Data' -> 'upload' in Splunk Web. Unfortunately, as per most of our logs, the input isn't in a structured format 😞

An example event looks like:-

my_application : access_live_05_6021 : 2015//08//18 20/:33/:24 Z : SUCCESS : apps.baplc.com%2Ftravel%2Fcarsproxy%2Fpublic%2Fen

My, initial, problem is that I can't get Splunk to recognise the timestamp - 2015//08//18 20/:33/:24 Z - I tried $Y//%m//%d $H/:$M/:%S Z in the 'Timestamp -> Advanced -> Timestamp Format' field, but it still couldn't detect the date field. I have a feeling that there is some kind of regex escape type stuff required, but ( I think ) I've tried everything except the correct solution!

The second question - for an extra bonus point 🙂 - is there an easy way in Splunk to change the apps.baplc.com%2Ftravel%2Fgeneral%2Fpublic%2Fen to apps.baplc.com/travel/general/public/en

Many thanks for any help,
Mark.

Re: How do I configure Splunk to recognize the non-standard timestamp format in my log file?

maciep — Thu, 20 Aug 2015 16:15:04 GMT

First off, are the dollar signs in your timestamp format typos or actually what you tried? They should be percent signs.

For the second part I think the urldecode function should work. As an example

| noop | stats count | eval blah="apps.baplc.com%2Ftravel%2Fgeneral%2Fpublic%2Fen"   | eval meh =urldecode(blah)

Re: How do I configure Splunk to recognize the non-standard timestamp format in my log file?

somesoni2 — Thu, 20 Aug 2015 16:17:05 GMT

In Splunk Web -> Data Preview, In Timestamps tab, use following

1) Timestamp is always prefaced by a pattern - ^\s*\w+\s*:\s*\w+\s*:\s*
2) Timestamp format (strptime) - %Y//%m//%d %H/:%M/:%S

Re: How do I configure Splunk to recognize the non-standard timestamp format in my log file?

richgalloway — Thu, 20 Aug 2015 16:30:59 GMT

Try this in the Timestamp Format box:

%Y//%m//%d %H/:%M/:%S %Z

No escaping is necessary, unless you want to include the literal '%' character in your format string. If it doesn't work, try specifying :\s+ as the time prefix.

For your second question, consider added a sed command to your props.conf file:

[mysourcetype]
SEDCMD-slash = s/%2F/\//g

Re: How do I configure Splunk to recognize the non-standard timestamp format in my log file?

markwymer — Fri, 21 Aug 2015 06:26:42 GMT

my apologies - yes the '$' were a typo

Re: How do I configure Splunk to recognize the non-standard timestamp format in my log file?

markwymer — Fri, 21 Aug 2015 08:04:05 GMT

My apologies - I accidentally used a '$' instead of a '%' in my previous comment! It should have read...

"I tried $Y//%m//%d %H/:%M/:%S Z in the 'Timestamp -> Advanced -> Timestamp Format' field"