Getting Data In

Struggling to get Splunk 6.0.1 to index EPOCH time for all events

mattchapple
Explorer

I'm struggling to get my Splunk 6.0.1 to recognise an epoch time for all events. I have specified a timestamp format of %s.%3N to assist identifying millisecond times but for some rows it's picking up an earlier field which is part of an IPv6 address.

For example the following line works correctly:

Request,555,10.22.16.23,100010001,endpointID,GECHO,COMMAND TYPE,2,1405918237788,,SUCCESS,

However this one doesn't, as it picks up 2:21 as the time:

Response,6c80f937-fb0c-4dd8-9df9-4e2d5d5eec95,2001:8888:0:2:21d:2300:5f6:811,100010001,,,ON_DEMAND_RESPONSE,,1405918239130,1405918239130,SUCCESS,

I can managed to get it to recognise but only if I moved the fields to the beginning and specified that "Timestamp never extends more than 13 chars into the event"

Can anyone provide assistance please. Unfortunately I'm not in a position whereby I can ask for a reordering of columns without incurring a commercial cost.

Many thanks.

Matt

Tags (3)
1 Solution

dshpritz
SplunkTrust
SplunkTrust

You will need to tell Splunk where in the event it should look for the timestamp. To do this, you will need a props.conf on your first parsing system (Heavy Forwarder or Indexer) that looks something like this:

[mysourcetype]
TIME_PREFIX = ^(?:[^,]*,){8}
MAX_TIMESTAMP_LOOKAHEAD = 13
TIME_FORMAT = %s%3N

Of course, you will need to change "mysourcetype" to the correct sourcetype for your events.

Thanks,

Dave

View solution in original post

dshpritz
SplunkTrust
SplunkTrust

You will need to tell Splunk where in the event it should look for the timestamp. To do this, you will need a props.conf on your first parsing system (Heavy Forwarder or Indexer) that looks something like this:

[mysourcetype]
TIME_PREFIX = ^(?:[^,]*,){8}
MAX_TIMESTAMP_LOOKAHEAD = 13
TIME_FORMAT = %s%3N

Of course, you will need to change "mysourcetype" to the correct sourcetype for your events.

Thanks,

Dave

mattchapple
Explorer

Got it, thanks Dave.
That worked a treat.

I now see that the {8} is specifying which comma delimitation pattern to skip along to. and then apply the lookahead 13 chars bit.

It all seems to be indexing fine. So thanks again.
Matt

dshpritz
SplunkTrust
SplunkTrust

You can use the TIME_PREFIX to tell Splunk "Hey, the time is going to come after the stuff that matches this regular expression". You then use the MAX_TIMESTAMP_LOOKAHEAD to tell it "the time will occur in the next _ characters". The TIME_FORMAT then tells Splunk what that timestamp will look like.

mattchapple
Explorer

Great, thanks Dave, I'll give that a try.

And just to confirm, that would allow me to have the epoch anywhere in the event rather than always at the beginning of each?

Sorry for my naivety.

Cheers,
Matt

0 Karma

mattchapple
Explorer

Yes you're right, the times for the first event are 1405918237788 and 1405918239130 for the second event. The additional value in the second event is a second gateway timestamp that we see, but this is not essential to be recognised.

Do you need particular lines from the transforms.conf file? or is there a way to attach files in here?

0 Karma

strive
Influencer

From your examples, i assume the epoch time stamps are:
1405918237788 and 1405918239130. Am i right?
Post here your transforms.conf configurations, so that will help us to help you better.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...