Getting Data In

Struggling to get Splunk 6.0.1 to index EPOCH time for all events

Explorer

I'm struggling to get my Splunk 6.0.1 to recognise an epoch time for all events. I have specified a timestamp format of %s.%3N to assist identifying millisecond times but for some rows it's picking up an earlier field which is part of an IPv6 address.

For example the following line works correctly:

Request,555,10.22.16.23,100010001,endpointID,GECHO,COMMAND TYPE,2,1405918237788,,SUCCESS,

However this one doesn't, as it picks up 2:21 as the time:

Response,6c80f937-fb0c-4dd8-9df9-4e2d5d5eec95,2001:8888:0:2:21d:2300:5f6:811,100010001,,,ON_DEMAND_RESPONSE,,1405918239130,1405918239130,SUCCESS,

I can managed to get it to recognise but only if I moved the fields to the beginning and specified that "Timestamp never extends more than 13 chars into the event"

Can anyone provide assistance please. Unfortunately I'm not in a position whereby I can ask for a reordering of columns without incurring a commercial cost.

Many thanks.

Matt

Tags (3)
1 Solution

SplunkTrust
SplunkTrust

You will need to tell Splunk where in the event it should look for the timestamp. To do this, you will need a props.conf on your first parsing system (Heavy Forwarder or Indexer) that looks something like this:

[mysourcetype]
TIME_PREFIX = ^(?:[^,]*,){8}
MAX_TIMESTAMP_LOOKAHEAD = 13
TIME_FORMAT = %s%3N

Of course, you will need to change "mysourcetype" to the correct sourcetype for your events.

Thanks,

Dave

View solution in original post

SplunkTrust
SplunkTrust

You will need to tell Splunk where in the event it should look for the timestamp. To do this, you will need a props.conf on your first parsing system (Heavy Forwarder or Indexer) that looks something like this:

[mysourcetype]
TIME_PREFIX = ^(?:[^,]*,){8}
MAX_TIMESTAMP_LOOKAHEAD = 13
TIME_FORMAT = %s%3N

Of course, you will need to change "mysourcetype" to the correct sourcetype for your events.

Thanks,

Dave

View solution in original post

Explorer

Got it, thanks Dave.
That worked a treat.

I now see that the {8} is specifying which comma delimitation pattern to skip along to. and then apply the lookahead 13 chars bit.

It all seems to be indexing fine. So thanks again.
Matt

SplunkTrust
SplunkTrust

You can use the TIME_PREFIX to tell Splunk "Hey, the time is going to come after the stuff that matches this regular expression". You then use the MAX_TIMESTAMP_LOOKAHEAD to tell it "the time will occur in the next _ characters". The TIME_FORMAT then tells Splunk what that timestamp will look like.

Explorer

Great, thanks Dave, I'll give that a try.

And just to confirm, that would allow me to have the epoch anywhere in the event rather than always at the beginning of each?

Sorry for my naivety.

Cheers,
Matt

0 Karma

Explorer

Yes you're right, the times for the first event are 1405918237788 and 1405918239130 for the second event. The additional value in the second event is a second gateway timestamp that we see, but this is not essential to be recognised.

Do you need particular lines from the transforms.conf file? or is there a way to attach files in here?

0 Karma

Influencer

From your examples, i assume the epoch time stamps are:
1405918237788 and 1405918239130. Am i right?
Post here your transforms.conf configurations, so that will help us to help you better.

0 Karma