Getting Data In

Struggling to get Splunk 6.0.1 to index EPOCH time for all events

mattchapple
Explorer

I'm struggling to get my Splunk 6.0.1 to recognise an epoch time for all events. I have specified a timestamp format of %s.%3N to assist identifying millisecond times but for some rows it's picking up an earlier field which is part of an IPv6 address.

For example the following line works correctly:

Request,555,10.22.16.23,100010001,endpointID,GECHO,COMMAND TYPE,2,1405918237788,,SUCCESS,

However this one doesn't, as it picks up 2:21 as the time:

Response,6c80f937-fb0c-4dd8-9df9-4e2d5d5eec95,2001:8888:0:2:21d:2300:5f6:811,100010001,,,ON_DEMAND_RESPONSE,,1405918239130,1405918239130,SUCCESS,

I can managed to get it to recognise but only if I moved the fields to the beginning and specified that "Timestamp never extends more than 13 chars into the event"

Can anyone provide assistance please. Unfortunately I'm not in a position whereby I can ask for a reordering of columns without incurring a commercial cost.

Many thanks.

Matt

Tags (3)
1 Solution

dshpritz
SplunkTrust
SplunkTrust

You will need to tell Splunk where in the event it should look for the timestamp. To do this, you will need a props.conf on your first parsing system (Heavy Forwarder or Indexer) that looks something like this:

[mysourcetype]
TIME_PREFIX = ^(?:[^,]*,){8}
MAX_TIMESTAMP_LOOKAHEAD = 13
TIME_FORMAT = %s%3N

Of course, you will need to change "mysourcetype" to the correct sourcetype for your events.

Thanks,

Dave

View solution in original post

dshpritz
SplunkTrust
SplunkTrust

You will need to tell Splunk where in the event it should look for the timestamp. To do this, you will need a props.conf on your first parsing system (Heavy Forwarder or Indexer) that looks something like this:

[mysourcetype]
TIME_PREFIX = ^(?:[^,]*,){8}
MAX_TIMESTAMP_LOOKAHEAD = 13
TIME_FORMAT = %s%3N

Of course, you will need to change "mysourcetype" to the correct sourcetype for your events.

Thanks,

Dave

mattchapple
Explorer

Got it, thanks Dave.
That worked a treat.

I now see that the {8} is specifying which comma delimitation pattern to skip along to. and then apply the lookahead 13 chars bit.

It all seems to be indexing fine. So thanks again.
Matt

dshpritz
SplunkTrust
SplunkTrust

You can use the TIME_PREFIX to tell Splunk "Hey, the time is going to come after the stuff that matches this regular expression". You then use the MAX_TIMESTAMP_LOOKAHEAD to tell it "the time will occur in the next _ characters". The TIME_FORMAT then tells Splunk what that timestamp will look like.

mattchapple
Explorer

Great, thanks Dave, I'll give that a try.

And just to confirm, that would allow me to have the epoch anywhere in the event rather than always at the beginning of each?

Sorry for my naivety.

Cheers,
Matt

0 Karma

mattchapple
Explorer

Yes you're right, the times for the first event are 1405918237788 and 1405918239130 for the second event. The additional value in the second event is a second gateway timestamp that we see, but this is not essential to be recognised.

Do you need particular lines from the transforms.conf file? or is there a way to attach files in here?

0 Karma

strive
Influencer

From your examples, i assume the epoch time stamps are:
1405918237788 and 1405918239130. Am i right?
Post here your transforms.conf configurations, so that will help us to help you better.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...