topic Force milliseconds into _raw when milliseconds not in source file time stamp in Splunk Search

Force milliseconds into _raw when milliseconds not in source file time stamp

jimdiconectiv — Thu, 29 Mar 2018 19:01:23 GMT

When have some queries where milliseconds are important. There is no difficulty if the ms value is stored in the index so that showing the epoch time. We get when milliseconds are in the original time stamp, but when it is not the original time stamp and the two types are intermixed. Examples:
Works well:
Source Time Stamp -- 2018-03-29T18:38:51.661Z
_time in epoch secs 1522348731.661 -- not decimal and milliseconds
Problem:
Source Time Stamp -- 2018-03-29T00:00:38+0000,
_time in epoch secs 1522281638 -- note NO milliseconds
I need this to show 1522281638.000

I would like all _time stamps to include a ms value even if the source does not.

I have tried SEDCMD and converted the stored log, but NOT splunk _time . I assume this is because the time stamp is
extracted before the SEDCMD.

What would be a good method? Is there a global parameter that will cause the ms value to always be filled to .000 on data with only whole seconds?

Re: Force milliseconds into _raw when milliseconds not in source file time stamp

hos_2 — Thu, 29 Mar 2018 22:08:38 GMT

Hey Jim,

You may need to try something like this http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Configuretimestamprecognition

I had a similar issue where one event would have milliseconds but the other did not, so I create a props.conf for it.

TIME_FORMAT = <strptime-style format>

In your case I think it would look something like this:

TIME_FORMAT =%Y-%m-%dT%H:%M:%S.%q

You can play with this in the GUI to see how it works, if you have a sample log and try Settings>Add Data>Upload

Re: Force milliseconds into _raw when milliseconds not in source file time stamp

jimdiconectiv — Tue, 29 Sep 2020 18:49:55 GMT

hos_2 ,
I have used time_stamp recognition before, but never in a case where there were intermixed formats like this, one with millisec and one without. Did you specify a single Format in your case? Does just showing a format with .%q force all to have millisecs. I will try it. Thanks for the reply !

Re: Force milliseconds into _raw when milliseconds not in source file time stamp

hos_2 — Thu, 29 Mar 2018 22:24:02 GMT

I believe I had to teach Splunk to recognize the different using Regex and the time_format props.conf settings.

The Add data GUI in the SH really helped me accomplish this as i could test my regex and time_format settings on the fly and see how it would affect my data before it was ingested.

Re: Force milliseconds into _raw when milliseconds not in source file time stamp

hos_2 — Thu, 29 Mar 2018 22:28:30 GMT

I used this Splunk answers when i ran into this problem:

https://answers.splunk.com/answers/499990/is-it-possible-to-assign-different-timestamps-base.html

I forgot to mention I also had to use Transforms