topic Re: TIME not getting milliseconds in Getting Data In

TIME not getting milliseconds

MERBAG — Fri, 06 Jul 2018 08:05:27 GMT

Hi all,

I do have a log which does look like this:

Jul  6 09:31:18.729: %SYS-5-CONFIG_I: Configured from console by username on vty1 (ip-address)

This data is received by syslog, but for any reason the internal _time variable does not contain the milliseconds, in this example .729 the _time variable in splunk does look like this: 2018-07-06T09:31:18.000+02:00

So I did some research and started to edit the props.conf for this sourcetyp

TIME_PREFIX=^
TIME_FORMAT=%b %d %H:%M:%S.%3N

But for any reason this did not impact the _time variable in splunk, can someone may tell me what I do wrong here?

Thanks a lot.

Re: TIME not getting milliseconds

FrankVl — Fri, 06 Jul 2018 08:15:05 GMT

Since the day of month is preceded with a space when single digit instead of 06, you need to use %e instead of %d.

Re: TIME not getting milliseconds

poete — Fri, 06 Jul 2018 08:17:43 GMT

Hello @MERBAG,

it seems that the fomat for the millisesonds is not OK.

According to the documentation (https://docs.splunk.com/Documentation/Splunk/7.1.1/SearchReference/Commontimeformatvariables),
replace %3Nwith %3Q

Re: TIME not getting milliseconds

FrankVl — Fri, 06 Jul 2018 08:23:21 GMT

I've always used %3N for that. Documentation is a bit confusing in that sense, I don't understand the difference between %xN and %xQ:

%N  Subseconds with width. (%3N = milliseconds, %6N = microseconds, %9N = nanoseconds)
 %Q The subsecond component of 2017-11-30 23:59:59.999 UTC. %3Q = milliseconds, with values of 000-999. %6Q = microseconds, with values of 000000-999999. %9Q = nanoseconds, with values of 000000000-999999999.

Re: TIME not getting milliseconds

MERBAG — Fri, 06 Jul 2018 08:29:30 GMT

Thanks for the quick feedback. I just did the change you said, but unfortunatly that didnt help - the _time does still look like this:

2018-07-06T10:25:34.000+02:00

The props.conf now looks like this:

TIME_PREFIX=^
TIME_FORMAT=%b %e %H:%M:%S.%3N

Re: TIME not getting milliseconds

poete — Fri, 06 Jul 2018 08:29:56 GMT

Well, that's good to know. I always used %Q up to now, and it always worked so far. May be worth a try

Re: TIME not getting milliseconds

MERBAG — Fri, 06 Jul 2018 08:34:15 GMT

Hey,

also just tried that, still did not change anything, time in the log does look like this:

Jul 6 10:30:21.987

the _time variable now contains this:

2018-07-06T10:30:22.000+02:00

Re: TIME not getting milliseconds

FrankVl — Fri, 06 Jul 2018 08:43:03 GMT

From how I read the docs they would be the same anyway, so not surprising %Q worked for you. Why they have 2 ways of doing the same is beyond me 🙂

Re: TIME not getting milliseconds

poete — Fri, 06 Jul 2018 08:44:27 GMT

| makeresults
| eval _time=strptime("Jul  6 09:31:18.729: %SYS-5-CONFIG_I: Configured from console by username on vty1 (ip-address)","%b %e %H:%M:%S.%3Q")

works like a charm. Are you sure you are modifying the right props.conf file?

Re: TIME not getting milliseconds

FrankVl — Fri, 06 Jul 2018 08:44:46 GMT

In that case, issue may be with how you've set up your props.conf and where you deployed it.

Can you show a bit more of the config files used for this data input? You set the time config based on sourcetype, right? Is that sourcetype determined in inputs.conf, or is it overridden using props and transforms?
What does your architecture look like and where in the architecture did you deploy the time config props.conf?

Re: TIME not getting milliseconds

MERBAG — Fri, 06 Jul 2018 09:00:22 GMT

@poete , @FrankVI
Since there came up to similar questions, I want to explain a bit more how the setup does look like. To be honest it is the first time I am doing this, so my answer to the question "do you modify the right props.conf file?" is: I think so, but I do not know 🙂

We did create a new app, in the app the inputs.conf does look like this:

[udp://515]
connection_host = dns
index = cisco
sourcetype = switch

in the same app the props.conf does look like this:

[switch]
category = Network & Security
TIME_PREFIX=^
TIME_FORMAT=%b %e %H:%M:%S.%3Q
pulldown_type = 1
EXTRACT-ciscoaudit = ^(?:[^ \n]* ){4}(?P<hostname>[^ ]+)(?:[^ \n]* ){2}(?P<iostime>\w+\s+\d+\s+\d+:\d+:\d+\.\d+)[^%\n]*%(?P<facility>\w+)\-(?P<severity>[^\-]+)\-(?P<mnemonic>[^:]+):\s+\w+:(?P<user>[a-z]+)[^:\n]*:(?P<command>.+)
EXTRACT-cisco = ^(?:[^ \n]* ){4}(?P<hostname>[^ ]+)[^:\n]*:\s+(?P<iostime>\w+\s+\d+\s+\d+:\d+:\d+\.\d+)[^%\n]*%(?P<facility>\w+)\-(?P<severity>[^\-]+)\-(?P<mnemonic>\w+)[^ \n]* (?P<message>.+)

I also already tried to remove the EXTRACT-* part from the props.conf since I though the may have any negativ impact, but that also did not change anything.

Re: TIME not getting milliseconds

poete — Fri, 06 Jul 2018 09:12:03 GMT

@MERBAG,

depending if you are running a single instance of Splunk, or an instance with separate indexer and search head, the steps for the props.conf to be taken into account are different.

If you are running on a single instance, in what directory is located the props.conf file you modified?

Re: TIME not getting milliseconds

MERBAG — Fri, 06 Jul 2018 09:15:55 GMT

It is a single instance, the directory for the inputs and props file I posted here is

C:\Program Files\Splunk\etc\apps\merbag_it\local

Re: TIME not getting milliseconds

FrankVl — Fri, 06 Jul 2018 09:40:37 GMT

Have you restarted Splunk after making the adjustments?

Re: TIME not getting milliseconds

MERBAG — Fri, 06 Jul 2018 10:41:12 GMT

yes, after every change I did a restart uf splunk and generated new logs to test

Re: TIME not getting milliseconds

MERBAG — Mon, 09 Jul 2018 06:18:21 GMT

Does someone have any other ideas?