topic Troubleshooting TIME_FORMAT by not being applied on indexes? in Getting Data In

Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Wed, 23 Mar 2022 22:28:26 GMT

I know this topic has been discussed many times in this thread, but I have not found a case like mine so far.
The index changes the day by the month and the month by the day from the 1st of each month until %d=%m. From 12/12 (for example) the data will be stored correctly in December.

The data I have in the log looks like this:

01/12/2021 12:10:04, ......

And the configuration I have in props.conf is as follows:

[source:://not/able/to/show/real/path/license_*.txt] TIME_FORMAT=%d/%m/%Y %H:%M:%S TIMESTAMP_FIELDS=Date

I have tried to analyze which props are taken into account with the command

splunk cmd btool props list --debug

The properties seem to be taken into account. In my case a TIME_PREFIX is not applicable either because there are no spaces or symbols at the beginning, I have tried everything.

Any suggestions? I ran out of ideas 😞

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

isoutamo — Wed, 01 Dec 2021 11:32:10 GMT

You can/should always use TIME_PREFIX=^ if it start from beginning of line.

You can try this with GUI and sample file. Just Settings -> Add Data -> Upload -> Select your sample file. Then in Set Source Type dashboard you can check that splunk recognise that data correctly.

Personally I test this always with my dev instance on laptop and then copy props + transforms.conf files to DS or needed cm/indexer etc.

r. Ismo

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Wed, 01 Dec 2021 11:53:06 GMT

Hi isoutamo, thanks for your answer,

I just edited props.conf to look like

[source:://not/able/to/show/the/path/license_*.txt] TIME_PREFIX=^ TIME_FORMAT=%d/%m/%Y %H:%M:%S TIMESTAMP_FIELDS=Date

I restarted splunk and checked with <splunk cmd btool props list --debug>. However the logs are still being saved on January the 12th

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

isoutamo — Wed, 01 Dec 2021 13:03:15 GMT

This props.conf is in your indexer / first HF on path and you have restarted it after change? And you are looking new events after change and restart?
Can you give some scrambled data samples?

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Thu, 02 Dec 2021 08:19:30 GMT

This props.conf is in my indexer, exactly, it belongs to the application I'm showing on splunk web interface, it works normally, just not between the 1st and the dd==mm..

Sure here a sample of my data.:

splunkuser@linuxSearchHead.org:: /here/the/path/to/the/file/license_usage.txt 02/12/2021 09:10:00,Application Test V1,X00X000XX00,XXXXX00,CurrentUser,XXX,0,2022-01-01 00:59:00,XXX,0,2022-03-01 00:59:00,0000000000000000,0,0 02/12/2021 09:10:00,Application Test V1,X00X000XX00,XXXXX00,CurrentUser,XXX,0,2022-01-01 00:59:00,XXX,0,2022-03-01 00:59:00,0000000000000000,0,0 02/12/2021 09:10:00,Application Test V1,X00X000XX00,XXXXX00,CurrentUser,XXX,0,2022-01-01 00:59:00,XXX,0,2022-03-01 00:59:00,0000000000000000,0,0

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

isoutamo — Thu, 02 Dec 2021 08:32:23 GMT

I think that this is something what you should remove your props.conf

TIMESTAMP_FIELDS=Date

I just test this with the next props.conf

[<sourcetype>] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) MAX_TIMESTAMP_LOOKAHEAD = 20 NO_BINARY_CHECK = true TIME_FORMAT = %d/%m/%Y %H:%M:%S TIME_PREFIX = ^

With this those days vs months are correctly shown in _time field.

r. Ismo

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Thu, 02 Dec 2021 09:50:21 GMT

This hasn't changed a thing. I have noticed that in the web interface, _time is shown on the exact same orther as in the log

_time: 02/12/2021 (meaning mm/dd/yyyy) --> splunk SH web interface

log: 02/12/2021 (meaning dd/mm/yyyy) --> input log

For some reason splunk uses the format mm/dd/yyyy, I don't really know if this is the issue. I am checking the datetime.xml file at the moment to see if I see something weird...

Thanks for you support

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

isoutamo — Thu, 02 Dec 2021 10:05:56 GMT

Splunk's default _time format with en-US locale is that mm/dd/YYYY.

When you are defining TIME_FORMAT then splunk shouldn't use datetime.xml at all. It will use datatime.xml for guessing the TIME_FORMAT if it hasn't defined manually (which is best practice).

When I open event (with your data and previous props.conf) _time is this

2021-12-02T09:10:00.000+02:00 and your event is "02/12/2021 09:10:00". As you can see those are exactly same based on your TIME_FORMAT definition.

isoutamo_0-1638439321328.png

If those are not matching in your own environment then I can guess the next reasons:

Splunk is using some other props.conf than what you are expecting
props.conf is defined in wrong instance (e.g. HF vs IDX)
You haven't restart splunk after manually edited props.conf and ingesting a new events.

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Thu, 02 Dec 2021 10:14:20 GMT

Hi again, indeed, you are right, _time has the aspect you just showed. In my comment before, I meant the column "Time" which is being shown on the search.. sorry for that.

I can tell that the props.conf I am using is actually being used, since yesterday I made a mistake in pupose to see if this was affecting the data inputs, and indeed, there was not data coming at all. As soon as I changed again the props.conf file, I was receiving data again.

By the way, I am using the IDX instance.

And yes, I restarted the splunk after manuall edition of props.conf, but there hasn't changed anything on the way splunk is picking mm and dd.

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

isoutamo — Thu, 02 Dec 2021 10:23:39 GMT

Now I have confused 🙂

If you are referring the clock time of _time field then your source are probably in different time zone (in reality or wrong configuration) than your indexer and for that reason.

If you are meaning those other field where are date time values, then you must manually handle those as splunk automatically handle only _time field.

r. Ismo

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Thu, 02 Dec 2021 10:35:50 GMT

Sorry for the confusion.

I mean, and copy your text, "those other field where are date time values, then you must manually handle those as splunk automatically handle only _time field"

My _time field looks exactly like the one you showed in the picture. It looks like this "2021-02-12T11:10:09.000+01:00"

Here you can see that splunk is understanding that this log belongs to Feb. and not Dec. which is wrong.

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

isoutamo — Thu, 02 Dec 2021 10:44:59 GMT

In that case your splunk is not using that TIME_FORMAT (your props.conf). Instead it try to guess the _time with datetime.xml.

Now you must find why your props.conf for that Sourcetype is not in use. You already have try to figure out it with (without <your sourcetype> part).

splunk btool props list <your sourcetype> --debug

Can you run the previous command again, with sourcetype part and post it here. Then is shows only this one Sourcetype nothing else. You are also sure that you haven't add any source or host definitions which match to this log/sourcetype?

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Thu, 02 Dec 2021 13:04:35 GMT

That's unfortunatelly not possible, since I am tied by the data protection policy.

You are talking about a sourcetype, but what I am using in this stranza is a source::/path/to/file/license_*.txt for the files coming from this path.

Doing a <splunk btool props source::/path/to/file/license_*.txt list --debug>, I can only see one props.conf path which is the same I am using, with the configuration you suggested before.

Sorry, I can't show you more..

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Thu, 02 Dec 2021 13:11:05 GMT

By the way, I just compared slowly my _time with your _time. Observe:

your _time: 2021-12-02T09:10:00.000+2:00 my _time: 2021-02-12T11:10:09.000+01:00

My _time has a wrong month. Could that be the issue? Where is this _time being configured?

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

isoutamo — Thu, 02 Dec 2021 14:11:33 GMT

Forgetting that this was a source not sourcetype 😞 But there shouldn't be any difference between those a time format point of view.

_time is configured with this TIME_FORMAT variable in props.conf. As it's wrongly in your case, it means that your source is not using this props.conf. Where you have set this props.conf?

You already check that source as

splunk btool props list source::.../license --debug

Basically this should work and at least it is recognized correctly when I create monitor (Settings -> Add Data -> monitor file) on local host. Then it shows those dates correctly on Add Data dashbord.

_time / Time	Event (only time part)
12/2/21 9:10:00.000 AM	02/12/2021 09:10:00, .....

Are this source on your IDX host or somewhere else and collected by UF? And there are no HF between UF and IDX? If there is any full Splunk Enterprise instance between source and your IDX then you must install that props.conf in that host, otherwise it didn't work!

Re: Troubleshooting TIME_FORMAT by not being applied on indexes?

Fernando_Sanch — Wed, 23 Mar 2022 13:54:55 GMT

The issue was solved by adding a props.conf file.. which we didn't have! but thanks anyway 🙂