Troubleshooting TIME_FORMAT by not being applied on indexes?

[Fernando_Sanch]

I know this topic has been discussed many times in this thread, but I have not found a case like mine so far.
The index changes the day by the month and the month by the day from the 1st of each month until %d=%m. From 12/12 (for example) the data will be stored correctly in December.

The data I have in the log looks like this:

01/12/2021 12:10:04, ......

And the configuration I have in props.conf is as follows:

[source:://not/able/to/show/real/path/license_*.txt]
TIME_FORMAT=%d/%m/%Y %H:%M:%S
TIMESTAMP_FIELDS=Date

I have tried to analyze which props are taken into account with the command

splunk cmd btool props list --debug

The properties seem to be taken into account. In my case a TIME_PREFIX is not applicable either because there are no spaces or symbols at the beginning, I have tried everything.

Any suggestions? I ran out of ideas 😞

[Fernando_Sanch · Accepted solution]

The issue was solved by adding a props.conf file.. which we didn't have! but thanks anyway 🙂

[isoutamo]

You can/should always use TIME_PREFIX=^ if it start from beginning of line.

You can try this with GUI and sample file. Just Settings -> Add Data -> Upload -> Select your sample file. Then in Set Source Type dashboard you can check that splunk recognise that data correctly.

Personally I test this always with my dev instance on laptop and then copy props + transforms.conf files to DS or needed cm/indexer etc.

r. Ismo

[Fernando_Sanch]

Hi isoutamo, thanks for your answer,

I just edited props.conf to look like

[source:://not/able/to/show/the/path/license_*.txt]
TIME_PREFIX=^
TIME_FORMAT=%d/%m/%Y %H:%M:%S
TIMESTAMP_FIELDS=Date

I restarted splunk and checked with <splunk cmd btool props list --debug>. However the logs are still being saved on January the 12th

[isoutamo]

This props.conf is in your indexer / first HF on path and you have restarted it after change? And you are looking new events after change and restart?
Can you give some scrambled data samples?

[Fernando_Sanch]

This props.conf is in my indexer, exactly, it belongs to the application I'm showing on splunk web interface, it works normally, just not between the 1st and the dd==mm..

Sure here a sample of my data.:

splunkuser@linuxSearchHead.org:: /here/the/path/to/the/file/license_usage.txt

02/12/2021 09:10:00,Application Test V1,X00X000XX00,XXXXX00,CurrentUser,XXX,0,2022-01-01 00:59:00,XXX,0,2022-03-01 00:59:00,0000000000000000,0,0
02/12/2021 09:10:00,Application Test V1,X00X000XX00,XXXXX00,CurrentUser,XXX,0,2022-01-01 00:59:00,XXX,0,2022-03-01 00:59:00,0000000000000000,0,0
02/12/2021 09:10:00,Application Test V1,X00X000XX00,XXXXX00,CurrentUser,XXX,0,2022-01-01 00:59:00,XXX,0,2022-03-01 00:59:00,0000000000000000,0,0

[isoutamo]

I think that this is something what you should remove your props.conf 

TIMESTAMP_FIELDS=Date

I just test this with the next props.conf

[<sourcetype>]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 20
NO_BINARY_CHECK = true
TIME_FORMAT = %d/%m/%Y %H:%M:%S
TIME_PREFIX = ^

With this those days vs months are correctly shown in _time field.

r. Ismo

[Fernando_Sanch]

This hasn't changed a thing. I have noticed that in the web interface, _time is shown on the exact same orther as in the log

_time: 02/12/2021 (meaning mm/dd/yyyy) --> splunk SH web interface

log: 02/12/2021 (meaning dd/mm/yyyy) --> input log

For some reason splunk uses the format mm/dd/yyyy, I don't really know if this is the issue. I am checking the datetime.xml file at the moment to see if I see something weird...

Thanks for you support

[isoutamo]

Splunk's default _time format with en-US locale is that mm/dd/YYYY.

When you are defining TIME_FORMAT then splunk shouldn't use datetime.xml at all. It will use datatime.xml for guessing the TIME_FORMAT if it hasn't defined manually (which is best practice).

When I open event (with your data and previous props.conf) _time is this

2021-12-02T09:10:00.000+02:00 and your event is "02/12/2021 09:10:00". As you can see those are exactly same based on your TIME_FORMAT definition.

If those are not matching in your own environment then I can guess the next reasons:

• Splunk is using some other props.conf than what you are expecting
• props.conf is defined in wrong instance (e.g. HF vs IDX)
• You haven't restart splunk after manually edited props.conf and ingesting a new events.

[Fernando_Sanch]

Hi again, indeed, you are right, _time has the aspect you just showed. In my comment before, I meant the column "Time" which is being shown on the search.. sorry for that.

I can tell that the props.conf I am using is actually being used, since yesterday I made a mistake in pupose to see if this was affecting the data inputs, and indeed, there was not data coming at all. As soon as I changed again the props.conf file, I was receiving data again.

By the way, I am using the IDX instance.

And yes, I restarted the splunk after manuall edition of props.conf, but there hasn't changed anything on the way splunk is picking mm and dd.

[isoutamo]

Now I have confused 🙂 

If you are referring the clock time of _time field then your source are probably in different time zone (in reality or wrong configuration) than your indexer and for that reason.

If you are meaning those other field where are date time values, then you must manually handle those as splunk automatically handle only _time field.

r. Ismo

[Fernando_Sanch]

Sorry for the confusion.

I mean, and copy your text, "those other field where are date time values, then you must manually handle those as splunk automatically handle only _time field"

My _time field looks exactly like the one you showed in the picture. It looks like this "2021-02-12T11:10:09.000+01:00"

Here you can see that splunk is understanding that this log belongs to Feb. and not Dec. which is wrong.

[isoutamo]

In that case your splunk is not using that TIME_FORMAT (your props.conf). Instead it try to guess the _time with datetime.xml.

Now you must find why your props.conf for that Sourcetype is not in use. You already have try to figure out it with (without <your sourcetype> part).

splunk btool props list <your sourcetype> --debug

Can you run the previous command again, with sourcetype part and post it here. Then is shows only this one Sourcetype nothing else. You are also sure that you haven't add any source or host definitions which match to this log/sourcetype?

[Fernando_Sanch]

That's unfortunatelly not possible, since I am tied by the data protection policy.

You are talking about a sourcetype, but what I am using in this stranza is a source::/path/to/file/license_*.txt for the files coming from this path.

Doing a <splunk btool props source::/path/to/file/license_*.txt list --debug>, I can only see one props.conf path which is the same I am using, with the configuration you suggested before.

Sorry, I can't show you more..

[Fernando_Sanch]

By the way, I just compared slowly my _time with your _time. Observe:

your _time:     2021-12-02T09:10:00.000+2:00

my _time:       2021-02-12T11:10:09.000+01:00

My _time has a wrong month. Could that be the issue? Where is this _time being configured?

[isoutamo]

Forgetting that this was a source not sourcetype 😞 But there shouldn't be any difference between those a time format point of view.

_time is configured with this TIME_FORMAT variable in props.conf. As it's wrongly in your case, it means that your source is not using this props.conf. Where you have set this props.conf?

You already check that source as 

splunk btool props list source::.../license --debug

Basically this should work and at least it is recognized correctly when I create monitor (Settings -> Add Data -> monitor file) on local host. Then it shows those dates correctly on Add Data dashbord.

_time / Time	Event (only time part)
12/2/21 9:10:00.000 AM	02/12/2021 09:10:00, .....

Are this source on your IDX host or somewhere else and collected by UF? And there are no HF between UF and IDX? If there is any full Splunk Enterprise instance between source and your IDX then you must install that props.conf in that host, otherwise it didn't work!