topic Re: sourcetype TIME_FORMAT changed in Getting Data In

sourcetype TIME_FORMAT changed

gballanti — Tue, 03 Dec 2019 07:57:46 GMT

hello to everyone,

the monthly logs received from the ivr has changed the time format. Until now it was %d/%m%/Y right now it is %m/%d/%Y .
Is it possible to modify the TIME_FORMAT so the new data will be recognized leaving the old ones unchanged or a global re-index for all logs is needed ?

Thanks in advanced

Re: sourcetype TIME_FORMAT changed

DavidHourani — Tue, 03 Dec 2019 12:12:21 GMT

Hi @gballanti,

Time extraction occurs at index time which means that the _time is an indexed field. By definition when you change the configuration of such a field only newly indexed fields will show change. Older logs will keep the originally defined format.

This works at your advantage because it's exactly what you're looking for. So all you have to do is make sure you set your TIME_FORMATwhere the data gets indexed (Indexers or HF depending on your setup). Then all new data will adapt to that new format.

Let me know if that helps.

Cheers,
David

Re: sourcetype TIME_FORMAT changed

gballanti — Tue, 03 Dec 2019 12:46:55 GMT

Hi David,

thank you for the detailed answer.

I have just a trouble about the indexed data (old and new) because the chart built on this data is grouped by quarter. In this last quarter will have the indexed date (_time) with month and day inverted.

Is it able to understand the right date ? For example 11/10 with old log means 11 october while with new log means 10 november, those situation a recognized and managed ?

Thanks

Re: sourcetype TIME_FORMAT changed

DavidHourani — Tue, 03 Dec 2019 13:55:08 GMT

your old data is indexed with the previously defined TIME_FORMAT so when your new data arrives under the new TIME_FORMAT the _time field will be extracted properly so you will not have any issues charting over both quarters.

You will only face issues if you have already indexed your new data with old TIME_FORMAT. In that case the only way to change the format is to delete then reindex this data because as I mentioned in the answer _time is an index time field so it's not something you can change after indexing.

Does that answer your question ? Let me know if it's not clear enough, happy to clarify 😄

Re: sourcetype TIME_FORMAT changed

gballanti — Tue, 03 Dec 2019 14:02:32 GMT

Thanks David, it is just what I wanted like to hear. I haven't indexed the new logs yet.

Cheers,
Giuseppe

Re: sourcetype TIME_FORMAT changed

DavidHourani — Tue, 03 Dec 2019 14:05:43 GMT

@gballanti you're welcome!