topic Re: Multiple time formats in same data source in Splunk Search

Multiple time formats in same data source

DonDandrea — Mon, 28 Sep 2020 17:27:36 GMT

I have a data source I am trying to ingest into Splunk. It is a txt file that is written to by multiple systems. My problem is that each system writing to the file has it's own date format. I have worked out two of the three data sources which have a two digit year The third data source has a four digit year.

Below is what we are currently using in props.conf and a sample of the data.

TIME_FORMAT =%m/%d/%y
TIME_PREFIX=NODE\w{2}\s+;

INTVCICS;08/29/2014 ;23:30 ;1B90;T100 ;1 ;0 ;0 ;0 ;0.608 ;2.659 ;0.000 ;0.000 ;0.000 ;VENDORS ;XXXXX
NODEPR ;08/27/14 ;1D81;ECM_storeDocumentToCI_MF ;Extract Message-Code/Decode ;0.002 ;0.001 ;103 ;XXXXXXXX ;ComputeNode ;XXXXXXX
NODENP ;08/29/14 ;1B90;5:58 ;CM_retrieveClaimHistoryDtlCMS_MF ;SOAP Reply ;0.001 ;0.001 ;25 ;XXXXXXXX ;SOAPReplyNode ;XXXXXXX

Thank you,
Don

Re: Multiple time formats in same data source

kristian_kolb — Tue, 02 Sep 2014 13:49:13 GMT

It would probably be better if you could get the applications to write to individual log files. By giving an explicit TIME_FORMAT that only matches a subset of the events, the others will not parse correctly.

Also, there seems to be no time element in some of the logs, just a date. Perhaps DATETIME_CONFIG = current in props.conf could work for you?

http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Configuretimestamprecognition

Re: Multiple time formats in same data source

DonDandrea — Thu, 18 Sep 2014 12:45:00 GMT

We reformatted the output data from the source so all event use a two digit year. Once that was complete I was still having a problem with some events having a time and others did not. I simply configured Splunk to index by date and ignore time. Then I created an extract in props.conf for each of the event types. Now I can use eval with striptime in my search parameters and replace the date/time stamp of the events. This now allows use of date and time ranges and timechart.