Splunk Search

Multiple time formats in same data source

DonDandrea
Path Finder

I have a data source I am trying to ingest into Splunk. It is a txt file that is written to by multiple systems. My problem is that each system writing to the file has it's own date format. I have worked out two of the three data sources which have a two digit year The third data source has a four digit year.

Below is what we are currently using in props.conf and a sample of the data.

TIME_FORMAT =%m/%d/%y
TIME_PREFIX=NODE\w{2}\s+;

INTVCICS;08/29/2014 ;23:30 ;1B90;T100 ;1 ;0 ;0 ;0 ;0.608 ;2.659 ;0.000 ;0.000 ;0.000 ;VENDORS ;XXXXX
NODEPR ;08/27/14 ;1D81;ECM_storeDocumentToCI_MF ;Extract Message-Code/Decode ;0.002 ;0.001 ;103 ;XXXXXXXX ;ComputeNode ;XXXXXXX
NODENP ;08/29/14 ;1B90;5:58 ;CM_retrieveClaimHistoryDtlCMS_MF ;SOAP Reply ;0.001 ;0.001 ;25 ;XXXXXXXX ;SOAPReplyNode ;XXXXXXX

Thank you,
Don

Tags (2)
0 Karma
1 Solution

DonDandrea
Path Finder

We reformatted the output data from the source so all event use a two digit year. Once that was complete I was still having a problem with some events having a time and others did not. I simply configured Splunk to index by date and ignore time. Then I created an extract in props.conf for each of the event types. Now I can use eval with striptime in my search parameters and replace the date/time stamp of the events. This now allows use of date and time ranges and timechart.

View solution in original post

0 Karma

DonDandrea
Path Finder

We reformatted the output data from the source so all event use a two digit year. Once that was complete I was still having a problem with some events having a time and others did not. I simply configured Splunk to index by date and ignore time. Then I created an extract in props.conf for each of the event types. Now I can use eval with striptime in my search parameters and replace the date/time stamp of the events. This now allows use of date and time ranges and timechart.

0 Karma

kristian_kolb
Ultra Champion

It would probably be better if you could get the applications to write to individual log files. By giving an explicit TIME_FORMAT that only matches a subset of the events, the others will not parse correctly.

Also, there seems to be no time element in some of the logs, just a date. Perhaps DATETIME_CONFIG = current in props.conf could work for you?

http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Configuretimestamprecognition

/K

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...