Getting Data In

Timestamp recognition of multi timestamp formats in one Logfile

tobiasvollrath
Explorer

Hey,
I have a problems with the timestamp recognition at index time.
How can Splunk recognize different time/date formats in one logfile?

For example:

2011-09-13T15:39:21.188 .....

15:39:21.188 .........

@15:39:22.3763 ......

Thank you!

Tags (1)
1 Solution

kristian_kolb
Ultra Champion

Why would you have different types of timestamp in a single log file?

If the reason is "well, for troubleshooting purposes we redirect the application log, STDERR, STDOUT and <insert_log_file_here> to the same file", then you should probably rethink. When logging with Splunk there is no need to keep different types of event in a single file in order to establish a sequence of events, since you will be viewing/searching/reporting on separate messages, regardless of which file they came from.

Also, a lot of the processing (field extractions/transforms/props etc) will in my experience be based on SOURCETYPES.

Just because it used to make sense to pipe several logs into one file, it is not the best way to work with Splunk.

Sorry for the ranting. As for what you can/should do about it, there are a few options (from best to worst);

  1. split the logging into several files, so that the timestamps are uniform within each file. Set the files to have appropriate sourcetypes. If this is at all possible - do that. It will save you a lot of headache.
  2. rewrite/reconfigure the logging functions of whatever applications/processes write to the file, so that all timestamps are uniform.
  3. use some sort of wrapper that will add an extra timestamp outside the original log message for all events. Don't forget to configure your props.conf that the correct timestamp is always read.

Hope this helps,

Kristian

View solution in original post

0 Karma

tobiasvollrath
Explorer

At first,thank you for your help.

But unfortunately we are not able to transact any one of these 3 possibilities.
Is where no possibility to edit the datetime.xml to extract multi timestamp formats?

This is an extract of our Logfiles (SIP Logfile):

gsip:TRN[303294]:05:41:29.569 TIMER SET [id=631242823,timeout=4000,arg=1]
gsip:TRN[303295]:05:41:29.580 TIMER FIRED [id=979370087,arg=1]
gsip:TRN[303295]: Timer E at state 1
gsip:TRN[303295]:[CTNI;R:1;F:0x0]:05:41:29.580: 1 >> 1 [E:1,T:1]
05:41:29.580:(1) Sending  [19,UDP] 433 bytes to 10.xxx.44.43:5070 >>>>>

2011-09-29T18:14:11.539 Std 21016 Location SCP disconnected
@18:14:11.5391 [ISCC] Message EventLocationInfo [Event (To Client):0000003b] is enqueued
@18:14:11.5391 [ISCC] {server TServer_SCP_9@SCP {role backup} {connecting}}
@18:14:11.5391 [ISCC] [connection b6da168] connecting [0]1
@18:14:11.5559 [ISCC] Message EventLocationInfo [Event (To Client):0000003a] is dequeued
@18:14:11.5560 [0] 8.0.400.62 send_to_client: message EventLocationInfo
    AttributeEventSequenceNumber    00000000000000a8
    AttributeTimeinuSecs    555999
    AttributeTimeinSecs 1317312851 (18:14:11)
    AttributeExtensions [76] 00 03 00 00..
        'LQ-location-name'  'SCP'
        'LQ-location-status'    1
        'LQ-link-status'    0
    AttributeLocationInfoType   1
2011-09-29T18:14:11.556 Trc 04542 EventLocationInfo sent to [15] (00000005 Router_9 164.25.118.74:1495)
@18:14:11.5560 [ISCC] Message [Event (To Client):0000003a] is deleted

kristian_kolb
Ultra Champion

Why would you have different types of timestamp in a single log file?

If the reason is "well, for troubleshooting purposes we redirect the application log, STDERR, STDOUT and <insert_log_file_here> to the same file", then you should probably rethink. When logging with Splunk there is no need to keep different types of event in a single file in order to establish a sequence of events, since you will be viewing/searching/reporting on separate messages, regardless of which file they came from.

Also, a lot of the processing (field extractions/transforms/props etc) will in my experience be based on SOURCETYPES.

Just because it used to make sense to pipe several logs into one file, it is not the best way to work with Splunk.

Sorry for the ranting. As for what you can/should do about it, there are a few options (from best to worst);

  1. split the logging into several files, so that the timestamps are uniform within each file. Set the files to have appropriate sourcetypes. If this is at all possible - do that. It will save you a lot of headache.
  2. rewrite/reconfigure the logging functions of whatever applications/processes write to the file, so that all timestamps are uniform.
  3. use some sort of wrapper that will add an extra timestamp outside the original log message for all events. Don't forget to configure your props.conf that the correct timestamp is always read.

Hope this helps,

Kristian

0 Karma

mookiie2005
Communicator

this is a horrible response to the question asked.

0 Karma

JoeIII
Path Finder

Sometimes you do not have control over the entries or timestamps used in log files. While I agree that it is stupid to have the same file with different messages with different timestamp formats, sometimes developers do things that are stupid and we have to work around those limitations.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...