topic Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index? in Getting Data In

How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

shbagautdinov — Thu, 29 Oct 2015 14:19:54 GMT

Hello, Splunkers!

I'm trying to add a new log file, but I can't extract the correct timestamp.
Help me to write any Timestamp format, which will use date and time from events.
Here in these 3 sample events, timestamp should be 01.09.2015 00:20:05 for the first event,
01.09.2015 00:20:05 for the second event, and so on.

<tr style="height:21px">
<td colspan="3" class="s18-90D19DFDD9934A0F8EEAA283057A16E6">01.09.15</td><td colspan="2" class="s19-90D19DFDD9934A0F8EEAA283057A16E6">00:20:05</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6" style="font-size:1px;background-image:none"> </td><td colspan="2" class="s20-90D19DFDD9934A0F8EEAA283057A16E6">0.039</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">Мб.</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">Мобильный интернет</td><td colspan="2" class="s20-90D19DFDD9934A0F8EEAA283057A16E6" style="font-size:1px;background-image:none"> </td><td colspan="5" class="s21-90D19DFDD9934A0F8EEAA283057A16E6">0.00</td>
</tr>
<tr style="height:21px">
<td colspan="3" class="s22-90D19DFDD9934A0F8EEAA283057A16E6">01.09.15</td><td colspan="2" class="s23-90D19DFDD9934A0F8EEAA283057A16E6">00:26:18</td><td class="s24-90D19DFDD9934A0F8EEAA283057A16E6">900</td><td colspan="2" class="s24-90D19DFDD9934A0F8EEAA283057A16E6">1</td><td class="s24-90D19DFDD9934A0F8EEAA283057A16E6">Шт.</td><td class="s24-90D19DFDD9934A0F8EEAA283057A16E6">Входящее SMS</td><td colspan="2" class="s24-90D19DFDD9934A0F8EEAA283057A16E6" style="font-size:1px;background-image:none"> </td><td colspan="5" class="s25-90D19DFDD9934A0F8EEAA283057A16E6">0.00</td>
</tr>
<tr style="height:21px">
<td colspan="3" class="s18-90D19DFDD9934A0F8EEAA283057A16E6">01.09.15</td><td colspan="2" class="s19-90D19DFDD9934A0F8EEAA283057A16E6">00:26:59</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">900</td><td colspan="2" class="s20-90D19DFDD9934A0F8EEAA283057A16E6">1</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">Шт.</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">Входящее SMS</td><td colspan="2" class="s20-90D19DFDD9934A0F8EEAA283057A16E6" style="font-size:1px;background-image:none"> </td><td colspan="5" class="s21-90D19DFDD9934A0F8EEAA283057A16E6">0.00</td>
</tr>

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

richgalloway — Tue, 29 Sep 2020 07:44:04 GMT

Something to consider is modifying the datetime_config file or, better yet, create a separate datetime_config file for this sourcetype. I should emphasize that this is completely untested.

Copy the existing SPLUNK_HOME/etc/datetime.xml file to SPLUNK_HOME/etc/mydatetime.xml. Add a new define near the bottom of the file.

<define name="mydatetime" extract="month, day, year, hour, minute, second">
    <text><![CDATA[\>(?P<month>[012]\d)\.(?P<day>[012]\d|3[01])\.(?P<year>\d{2})\<.*?\>(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})\<]]></text>
</define>

Then add `to each of thedatePatternsandtimePatterns` stanzas.

In your props.conf file put:

[mysourcetype]
DATETIME_CONFIG = /etc/mydatetime.xml

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

shbagautdinov — Tue, 29 Sep 2020 07:48:00 GMT

Thank you, man!
I have copied the existing SPLUNK_HOME/etc/datetime.xml file to SPLUNK_HOME/etc/megafon.xml.
I have added your code

<define name="megafon" extract="">
     <text><![CDATA[\>(?P<month>[012]\d)\.(?P<day>[012]\d|3[01])\.(?P<year>\d{2})\<.*?\>(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})\<]]></text>
 </define>

<timePatterns>
<use name="megafon"/>
      <use name="_time"/>
      <use name="_hmtime"/>
      <use name="_hmtime"/>
      <use name="_dottime"/>
      <use name="_combdatetime"/>
      <use name="_utcepoch"/>
      <use name="_combdatetime2"/>
</timePatterns>
<datePatterns>
<use name="megafon"/>
      <use name="_usdate1"/> 
      <use name="_usdate2"/> 
      <use name="_isodate"/>
      <use name="_eurodate1"/> 
      <use name="_eurodate2"/> 
      <use name="_bareurlitdate"/> 
      <use name="_orddate"/>
      <use name="_combdatetime"/>
      <use name="_masheddate"/>
      <use name="_masheddate2"/>
      <use name="_combdatetime2"/>
</datePatterns>

</datetime>

I have modified C:\Program Files\Splunk\etc\apps\search\local\props.conf

[Megafon]
DATETIME_CONFIG = /etc/mydatetime.xml 
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true

Source type Megafon was created in Search app context.

And now it is still only date in the timestamp

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

richgalloway — Fri, 30 Oct 2015 12:35:36 GMT

A modified my answer to include field names in the 'extract' clause.

Double-check the DATETIME_CONFIG setting in your props.conf.

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

shbagautdinov — Tue, 29 Sep 2020 07:48:02 GMT

Thanks!
You found my mistake in my props.conf I have wrote DATETIME_CONFIG = /etc/mydatetime.xml instead of DATETIME_CONFIG = /etc/megafon.xml
now name of xml file in etc folder and parameter in DATETIME_CONFIG = are the same
In SPLUNK_HOME/etc/megafon.xml I have specified extract

<define name="megafon" extract="day, month, year, hour, minute, second">
     <text><![CDATA[\>(?P<month>[012]\d)\.(?P<day>[012]\d|3[01])\.(?P<year>\d{2})\<.*?\>(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})\<]]></text>
</define>

<timePatterns>
      <use name="megafon"/>
      <use name="_time"/>
      <use name="_hmtime"/>
      <use name="_hmtime"/>
      <use name="_dottime"/>
      <use name="_combdatetime"/>
      <use name="_utcepoch"/>
      <use name="_combdatetime2"/> 
</timePatterns>
<datePatterns>
      <use name="megafon"/>
      <use name="_usdate1"/> 
      <use name="_usdate2"/> 
      <use name="_isodate"/>
      <use name="_eurodate1"/> 
      <use name="_eurodate2"/> 
      <use name="_bareurlitdate"/> 
      <use name="_orddate"/>
      <use name="_combdatetime"/>
      <use name="_masheddate"/>
      <use name="_masheddate2"/>
      <use name="_combdatetime2"/>
</datePatterns>

</datetime>

But it is still only date in the timestamp

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

richgalloway — Wed, 04 Nov 2015 14:18:23 GMT

I am at a loss. Did you restart Splunk after modifying props.conf?

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

shbagautdinov — Wed, 04 Nov 2015 15:07:37 GMT

Yes, sure. I have restarted my splunk server several times. The log file is on splunk servers local disk.