Getting Data In
Highlighted

How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

Path Finder

Hello, Splunkers!

I'm trying to add a new log file, but I can't extract the correct timestamp.
Help me to write any Timestamp format, which will use date and time from events.
Here in these 3 sample events, timestamp should be 01.09.2015 00:20:05 for the first event,
01.09.2015 00:20:05 for the second event, and so on.

<tr style="height:21px">
<td colspan="3" class="s18-90D19DFDD9934A0F8EEAA283057A16E6">01.09.15</td><td colspan="2" class="s19-90D19DFDD9934A0F8EEAA283057A16E6">00:20:05</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6" style="font-size:1px;background-image:none"> </td><td colspan="2" class="s20-90D19DFDD9934A0F8EEAA283057A16E6">0.039</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">Мб.</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">Мобильный интернет</td><td colspan="2" class="s20-90D19DFDD9934A0F8EEAA283057A16E6" style="font-size:1px;background-image:none"> </td><td colspan="5" class="s21-90D19DFDD9934A0F8EEAA283057A16E6">0.00</td>
</tr>
<tr style="height:21px">
<td colspan="3" class="s22-90D19DFDD9934A0F8EEAA283057A16E6">01.09.15</td><td colspan="2" class="s23-90D19DFDD9934A0F8EEAA283057A16E6">00:26:18</td><td class="s24-90D19DFDD9934A0F8EEAA283057A16E6">900</td><td colspan="2" class="s24-90D19DFDD9934A0F8EEAA283057A16E6">1</td><td class="s24-90D19DFDD9934A0F8EEAA283057A16E6">Шт.</td><td class="s24-90D19DFDD9934A0F8EEAA283057A16E6">Входящее SMS</td><td colspan="2" class="s24-90D19DFDD9934A0F8EEAA283057A16E6" style="font-size:1px;background-image:none"> </td><td colspan="5" class="s25-90D19DFDD9934A0F8EEAA283057A16E6">0.00</td>
</tr>
<tr style="height:21px">
<td colspan="3" class="s18-90D19DFDD9934A0F8EEAA283057A16E6">01.09.15</td><td colspan="2" class="s19-90D19DFDD9934A0F8EEAA283057A16E6">00:26:59</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">900</td><td colspan="2" class="s20-90D19DFDD9934A0F8EEAA283057A16E6">1</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">Шт.</td><td class="s20-90D19DFDD9934A0F8EEAA283057A16E6">Входящее SMS</td><td colspan="2" class="s20-90D19DFDD9934A0F8EEAA283057A16E6" style="font-size:1px;background-image:none"> </td><td colspan="5" class="s21-90D19DFDD9934A0F8EEAA283057A16E6">0.00</td>
</tr>
0 Karma
Highlighted

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

SplunkTrust
SplunkTrust

Something to consider is modifying the datetimeconfig file or, better yet, create a separate datetimeconfig file for this sourcetype. I should emphasize that this is completely untested.

Copy the existing SPLUNKHOME/etc/datetime.xml file to SPLUNKHOME/etc/mydatetime.xml. Add a new define near the bottom of the file.

<define name="mydatetime" extract="month, day, year, hour, minute, second">
    <text><![CDATA[\>(?P<month>[012]\d)\.(?P<day>[012]\d|3[01])\.(?P<year>\d{2})\<.*?\>(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})\<]]></text>
</define>

Then add `to each of thedatePatternsandtimePatterns` stanzas.

In your props.conf file put:

[mysourcetype]
DATETIME_CONFIG = /etc/mydatetime.xml
---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

Path Finder

Thank you, man!
I have copied the existing SPLUNKHOME/etc/datetime.xml file to SPLUNKHOME/etc/megafon.xml.
I have added your code

<define name="megafon" extract="">
     <text><![CDATA[\>(?P<month>[012]\d)\.(?P<day>[012]\d|3[01])\.(?P<year>\d{2})\<.*?\>(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})\<]]></text>
 </define>

<timePatterns>
<use name="megafon"/>
      <use name="_time"/>
      <use name="_hmtime"/>
      <use name="_hmtime"/>
      <use name="_dottime"/>
      <use name="_combdatetime"/>
      <use name="_utcepoch"/>
      <use name="_combdatetime2"/>
</timePatterns>
<datePatterns>
<use name="megafon"/>
      <use name="_usdate1"/> 
      <use name="_usdate2"/> 
      <use name="_isodate"/>
      <use name="_eurodate1"/> 
      <use name="_eurodate2"/> 
      <use name="_bareurlitdate"/> 
      <use name="_orddate"/>
      <use name="_combdatetime"/>
      <use name="_masheddate"/>
      <use name="_masheddate2"/>
      <use name="_combdatetime2"/>
</datePatterns>

</datetime>

I have modified C:\Program Files\Splunk\etc\apps\search\local\props.conf

[Megafon]
DATETIME_CONFIG = /etc/mydatetime.xml 
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true

Source type Megafon was created in Search app context.

And now it is still only date in the timestamp
alt text

0 Karma
Highlighted

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

SplunkTrust
SplunkTrust

A modified my answer to include field names in the 'extract' clause.

Double-check the DATETIME_CONFIG setting in your props.conf.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

Path Finder

Thanks!
You found my mistake in my props.conf I have wrote DATETIMECONFIG = /etc/mydatetime.xml instead of DATETIMECONFIG = /etc/megafon.xml
now name of xml file in etc folder and parameter in DATETIMECONFIG = are the same
In SPLUNK
HOME/etc/megafon.xml I have specified extract

<define name="megafon" extract="day, month, year, hour, minute, second">
     <text><![CDATA[\>(?P<month>[012]\d)\.(?P<day>[012]\d|3[01])\.(?P<year>\d{2})\<.*?\>(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})\<]]></text>
</define>

<timePatterns>
      <use name="megafon"/>
      <use name="_time"/>
      <use name="_hmtime"/>
      <use name="_hmtime"/>
      <use name="_dottime"/>
      <use name="_combdatetime"/>
      <use name="_utcepoch"/>
      <use name="_combdatetime2"/> 
</timePatterns>
<datePatterns>
      <use name="megafon"/>
      <use name="_usdate1"/> 
      <use name="_usdate2"/> 
      <use name="_isodate"/>
      <use name="_eurodate1"/> 
      <use name="_eurodate2"/> 
      <use name="_bareurlitdate"/> 
      <use name="_orddate"/>
      <use name="_combdatetime"/>
      <use name="_masheddate"/>
      <use name="_masheddate2"/>
      <use name="_combdatetime2"/>
</datePatterns>

</datetime>

But it is still only date in the timestamp

0 Karma
Highlighted

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

SplunkTrust
SplunkTrust

I am at a loss. Did you restart Splunk after modifying props.conf?

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: How do I configure Splunk to extract the timestamp from a new log file I'm trying to index?

Path Finder

Yes, sure. I have restarted my splunk server several times. The log file is on splunk servers local disk.

0 Karma