Time pulled from log file is sometimes incorrect

[ak1508]

Hi, I am new to using Splunk and am currently experimenting on my desktop using a few different add-ons.

I have been using the BT HomeHub app, which although doesn't seem to extract the log files from my newer BT SmartHub, does manage to perform regular speed tests, which has given me some nice data to play with.

That is until 14:00 passes each day when the way the time/date in log file is parsed by Splunk seems to fail.

Before 2pm, Time field populated correctly:

After 2pm, Time field populated incorrectly:

Is this bug down to the way the App is coded, or is there a setting in Splunk I can change.

Alternatively is there a way I can make my own 'time' field? I have managed to extract the individual parts of the date to individual fields but do not know how to combine these into a time field that Splunk can use e.g. in a timechart.

Thanks, Andy

Edit:
It seems the line in the log file is being created by a bash script called speedtest.sh:

#!/bin/bash

echo $(date) > /tmp/st.res |python /home/andy/Splunk/splunk/etc/apps/BTHomeHub/bin/speedtest.py |grep load: >> /tmp/st.res
cat /tmp/st.res |xargs -n3 -d'\n'

So the date format is the standard bash format (though it does seem odd the month and year are separated by the time). Does Splunk handle the conversion or is that configured in the app?

Edit 2:
So it turns out you can show Splunk what format to expect the timestamp to be in for different sourcetypes.

The setting is found on the 'Source Type' page, access from the drop down settings menu under the Data heading. You can then search for the relevant sourcetype (in my case "bt:homehub:speedtest"). Choose the 'edit' option and go to the 'Timestamp' tab. The expected format of the date can then be entered in the 'Timestamp Format' input (For me %a %d %b %H:%M:%S %Z %Y as suggested by richgallloway). This is known as strptime() format and I believe is based on a Python standard.

This caused all timestamps of future log entries to be processed correctly, but did not alter past entries.

[richgalloway · Accepted solution]

Try adding a time format to the props.conf settings.

TIME_FORMAT = %a %d %b %H:%M:%S %Z %Y

[ak1508]

Thank you for your help, much appreciated.

[richgalloway]

Check the TIME_FORMAT settings in the app's props.conf file .

[ak1508]

There isn't a TIME_FORMAT in the props.conf.

It looks like the last line of that file does correspond to the line of the log though:

EXTRACT-download_value,download_unit,upload_value,upload_unit = Download: (?P<download_value>[^ ]+) (?P<download_unit>[^ ]+) Upload: (?P<upload_value>[^ ]+) (?P<upload_unit>[^ ]+)