Getting Data In

Non-Padded Milliseconds on Timestamp

darrenfuller
Contributor

Hello..

Splunk 7.0.5, I have a data source as follows, which has 1, 2 or 3 digit values for millisecond. :

Tue Oct 08 2019 14:47:33 tid="d83af63f5acd9c510bd440d" object="EnterpriseSalesInformation" rule_type="request" domain="SOAPvs" domain="SOAPvs" trace_id="30fb2039-9910-4bd5-b012-3fe63dd874af" time="2019-10-08T14:47:33:4-04:00" 
Tue Oct 08 2019 14:47:29 tid="d83af63f5acd9c180b220fd" object="EnterpriseSalesInformation" rule_type="request" domain="SOAPvs" domain="SOAPvs" trace_id="24c792c0-80ea-476b-95b2-3fe63dd8876c" time="2019-10-08T14:47:29:91-04:00" 
Tue Oct 08 2019 14:47:29 tid="d83af63f5acd9c180bd203d" object="EnterpriseSalesInformation" rule_type="request" domain="SOAPvs" domain="SOAPvs" trace_id="7446787d-76b8-4eca-924f-3fe63dd8c9ed" time="2019-10-08T14:47:29:181-04:00" 

Which i am trying to process using this props.conf:

[teshting]
disabled = false
DATETIME_CONFIG = /etc/apps/teshtapp/local/customdatetime.xml
TIME_PREFIX = time="
# TIME_FORMAT = %Y-%m-%dT%H:%M:%S:%3Q%z
# TIME_FORMAT = %Y-%m-%dT%H:%M:%S:%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 40
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false
TRUNCATE = 1000

( You can see from my commented lines, i tried to deal with this using %3Q and %3N to no avail, both caused a ms value of 4 to be 400 and 91 to be 910 )

In reading other comments on here, i think i need to use a custom datetime.xml file to handle this... so i wrote customdatetime.xml referenced above in my props.conf and detailed here:

<datetime>
    <define name="test_dateformat" extract="year, month, day">
        <text><![CDATA[(\d{4})-(\d{2})-(\d{2})]]></text>
    </define>
    <define name="test_timeformat" extract="hour, minute, second, subsecond, zone">
        <text><![CDATA[(\d{2}):(\d{2}):(\d{2}):(\d+)(-\d{2}:\d{2})]]></text>
    </define>
    <timePatterns>
        <use name="test_timeformat" />
    </timePatterns>
    <datePatterns>
        <use name="test_dateformat" />
    </datePatterns>
</datetime>

That does not seem to work either. Where am i going wrong?

0 Karma
1 Solution

darrenfuller
Contributor

I have decided (due to the lack of answers) that this is in fact not possible.

View solution in original post

0 Karma

darrenfuller
Contributor

I have decided (due to the lack of answers) that this is in fact not possible.

0 Karma

ivanreis
Builder

Hi darrenfuller, I was reviewing the splunk docs and I would like to suggest the configuration below:

TIME_FORMAT = %Y-%m-%dT%H:%M:%S:%3Q%:z
Where : %:z is to specify hour and minute separated by a colon

OR

TIME_FORMAT = %Y-%m-%dT%H:%M:%S:%Q%:z

%z The timezone offset from UTC, in hour and minute: +hhmm or -hhmm. For example, for 5 hours before UTC the values is -0500 which is US Eastern Standard Time.
Examples:

Use %z to specify hour and minute, for example -0500
Use %:z to specify hour and minute separated by a colon, for example -5:00
Use %::z to specify hour minute and second separated with colons, for example -05:00:00
Use %:::z to specify hour only, for example -05

Also comment #MAX_TIMESTAMP_LOOKAHEAD = 40, because maybe splunk is looking for the date/time on a position where this data is not found, because generally the date/time is being on the begin of the log. This is only a suggestion for troubleshooting. If it works, you can try to play with numbers to identify the right characters positions to get this data properly indexed and re-enable it again.

MAX_TIMESTAMP_LOOKAHEAD = Specify how far (how many characters) into an event Splunk software should look for a timestamp.
This constraint is applied starting from the location positioned by TIME_PREFIX.

For example, if TIME_PREFIX positions a location 11 characters into the event, and MAX_TIMESTAMP_LOOKAHEAD is set to 10, timestamp extraction will be constrained to characters 11 through 20.

If set to 0 or -1, the length constraint for timestamp recognition is effectively disabled. This can have negative performance implications which scale with the length of input lines (or with event size when LINE_BREAKER is redefined for event splitting).

I hope this can assist you to fix your issue.

0 Karma

darrenfuller
Contributor

Thanks for your comment. I tried both of these configs and neither worked. The result was the same for both of the examples above with < 3 digits on the millisecond...

"2019-10-08T14:47:33:4-04:00" was transformed into "2019-10-08T14:47:33:400-04:00"

and

"2019-10-08T14:47:29:91-04:00" was transformed into ""2019-10-08T14:47:29:910-04:00"

0 Karma

darrenfuller
Contributor

and my TIME_PREFIX is time=" which only occurs towards the end of these events . if you start there... 40 characters is more than enough to match these timestamps (which are 27-29 characters in length)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...