Getting Data In

Why isn't the timestamp being recognized ?

ddrillic
Ultra Champion

We have a case that looks like this -

alt text

So, the events are not broken by the timestamp. Is it because of the underscore between the date and time?

Tags (1)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

Splunk has a number of built in time formats, I would recommend you manually parse your timestamps to improve indexing performance and ensure that you have the timestamps consistently recognised.

In this case in your props.conf

[replace_with_your_sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d_%H:%M:%S.%3N

Also refer to configure timestamp recognition. I agree with skoelpin it's probably the _ throwing the default timestamp parsing off.

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

View solution in original post

0 Karma

gjanders
SplunkTrust
SplunkTrust

Splunk has a number of built in time formats, I would recommend you manually parse your timestamps to improve indexing performance and ensure that you have the timestamps consistently recognised.

In this case in your props.conf

[replace_with_your_sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d_%H:%M:%S.%3N

Also refer to configure timestamp recognition. I agree with skoelpin it's probably the _ throwing the default timestamp parsing off.

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

View solution in original post

0 Karma

ddrillic
Ultra Champion

Btw, this by itself did it ; -) with no line breaking

0 Karma

ddrillic
Ultra Champion

Gorgeous - thank you @garethatiagg

skoelpin
SplunkTrust
SplunkTrust

Remember that this is a start and you should apply line breaking to your props.conf aswell.. This will take some load off your indexers

gjanders
SplunkTrust
SplunkTrust

The documentation also covers line breaking

This is a slightly more advanced topic but potentially worth discussing here, the default for the MAX_EVENTS in props.conf is 256 lines at the time of writing, if you always have single line events you could switch SHOULD_LINEMERGE=false on and never line merge (this entry would go in the props.conf)

Alternatively if you have very large events (>1000 lines) you could make a more complicated LINE_BREAKER setting and change SHOULD_LINEMERGE to false such as:

[replace_with_your_sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d_%H:%M:%S.%3N
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}_\d

Of course the above would require testing, and it would only make sense if you have larger multi-line events (I've used examples like the above for events >1000 lines), doing this can remove pressure from the aggregation queue as the merging will move to the parsing queue...

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

skoelpin
SplunkTrust
SplunkTrust

Most likely the underscores are throwing it off. What's your props.conf look like?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!