topic Re: Need help with TIME_PREFIX in Splunk Search

Need help with TIME_PREFIX

reswob4 — Fri, 26 Jan 2018 15:24:02 GMT

Given a representative sample of my logs:

Jan 25 14:19:20 1.1.1.1 64: Jan 25 22:19:19.281: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx
Jan 25 14:19:15 1.1.1.1 74: Jan 25 22:19:15.282: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx
Jan 25 14:18:56 1.1.1.1 79: Jan 25 22:18:56.285: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx
Jan 25 14:18:25 1.1.1.1 66: Jan 25 22:18:25.284: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx
Jan 25 14:18:15 1.1.1.1 62: Jan 25 22:18:15.274: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx
Jan 25 14:17:22 1.1.1.1 34: Jan 25 22:17:22.287: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx

These logs are being written to a file on my Heavy Forwarder and the HF is monitoring the file and sending to the indexers. Currently I'm testing a new source on a new HF. The same configuration is being used on another HF and I've just copied the props.conf from the previous HF to the new one. But I'm having some weird behavior when trying to extract the time field.

I want to use the second timestamp as the time. Here is my props.conf:

TIME_PREFIX = \w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s\d+:\s+
MAX_TIMESTAMP_LOOKAHEAD = 15
TIME_FORMAT = %b\s+%d\s+%H:%M:%S.%3N

Splunk shows the time for all events as _time = "Jan 25 2018 5:19:19 PM"

Which is the time of the top event and the timestamp of the file it is reading from. Which makes me thing that Splunk could not parse the timestamp.

I've tried the following TIME_PREFIX modifications:

TIME_PREFIX = ^(?:[^\n]* ) {5}
TIME_PREFIX = \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s\d+:\s+

and I've tried TIME_PREFIX = ^to try to get the first timestamp.

Same issue. Did I make a config mistake, syntax?

I saw other people had found an error that says "Could not use strptime to parse timestamp from xxxxx", but I'm not sure where splunk would write that (splunkd.log?)....

Thanks for any suggestions

Re: Need help with TIME_PREFIX

mayurr98 — Fri, 26 Jan 2018 15:35:07 GMT

hey try this

TIME_FORMAT = %b %d %H:%M:%S.%3N
TIME_PREFIX = \s\d{1,3}:\s

let me know if this helps!

Re: Need help with TIME_PREFIX

reswob4 — Fri, 26 Jan 2018 16:47:12 GMT

Again, this SHOULD work, but doesn't and returns the same result. Is there anywhere in the splunk logs to see where it fails?

Re: Need help with TIME_PREFIX

mayurr98 — Fri, 26 Jan 2018 16:57:41 GMT

have you changed your MAX_TIMESTAMP_LOOKAHEAD = 15 to MAX_TIMESTAMP_LOOKAHEAD = 30 ?
Also try putting the same on indexer as well!

look for warning on search head

index=_internal "Failed to parse timestamp"

Re: Need help with TIME_PREFIX

reswob4 — Fri, 26 Jan 2018 19:53:56 GMT

That didn't work either.

So I copied some logs in to a new file, then ran through the wizard to import those logs into splunk and let splunk create the props.conf.

Here's what Splunk created:

DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = true

That worked.

----- whatever.....

Now the problem is that props.conf is not tied to the syslog autoparsing that splunk does....

problem for another day... I'll update next week...

Re: Need help with TIME_PREFIX

reswob4 — Fri, 26 Jan 2018 20:10:28 GMT

Fixed. Not the most elegant, but it works. Took the sourcetype Splunk created in the above step, found the built in Cisco/syslog config, copied it to the created sourcetype, restarted and it works.

for now.

It's ugly but it works.

EDIT: To clarify, that is the sourcetype Splunk created as I describe in my answer above.

Re: Need help with TIME_PREFIX

Anam — Fri, 26 Jan 2018 20:20:21 GMT

Hey reswob4

If you found the solution yourself, make sure you accept the answer.

Thanks