topic Re: Why is Splunk not parsing the correct datetime? in Getting Data In

Why is Splunk not parsing the correct datetime?

dpanych — Thu, 30 Jun 2016 21:06:05 GMT

I have logs that contain the following datetime format:

29-06-2016_00-08-17

The props contain:

[odb]
TIME_PREFIX = ".+",".+","
TIME_FORMAT = %d-%m-%Y_%H-%M-%S
MAX_TIMESTAMP_LOOKAHEAD = 50
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
disabled = false
pulldown_type = true
SEDCMD-01_change_delims_in_oracle_logs = s/\|;\|/,/g
SEDCMD-02_remove_the_end = s/"\|\|\?--END---\?\|\|//g
REPORT-set_delimiters_oracle_logs = REPORT-delims_odb_logs

What could be the problem?

Re: Why is Splunk not parsing the correct datetime?

maciep — Fri, 01 Jul 2016 00:41:33 GMT

Not sure if I'm reading that correctly, but are you replacing the delimiters in the raw text from "|;|" to ","? If so, is that the same comma you're expecting to be there for the time prefix? If it is, I'm wondering if the timestamp recognition happens before sedcmd is applied?

Also, be sure 50 characters is enough to look ahead (assuming you did, but thought I'd mention it).

Re: Why is Splunk not parsing the correct datetime?

gcusello — Fri, 01 Jul 2016 06:40:30 GMT

I found a similar problem: time recognition is done before the character replacing.
Bye.
Giuseppe

Re: Why is Splunk not parsing the correct datetime?

jplumsdaine22 — Fri, 01 Jul 2016 10:38:19 GMT

Can you post a sample event as well?

Re: Why is Splunk not parsing the correct datetime?

woodcock — Fri, 01 Jul 2016 13:45:57 GMT

We cannot say without you posting a sample event.

Re: Why is Splunk not parsing the correct datetime?

dpanych — Fri, 01 Jul 2016 15:27:41 GMT

Here's what the logs looks like after the sed:

"host1","MON","LOGOFF","30-06-2016_11-15-01","","0"
"host2","ODS","UPDATE","30-06-2016_12-51-05","UPDATE DS_ATTRSTORE SET ATTRVAL = :B1 WHERE ENTRY = :B2 AND ATTRNAME = 'modname'","0"
"host3","ODS","UPDATE","30-06-2016_08-28-43","UPDATE DS_ATTRSTORE SET ATTRVER = :B4 || CHR(94) || :B5 || CHR(94) || :B6 , ATTRVAL = :B3 , ATTRKIND = :B2 , ATTRSTYPE = :B1 WHERE ENTRY = :B8 AND ATTRNAME = :B7","0"

Re: Why is Splunk not parsing the correct datetime?

dpanych — Fri, 01 Jul 2016 15:41:13 GMT

So I updated the time prefix to include the delims before the sed, TIME_PREFIX = ".+"|;|".+"|;|".+"|;|"
and still no luck.

Re: Why is Splunk not parsing the correct datetime?

woodcock — Fri, 01 Jul 2016 15:49:18 GMT

Your TIME_PREFIX is wrong. Try these:

TIME_PREFIX = ("[^"]*",){3}"
MAX_TIMESTAMP_LOOKAHEAD = 19

Re: Why is Splunk not parsing the correct datetime?

jplumsdaine22 — Fri, 01 Jul 2016 15:54:49 GMT

Does it make a difference if you use a capture group instead of a non capture group there? As in:

(?:"[^"]*",){3}"

Re: Why is Splunk not parsing the correct datetime?

woodcock — Fri, 01 Jul 2016 15:59:53 GMT

In this case, they are the same.

Re: Why is Splunk not parsing the correct datetime?

woodcock — Fri, 01 Jul 2016 16:02:06 GMT

Also, this needs to be deployed to your indexers, the splunk instances must be restarted, and only newly-indexed (post-restart) data will be effected (bad events will stay bad).

Re: Why is Splunk not parsing the correct datetime?

dpanych — Fri, 01 Jul 2016 16:11:34 GMT

I'm no regex guru, but how do I make the regex above work with the |;| delims? ---- "aaa"|;|"OIM"|;|"DELETE"|;|"29-06-2016_01-53-16"

Re: Why is Splunk not parsing the correct datetime?

jplumsdaine22 — Fri, 01 Jul 2016 16:15:06 GMT

Just modify the regex to pick the delimiters instead of a comma

 ("[^"]*"\|;\|){3}"

Re: Why is Splunk not parsing the correct datetime?

gcusello — Fri, 01 Jul 2016 16:18:11 GMT

you have to use \ before " so the prefix is
\"\w+\",\"\w+\",\"\w+\",\"

Bye.
Giuseppe

Re: Why is Splunk not parsing the correct datetime?

dpanych — Fri, 01 Jul 2016 16:19:54 GMT

Thank you guys, this worked!

Re: Why is Splunk not parsing the correct datetime?

jplumsdaine22 — Fri, 01 Jul 2016 16:22:21 GMT

https://regex101.com/ is your friend