topic Can you extract JSON syslog automatically? in Getting Data In

Can you extract JSON syslog automatically?

tkwaller — Tue, 29 Sep 2020 14:29:40 GMT

Trying to get my syslog in json format to extract properly.

I've tried using INDEXED_EXTRACTIONS=JSON as well as KV_MODE=json(not at the same time) and neither have worked.

Here is a short sample log

Jun 13 15:41:27 host.domain.com {"account_id":"678", "legacy_domain_id":"12345", "visitor_ip":"xx.xxx.xxx.xx", "time":"1497368487.007","request":"GET /stuff/search/stuff/morestuff"}

Unfortunately it doesnt seem to be working as nothing gets extracted.
I'm sure its something I'm doing or something with the log format.
Any help would be apprciated.
Thanks!

Re: Can you extract JSON syslog automatically?

masonmorales — Tue, 29 Sep 2020 14:29:45 GMT

JSON will not auto-extract with the timestamp/host prefix in your data. You can use a SEDCMD to strip it out in props.conf.

e.g.

[my_sourcetype]
SEDCMD-strip_prefix = s/^[^{]+//g
INDEXED_EXTRACTIONS=JSON
KV_MODE=none

You'll want to add TIME_PREFIX, TIME_FORMAT, etc. as well.

Re: Can you extract JSON syslog automatically?

tkwaller — Tue, 29 Sep 2020 14:31:33 GMT

Hello
Sorry for the delay, was super busy with work.
I knew it had to be something with the preceding data but wasn't sure. I added to the config and now it looks as such:
[syslogtest]
SEDCMD-strip_prefix = s/^[^{]+//g
INDEXED_EXTRACTIONS=JSON
KV_MODE=none
DATETIME_CONFIG =
TIME_FORMAT = %s.%6N
TIME_PREFIX = "time":"
NO_BINARY_CHECK = true
category = Custom
description = syslogtest
disabled = false
pulldown_type = true

I left the default stuff there.

Only now parsing doesnt take place:
06-19-2017 09:36:04.310 -0500 ERROR JsonLineBreaker - JSON StreamId:5377901818726410745 had parsing error:Unexpected character while looking for value: 'J' - data_source="C:\syslog.log", data_host="L-BDL-10007862", data_sourcetype="syslogtest"

Any thoughts?
Thanks!

Re: Can you extract JSON syslog automatically?

tkwaller — Mon, 19 Jun 2017 16:18:39 GMT

So it seems it wouldn't work with:
INDEXED_EXTRACTIONS=JSON

but it DID work with:
KV_MODE=json

Thanks for the help, I appreciate it!

Re: Can you extract JSON syslog automatically?

andygerber — Thu, 28 Sep 2017 21:35:45 GMT

Thank you, just what I needed....

Re: Can you extract JSON syslog automatically?

darlas — Fri, 23 Feb 2018 21:01:26 GMT

Hi.

I'm facing the same issue.

I've added the following to my props.conf but still not working.

SEDCMD-StripHeader = s/^[^{]+//g
KV_MODE=json

Does the associated TA need to be pushed to indexers as well as search heads?

Does there need to be a reference later in props.conf to the SEDCMD-StripHeader line?

Thanks!