topic Re: Splunk Parser Ignore Datetime and Host Fields in Getting Data In

Splunk Parser Ignore Datetime and Host Fields

jmsilva9500 — Wed, 20 May 2020 12:51:09 GMT

Hello,

I'm struggling with finding a parser in splunk for the following log:

May 20 12:22:21 127.0.0.1 {"rootId": "AXIxikL8ao-yaSvA", "requestId": "f6a873jkjjkjk:-8000:5738", "details": {"flag": false, "title": "task 1", "status": "Waiting", "group": "", "order": 0}, "operation": "Creation", "objectId": "AXIyCN5Oao-H5aYyaSvd", "startDate": 1589977341890, "objectType": "case_task", "base": true, "object": {"_routing": "AXIxikL8ao-H5aYyaSvA", "flag": false, "_type": "case_task", "title": "task 1", "createdAt": 1589977341516, "_parent": "AXIxikL8ao-H5aYyaSvA", "createdBy": "user", "_id": "AXIyCN5Oao-H5aYyaSvd", "id": "AXIyCN5Oao-H5aYyaSvd", "_version": 1, "order": 0, "status": "Waiting", "group": ""}}

The log itself is a valid json, and i can parse it well with the default _json parser. However, splunk inserts the datetime and hostname at the beginning of the log, which makes the parser stop working..

Is there any workaround for this?

Thanks!

Re: Splunk Parser Ignore Datetime and Host Fields

richgalloway — Wed, 20 May 2020 14:17:28 GMT

The parser works, but it doesn't work. Please explain.
How is the data getting to Splunk?

Re: Splunk Parser Ignore Datetime and Host Fields

jmsilva9500 — Wed, 20 May 2020 14:30:31 GMT

The data is being sent by remote syslog. the _json parser works for the log that is sent to splunk, but at indexing time splunk inserts at the beginning of the log the datetime and the host values, which makes the log itself no longer a valid json, therefore the parser fails

Re: Splunk Parser Ignore Datetime and Host Fields

richgalloway — Wed, 20 May 2020 16:35:13 GMT

I think it's syslog that is adding the time and host values. Splunk also adds timestamp and host, but it's in metadata, not the raw data.

Re: Splunk Parser Ignore Datetime and Host Fields

jmsilva9500 — Wed, 20 May 2020 17:21:11 GMT

Do you know how i can parse to log giving that the time and host values are in the beggining of it?

Re: Splunk Parser Ignore Datetime and Host Fields

richgalloway — Wed, 20 May 2020 18:19:38 GMT

Try using SEDCMD in props.conf to strip out the timestamp and host info. Should be easy with "s/^.*\{//".

Re: Splunk Parser Ignore Datetime and Host Fields

to4kawa — Sat, 23 May 2020 23:39:43 GMT

default _json parser is INDEXED_EXTRACTION=json
This can't work on using SEDCMD

props.conf

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
INDEXED_EXTRACTION = none
KV_MODE = json
TRUNCATE = 0
SEDCMD-trim = s/.*?{/{/