topic How to configure Splunk to use a specific JSON field as the event's timestamp? in Getting Data In

How to configure Splunk to use a specific JSON field as the event's timestamp?

jchoksi — Wed, 05 Feb 2020 20:55:36 GMT

For the following example JSON message (formated to make it easier to read), how can I configure props.conf to inform Splunk that it should use data.timestamp for its event timestamp?

{
    "publish_time": 1580824871.446,
    "data": {
        "textPayload": "DEBUG | 2020-02-04T14:01:05,760 | A very long string here...<snip>",
        "logName": "blah0",
        "receiveTimestamp": "2020-02-04T14:01:07.707699223Z",
        "labels": {
            "k8s-pod/version": "blah2",
            "k8s-pod/track": "blah3",
            "k8s-pod/app": "blah4",
            "k8s-pod/pod-template-hash": "blah5"
        },
        "insertId": "blah6",
        "resource": {
            "type": "k8s_container",
            "labels": {
                "project_id": "blah7",
                "pod_name": "blah8",
                "cluster_name": "blah9",
                "location": "blah10",
                "container_name": "blah11",
                "namespace_name": "blah12"
            }
        },
        "severity": "INFO",
        "timestamp": "2020-02-04T14:01:05.760888513Z"
    },
    "attributes": {
        "logging.googleapis.com/timestamp": "2020-02-04T14:01:05.760888513Z"
    }
}

Would the following be correct & performant ?

File: props.conf
---snip---
[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = data.timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z
---snip---

Re: How to configure Splunk to use a specific JSON field as the event's timestamp?

manjunathmeti — Wed, 30 Sep 2020 04:07:37 GMT

Yes configuration you posted works. Also please set KV_MODE = none and AUTO_KV_JSON = false for this source/sourcetype on search head(s) to avoid multi values in fields.

File: props.conf

[google:gcp:pubsub:message]
KV_MODE = none
AUTO_KV_JSON = false

Re: How to configure Splunk to use a specific JSON field as the event's timestamp?

jchoksi — Thu, 06 Feb 2020 10:45:03 GMT

Thanks for your response.

I tried using the following:

File: props.conf
---snip---
[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
AUTO_KV_JSON = false
TIMESTAMP_FIELDS = data.timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z
---snip---

and found that Splunk's _time field was not being set to the value of the data.timestamp field.

Maybe Splunk doesn't support nested JSON fields in TIMESTAMP_FIELDS ?

Currently, I've configured the props.conf file to use:

[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_PREFIX = \"timestamp\": \"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z

which correctly sets Splunk's _time field to the value of data.timestamp

Re: How to configure Splunk to use a specific JSON field as the event's timestamp?

manjunathmeti — Fri, 07 Feb 2020 05:41:13 GMT

Splunk does support nested json parsing.Please remove attribute TIME_FORMAT from your configurations and try. I am able to parse above json with below configurations.

 [google:gcp:pubsub:message]
 INDEXED_EXTRACTIONS = json
 KV_MODE = none
 NO_BINARY_CHECK = true
 SHOULD_LINEMERGE = false
 AUTO_KV_JSON = false
 TIMESTAMP_FIELDS = data.timestamp