Getting Data In

How to configure Splunk to use a specific JSON field as the event's timestamp?

jchoksi
New Member

For the following example JSON message (formated to make it easier to read), how can I configure props.conf to inform Splunk that it should use data.timestamp for its event timestamp?

{
    "publish_time": 1580824871.446,
    "data": {
        "textPayload": "DEBUG | 2020-02-04T14:01:05,760 | A very long string here...<snip>",
        "logName": "blah0",
        "receiveTimestamp": "2020-02-04T14:01:07.707699223Z",
        "labels": {
            "k8s-pod/version": "blah2",
            "k8s-pod/track": "blah3",
            "k8s-pod/app": "blah4",
            "k8s-pod/pod-template-hash": "blah5"
        },
        "insertId": "blah6",
        "resource": {
            "type": "k8s_container",
            "labels": {
                "project_id": "blah7",
                "pod_name": "blah8",
                "cluster_name": "blah9",
                "location": "blah10",
                "container_name": "blah11",
                "namespace_name": "blah12"
            }
        },
        "severity": "INFO",
        "timestamp": "2020-02-04T14:01:05.760888513Z"
    },
    "attributes": {
        "logging.googleapis.com/timestamp": "2020-02-04T14:01:05.760888513Z"
    }
}

Would the following be correct & performant ?

File: props.conf
---snip---
[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = data.timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z
---snip---
Tags (2)
0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Yes configuration you posted works. Also please set KV_MODE = none and AUTO_KV_JSON = false for this source/sourcetype on search head(s) to avoid multi values in fields.

File: props.conf

[google:gcp:pubsub:message]
KV_MODE = none
AUTO_KV_JSON = false
0 Karma

jchoksi
New Member

Thanks for your response.

I tried using the following:

File: props.conf
---snip---
[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
AUTO_KV_JSON = false
TIMESTAMP_FIELDS = data.timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z
---snip---

and found that Splunk's _time field was not being set to the value of the data.timestamp field.

Maybe Splunk doesn't support nested JSON fields in TIMESTAMP_FIELDS ?

Currently, I've configured the props.conf file to use:

[google:gcp:pubsub:message]
INDEXED_EXTRACTIONS = json
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_PREFIX = \"timestamp\": \"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%Z

which correctly sets Splunk's _time field to the value of data.timestamp

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Splunk does support nested json parsing.Please remove attribute TIME_FORMAT from your configurations and try. I am able to parse above json with below configurations.

 [google:gcp:pubsub:message]
 INDEXED_EXTRACTIONS = json
 KV_MODE = none
 NO_BINARY_CHECK = true
 SHOULD_LINEMERGE = false
 AUTO_KV_JSON = false
 TIMESTAMP_FIELDS = data.timestamp
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...