topic Re: Unable to set indexed time value from JSON file to _time in Getting Data In

Unable to set indexed time value from JSON file to _time

shahid285 — Tue, 29 Sep 2020 21:54:36 GMT

Hi All,
I am finding it difficult trying to set the event time(_time) to be taken from the indexed JSON format file.

i have done the configurations as below

in props.conf
[aci-inventory]
INDEXED_EXTRACTIONS = json
KV_MODE = json
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ({|[\s+{)
MUST_BREAK_AFTER = (}|}\s+])
SEDCMD-remove_header = s/({\s+.+?[)//g
SEDCMD-remove_trailing_commas = s/},/}/g
SEDCMD-remove_footer = s/]\s+}//g
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIMESTAMP_FIELDS = requested_on
TRANSFORMS-datetime = aci_timestamp

in transforms.conf
[aci_timestamp]
SOURCE_KEY = requested_on
DEST_KEY = _time

example JSON file i used is as below

{
"IPRequestLog": [
{
"access_key": "test",
"id": "0ac03844-a374-4237-9172-a7af9122bed2",
"ip_address": "192.168.1.245",
"requested_on": "2015-07-28 06:47:48",
"source_ip": "49.248.183.29"
},
{
"access_key": "test",
"id": "0ac03844-a374-4237-9172-e33f206474ff",
"ip_address": "192.168.1.245",
"requested_on": "2015-07-28 06:47:54",
"source_ip": "49.248.183.29"
},
{
"access_key": "test",
"id": "0ac03844-a374-4237-9172-5c256d8d502e",
"ip_address": "192.168.1.245",
"requested_on": "2015-07-28 06:47:51",
"source_ip": "49.248.183.29"
}
]
}

Also, please let me know if we really need to set the following keys in transforms.conf. If so what it would be as per the above input file.
REGEX =
FORMAT =

I have checked the sample with csv been posted here , using which i came to this conclusion and which is not working for me.
Requesting help at the earliest.
Thanks
Mohammed Shahid Nawaz

Re: Unable to set indexed time value from JSON file to _time

adonio — Tue, 29 Sep 2020 21:56:04 GMT

without testing,
in TIME_FORMAT try to get rid of the T
TIME_FORMAT = %Y-%m-%d %H:%M:%S

Re: Unable to set indexed time value from JSON file to _time

anilsharmahk — Tue, 29 Sep 2020 21:55:45 GMT

step 1 try only on props

TIME_FORMAT = %Y-%m-%d %H:%M
TIME_PREFIX = \"requested_on\":\s

just check exact bracket & quotes etc

play on props -- should give you
You dont need transform for this

also take this out - TIMESTAMP_FIELDS = requested_on

Re: Unable to set indexed time value from JSON file to _time

shahid285 — Tue, 29 Sep 2020 21:56:09 GMT

@adonio: The TIME_FORMAT change did not make any difference, as i tried earlier. I am getting the indexed data as expected but event time i.e., _time needs to be the value of requested_on

@anilsharmahk : I tried your approach and now not even the data is getting indexed, where earlier i was getting indexed data.

Re: Unable to set indexed time value from JSON file to _time

anilsharmahk — Tue, 29 Sep 2020 21:55:50 GMT

is it @adonio .. why data indexing stoped. you r prop is messed up you cant have

"INDEXED_EXTRACTIONS = json
KV_MODE = json"
eithe one..
i thought you need just help on time format and Time_prefix ..
please see prop.conf documentation for jason file

Re: Unable to set indexed time value from JSON file to _time

anilsharmahk — Wed, 07 Nov 2018 16:05:44 GMT

https://answers.splunk.com/answers/168959/parsing-json-data-structures.html

Re: Unable to set indexed time value from JSON file to _time

anilsharmahk — Tue, 29 Sep 2020 21:55:53 GMT

[aci-inventory]
INDEXED_EXTRACTIONS =
KV_MODE = json
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ({|[\s+{)
MUST_BREAK_AFTER = (}|}\s+])
SEDCMD-remove_header = s/({\s+.+?[)//g
SEDCMD-remove_trailing_commas = s/},/}/g
SEDCMD-remove_footer = s/]\s+}//g
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = \"requested_on\":\s**
TRANSFORMS-datetime = aci_timestamp

Re: Unable to set indexed time value from JSON file to _time

shahid285 — Tue, 29 Sep 2020 21:56:12 GMT

Hi @anilsharmahk even if i have both or remove either of KV_MODE or INDEXED_EXTRACTIONS the result is same as i am presently getting.
I checked out the documentation part and was clueless as to why the index logic is behaving so different for JSON and at the same time working for CSV.

Thanks
Mohammed Shahid Nawaz

Re: Unable to set indexed time value from JSON file to _time

anilsharmahk — Wed, 07 Nov 2018 17:02:59 GMT

ok lets do this mate .. post the jason and prop...
use only prop you dont need transform.
i can re code you full prop for you .. is your splunk on window or linux ? on cluster and what version ? are you putting this prop on forwader ?

Re: Unable to set indexed time value from JSON file to _time

shahid285 — Tue, 29 Sep 2020 21:56:18 GMT

@adonio: The TIME_FORMAT change did not make any difference, as i tried earlier. I am getting the indexed data as expected but event time i.e., _time needs to be the value of requested_on

Re: Unable to set indexed time value from JSON file to _time

shahid285 — Tue, 29 Sep 2020 21:56:20 GMT

alright,
the JSON i am trying to use is as follows,
{
"imdata": [
{
"cause":"resolution-failed",
"childAction":"",
"code":"F0955",
"count":"1",
"descr":"The object refers to an object that was not found.",
"dn":"fltcode-F0955",
"domain":"infra",
"nonAcked":"1",
"nonDelegated":"1",
"nonDelegatedAndNonAcked":"1",
"rule":"fv-rs-bd-resolve-fail",
"severity":"warning",
"status":"",
"subject":"relation-resolution",
"requested_on": "2015-07-12 06:47:51",
"type":"config"
},
{
"cause":"protocol-ntp-provisioning-failed",
"childAction":"",
"code":"F1699",
"count":"1",
"descr":"This fault occurs when a ntp configuration on a switch has problems",
"dn":"fltcode-F1699",
"domain":"infra",
"nonAcked":"1",
"nonDelegated":"1",
"nonDelegatedAndNonAcked":"1",
"rule":"datetime-ntp-provider-status-ntp-prov-fault",
"severity":"warning",
"status":"",
"subject":"ntp",
"requested_on": "2015-07-12 06:47:51",
"type":"operational"
}
]
}

The props.conf is the default one, and it has been untouched except with my newly added source type settings for indexing, which is same as before.

[aci-inventory]
INDEXED_EXTRACTIONS = json
KV_MODE = json
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ({|[\s+{)
MUST_BREAK_AFTER = (}|}\s+])
SEDCMD-remove_header = s/({\s+.+?[)//g
SEDCMD-remove_trailing_commas = s/},/}/g
SEDCMD-remove_footer = s/]\s+}//g
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIMESTAMP_FIELDS = requested_on

I am doing a basic setup with universal forwarder and indexer and search head on linux vm (Cent OS 7)
the splunk i use is the most recent as i had downloaded just yesterday.

Thanks
Mohammed Shahid Nawaz

Re: Unable to set indexed time value from JSON file to _time

shahid285 — Fri, 09 Nov 2018 03:50:06 GMT

@anilsharmahk Hi , were you able to debug the issue by any chance?

thanks
Mohammed Shahid Nawaz

Re: Unable to set indexed time value from JSON file to _time

anilsharmahk — Tue, 29 Sep 2020 22:01:49 GMT

Thanks @shahid285
this how i will try to resolve
Step 1: clear all the indexs for this source type [aci-inventory]
step 2: remove any bucket copies of index (optional)
step 3: create a source type [aci-inventory] on the forwader
step 4: open prop.conf and first get the date right
[aci-inventory]
pulldown_type = true
KV_MODE = json
description = For indexing JSON comments
category = Custom
BREAK_ONLY_BEFORE = {(")ca
TIME_PREFIX = \"requested_on\":\s
TRUNCATE = 0
LINE_BREAKER = }(,){(")ca
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d %H:%M
disabled = false
SEDCMD-remove = s/{\"imdata\":\s[{/{/g
SEDCMD-find_and_replace = s/}]}]}/}]}/1

Step 5: save props.conf and restart splunk
step 6: push data

Step 7 : check data on search head

just tune the above -- take few runs to get right.
thats it ..

give it a go bro

thanks

Re: Unable to set indexed time value from JSON file to _time

anilsharmahk — Tue, 29 Sep 2020 22:01:52 GMT

@shahid @adonio
yes i get thats what TIME_PREFIX will do

TIME_PREFIX = \"requested_on\":\s
tune this
"requested_on": "2015-07-12 06:47:51",