topic Re: extracting data using script getting none in timestamp field in Getting Data In

extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 09:00:18 GMT

Hi Team,

need your help, while i am ingesting data using python script i.e scripted input. for timestamp field i am getting none value . even in script data is populating fine but when it is ingesting in splunk it is getting extra field value none for timestamp

need your help

Re: extracting data using script getting none in timestamp field

isoutamo — Tue, 04 Jul 2023 09:05:09 GMT

we need some more information for helping you!

Are this a scripted input or modular input?
If scripted input can you show your inputs.conf and also what are output of script.
Have you test it it "splunk cmd python <your script>"
What is your props.conf for this sourcetype/source
Are you running this on UF, HF or where

r. Ismo

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 09:27:28 GMT

script is inside datainputs --> scripts, it may be scripted input

running on HF

no props.conf in local, default one only

tested using .. cmd python , data coming fine in timestamp field , none value not there

Re: extracting data using script getting none in timestamp field

isoutamo — Tue, 04 Jul 2023 09:58:36 GMT

You probably need a props.conf for it on HF to get timestamp correctly?

Could you show your inputs.conf, where you have defined input and also example output from script?

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 11:33:29 GMT

inputs.conf like this

[script://./bin/networker_alerts.py]
disabled = 0
index = test
interval = 4-59/5 * * * *
source = test_1
sourcetype = _json

and output of script is like this

{"category": "disk space", "message": "'xxx' host '/nsr' disk path occupied with '92.42%' of disk space. Free up the space.", "priority": "warning", "timestamp": "2023-07-03T08:51:25+02:00"}

here timestamp is having only time value but when when that data populating in splunk it is showing like this

Re: extracting data using script getting none in timestamp field

isoutamo — Tue, 04 Jul 2023 11:48:18 GMT

You should always define your own sourcetype instead of use _json. So change your inputs.conf like

[script://./bin/networker_alerts.py] disabled = 0 index = test interval = 4-59/5 * * * * source = script:networker_alerts.py sourcetype = json:with:timestamp

Then you should define your props.conf for that sourcetype on that HF. Please create own app for it

[json:with:timestamp] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=UTF-8 INDEXED_EXTRACTIONS=json KV_MODE=none category=Structured description=Your own JSON definition for networker_alerts.py script disabled=false pulldown_type=true TIME_FORMAT=%Y-%m-%dT%H:%M:%S%:z TIMESTAMP_FIELDS=timestamp

Then you must restart that HF for reading that props.conf.

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 13:40:45 GMT

thankyou isoutamo, this solution solved my problem

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 13:42:48 GMT

@isoutamo , can you help me to understand what was the issue ,

one doubt, if i put props.conf in /opt/splunk/etc/system/local, it will be used by all data inputs right.

here i have kept it inside /etc/apps/<app_name>/local folder . only

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 13:45:12 GMT

none is removed from timestamp field . but now i am getting duplicate values in timestamp field for single event

see below. please help to resolve this

Re: extracting data using script getting none in timestamp field

isoutamo — Tue, 04 Jul 2023 13:54:19 GMT

It depending how apps configurations have exported. Usually on indexing phase you should export those globally. Then it's no matter where you have put those. Expect the precedence which are defined by app names.

If/when your job is to do data onboarding to splunk I suggest to you to take Splunk Data Administration course. It will explain that well.

Re: extracting data using script getting none in timestamp field

isoutamo — Tue, 04 Jul 2023 13:55:59 GMT

Usually duplicate events "exists" when you have both KV_MODE=json and INDEXED_EXTRACTIONS=json defined.

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 14:01:08 GMT

there is only single event in splunk web it is showing but when i am checking timestamp field with

index=xx | table timestamp. it is showing multiple values in single field.

both parameter are not json as per your props.conf values given by you these are the settings.

indexed_extractions= json

kv_mode = none

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 14:11:13 GMT

@isoutamo , what will be the solution. please help

Re: extracting data using script getting none in timestamp field

isoutamo — Tue, 04 Jul 2023 14:30:23 GMT

Is this separate field than _time? And is there any other fields duplicated or only this one?

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 14:37:21 GMT

all fields duplicated which are coming in scripted input output. like below

Re: extracting data using script getting none in timestamp field

isoutamo — Tue, 04 Jul 2023 16:34:07 GMT

What kind of environment you have? Single node where you have indexer, search head and running this modular input or is this a distributed environment with own node for different nodes?

If last one are you sure that you haven't KV_MODE=json on search head? You cannot use is as you already have INDEXED_EXTRACTS=json on you scripted input on HF/Indexer.

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 16:57:35 GMT

when i removed indexed_extractions from props.conf again i start getting same data as it was in starting like

category, priority and message field as single value but timestamp field again with 2 values like none value timestamp

if there was any kv_mode=json on any node like search head . data would not have come back in old format

it would have been duplicate values only

Re: extracting data using script getting none in timestamp field

isoutamo — Tue, 04 Jul 2023 17:04:49 GMT

On HF/IDX (where script inputs is running) you should use INDEXED_EXTRACTIONS=json and on SH (I suppose that this is a different node) you must have a KV_MODE=none. Or vice versa, but both cannot be at the same time. If you are not using INDEXED_EXTRACTIONS=json you must take care of timestamp -> _time modification otherwise.

Re: extracting data using script getting none in timestamp field

anilkapoor123 — Tue, 04 Jul 2023 21:50:23 GMT

i have distributed env. search heads and indexers are in clustering , search heads are not in shclustering heavy forwarder is also there

i was testing this script on dev environment before making changes in prod. so i have placed scripted inputs on search head not on HF

props.conf is also on search head containing below configs as you mentioned in above post

[json_scripted_input]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
INDEXED_EXTRACTIONS=json [on search head]
KV_MODE=none [on search head]
category=Structured
description=Your own JSON definition for networker_alerts.py script
disabled=false
pulldown_type=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S%:z
TIMESTAMP_FIELDS=timestamp
#AUTO_KV_JSON=false

INDEXED_EXTRACTIONS=json ( when i disable it , category, message and priority field goes fine with single value, but timestamp got 2 values i.e none and timestamp)

and when i enable it, all field having duplicates values, timestamp also having duplicate values without none

please suggest

also let me know if i am not using indexed_extractions=json, how can i convert timestamp into _time

Re: extracting data using script getting none in timestamp field

isoutamo — Wed, 05 Jul 2023 07:04:43 GMT

In distributed onprem environment you shouldn't use SH as a HF to run that script if you could use separate HF.

Can you do on SH's command line

splunk btool props list json_scripted_input --debug | egrep'(INDEXED_EXTRACTIONS|KV_MODE|AUTO_KV_JSON)'

This will show what those values are on SH