topic Re: Extract event fields in desired format in Getting Data In

Extract event fields in desired format

rantravee — Mon, 28 Sep 2020 14:58:08 GMT

hi,

I'd need some hints regarding the propertiesI should have in in props.conf and transforms.conf so that I have data in the needed format. So I within my scripted input I print to standard output a json object of the following format

{

"statistics" :[{stats_resource_json_obj_1},....,{stats_resource_json_obj_n}]

}

each stats_resource_json_obj_i has the folowing format
{
"id":value,
............
"stat_i":"val_i"
}
I would like to have an event for each stats-resource-json-obj and to have recognized/extracted as fields (at search time) every pair of key/value within the object .

Thanks a lot

Re: Extract event fields in desired format

alacercogitatus — Tue, 15 Oct 2013 11:18:56 GMT

In your props.conf, make sure that your KV_MODE is set to JSON. In order to split up the events, since you have control of the script, I'd rework the script to print a new json object on each line, Splunk should handle the rest.

Re: Extract event fields in desired format

rantravee — Mon, 28 Sep 2020 14:58:13 GMT

It seems Splunk is incapable of automatically extracting the fields .Only the default fields are extracted . I see a long string ,containing all the key/pair values . Something like

{'stat_1': 0, 'stat_2': 0, 'stat_3': 0, 'stat_4': 0, 'stat_4': 0, ....., 'stat_n': 0 }

Re: Extract event fields in desired format

alacercogitatus — Tue, 15 Oct 2013 11:45:42 GMT

remove the single quotes from the field names.

Re: Extract event fields in desired format

rantravee — Tue, 15 Oct 2013 12:07:42 GMT

How could I achieve that ? I realize that my question may sound ridiculous , but I just recently crushed Pyhton Programming.

Re: Extract event fields in desired format

alacercogitatus — Tue, 15 Oct 2013 12:18:34 GMT

I would need to see a pastebin of you code to see what you are doing to accurately diagnose it.

Re: Extract event fields in desired format

rantravee — Mon, 28 Sep 2020 14:58:16 GMT

def printResponseToSplunk(self,s):
jdata = json.loads(s)

# Augment json object with additional information
stats = jdata[JSON_STATS_OBJ_NAME]

for innerOBj in stats
innerOBj[JSON_CHASSIS_KEY]=self.getHost()
print (json.dumps(innerOBj))
sys.stdout.flush()

Re: Extract event fields in desired format

alacercogitatus — Tue, 15 Oct 2013 12:30:29 GMT

Can you paste the entire script to pastebin please? I want to see how you are crafting the string that you then dump with the json object. Don't forget we can help you out in Real Time on the IRC #splunk channel on efnet.

Re: Extract event fields in desired format

rantravee — Tue, 15 Oct 2013 12:32:44 GMT

Thanks for the channel hint. I was unware of that.

Re: Extract event fields in desired format

rantravee — Mon, 28 Sep 2020 14:58:22 GMT

The json obj comes in the format described in the question from a network resource :

resp, content = http.request(statistics_url,
method='POST',
headers={'Content-Type': 'application/json', 'charset':'UTF-8','Connection':'keep-alive', 'Host':'theHost'}, body=json_body)

and then I call printResponseToSplunk(), passing the content object to it