Getting Data In

Could someone aid with event field extractions and breaking?

Splunk4
Explorer

Hi Everyone,

I am getting big single event through a python script from an API containing the performance data from an API but it is not autoextracting all the KV fields and i need to get those details to get the meaningful data.Also the timestamp is coming in epoch format.Below is the event format :

 

{'d': {'__count': '0', 'results': [{'ID': '6085', 'Name': 'device1', 'DisplayName': None, 'DisplayDescription': None, 'cpumfs': {'results': [{'ID': '6117', 'Timestamp': '1649157300', 'DeviceItemID': '6085', 'pct_im_Utilization': '4.0'}, {'ID': '6117', 'Timestamp': '1649157600', 'DeviceItemID': '6085', 'pct_im_Utilization': '1.0'}, {'ID': '6117', 'Timestamp': '1649157900', 'DeviceItemID': '6085', 'pct_im_Utilization': '4.0'}, {'ID': '6117', 'Timestamp': '1649158200', 'DeviceItemID': '6085', 'pct_im_Utilization': '1.0'}, {'ID': '6117', 'Timestamp': '1649158500', 'DeviceItemID': '6085', 'pct_im_Utilization': '4.0'}, {'ID': '6117', 'Timestamp': '1649158800', 'DeviceItemID': '6085', 'pct_im_Utilization': '1.0'}, {'ID': '6117', 'Timestamp': '1649159100', 'DeviceItemID': '6085', 'pct_im_Utilization': '4.0'}, {'ID': '6117', 'Timestamp': '1649159400', 'DeviceItemID': '6085', 'pct_im_Utilization': '1.0'}, {'ID': '6117', 'Timestamp': '1649159700', 'DeviceItemID': '6085', 'pct_im_Utilization': '4.0'}, {'ID': '6117', 'Timestamp': '1649160000', 'DeviceItemID': '6085', 'pct_im_Utilization': '1.0'}, {'ID': '6117', 'Timestamp': '1649160300', 'DeviceItemID': '6085', 'pct_im_Utilization': '4.0'}, {'ID': '6117', 'Timestamp': '1649160600', 'DeviceItemID': '6085', 'pct_im_Utilization': '1.0'}]}, 'memorymfs': {'results': [{'ID': '6118', 'Timestamp': '1649157300', 'DeviceItemID': '6085', 'im_Free': '2.809298944E9', 'pct_im_Utilization': '83.0702196963489'}, {'ID': '6118', 'Timestamp': '1649157600', 'DeviceItemID': '6085', 'im_Free': '2.741796864E9', 'pct_im_Utilization': '83.4770099337781'}, {'ID': '6118', 'Timestamp': '1649157900', 'DeviceItemID': '6085', 'im_Free': '2.784014336E9', 'pct_im_Utilization': '83.2225932482694'}, {'ID': '6118', 'Timestamp': '1649158200', 'DeviceItemID': '6085', 'im_Free': '2.739892224E9', 'pct_im_Utilization': '83.4884879350163'}, {'ID': '6118', 'Timestamp': '1649158500', 'DeviceItemID': '6085', 'im_Free': '2.812264448E9', 'pct_im_Utilization': '83.0523485718404'}, {'ID': '6118', 'Timestamp': '1649158800', 'DeviceItemID': '6085', 'im_Free': '2.747793408E9', 'pct_im_Utilization': '83.4408727427832'}, {'ID': '6118', 'Timestamp': '1649159100', 'DeviceItemID': '6085', 'im_Free': '2.808725504E9', 'pct_im_Utilization': '83.0736754386571'}, {'ID': '6118', 'Timestamp': '1649159400', 'DeviceItemID': '6085', 'im_Free': '2.744528896E9', 'pct_im_Utilization': '83.4605457900666'}, {'ID': '6118', 'Timestamp': '1649159700', 'DeviceItemID': '6085', 'im_Free': '2.804084736E9', 'pct_im_Utilization': '83.1016422674804'}, {'ID': '6118', 'Timestamp': '1649160000', 'DeviceItemID': '6085', 'im_Free': '2.740002816E9', 'pct_im_Utilization': '83.4878214704282'}, {'ID': '6118', 'Timestamp': '1649160300', 'DeviceItemID': '6085', 'im_Free': '2.7926528E9', 'pct_im_Utilization': '83.1705349587829'}, {'ID': '6118', 'Timestamp': '1649160600', 'DeviceItemID': '6085', 'im_Free': '2.736328704E9', 'pct_im_Utilization': '83.5099629050747'}]}}

In the above event , it is displaying CPU , memory utilization multiple times at different epoch times for each device.I have removed the trailing event containing data for other devices as it was exceeding the forum limit to post.I need to get the utilization data device wise.Please help on this.

 

Thanks

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where should the event(s) break?

What are the props.conf settings for this sourcetype?

---
If this reply helps you, Karma would be appreciated.
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What is the ask here? Do you want some SPL to extract some fields from the event (which has already been ingested) or so python script to break up the string into separate or smaller events or something else?

0 Karma

Splunk4
Explorer

In the above event, there is device 1 mentioned and i want to get the CPU and memory utilization at the specified timestamps.For CPU utilization is starts with 'cpumfs' and under it contains multiple values at different timestamps like below :

cpumfs': {'results': [{'ID': '6117', 'Timestamp': '1649157300', 'DeviceItemID': '6085', 'pct_im_Utilization': '4.0'}, {'ID': '6117', 'Timestamp': '1649157600', 'DeviceItemID': '6085', 'pct_im_Utilization': '1.0'}, {'ID': '6117', 'Timestamp': '1649157900', 'DeviceItemID': '6085', 'pct_im_Utilization': '4.0'}

Similarly is the case for memory as below :

 'memorymfs': {'results': [{'ID': '6118', 'Timestamp': '1649157300', 'DeviceItemID': '6085', 'im_Free': '2.809298944E9', 'pct_im_Utilization': '83.0702196963489'}, {'ID': '6118', 'Timestamp': '1649157600', 'DeviceItemID': '6085', 'im_Free': '2.741796864E9', 'pct_im_Utilization': '83.4770099337781'}, {'ID': '6118', 'Timestamp': '1649157900', 'DeviceItemID': '6085', 'im_Free': '2.784014336E9', 'pct_im_Utilization': '83.2225932482694'}

But the device name is mentioned only once in the first line of event and not coming in the cpu and memory as mentioned above.It is coming like this in the first line of event :

{'d': {'__count': '0', 'results': [{'ID': '6085', 'Name': 'device1', 'DisplayName': None, 'DisplayDescription

 

I am not sure how to do the processing,whether through props or through python script

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Given the desire to map the device name in the beginning to later entries, I think I'd write a Python script to process the data.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Splunk4
Explorer

Can someone help me with the props configuration for this.I am trying to use below but its not working.

LINE_BREAKER = (,){"id":
INDEXED_EXTRACTIONS=json
KV_MODE=none
SHOULD_LINEMERGE=true
disabled=false
pulldown_type=true

 

I am trying to break the events starting with id

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...