topic Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events in All Apps and Add-ons

REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

Aftab_alam — Tue, 29 Sep 2020 16:35:38 GMT

Hi All,
I am pulling this json from REST input.
{"meta":{"bucket":"second","bucketsize":"1","tstart":1510090302753,"tend":1510093902753,"group":{"mname":{"desc":"Monitor name of measurement","type":"string","name":"Monitor Name"},"monid":{"desc":"Slot ID of measurement","type":"number","name":"Slot ID"}},"monid":[22762032],"metrics":{"count":{"desc":"Number of total hits or datapoints","unit":"number","name":"Total number of hits"},"avail":{"desc":"Average Availability of selected Measurements","unit":"%","name":"Availability"},"nwtme":{"desc":"Total time of all network traffic measured by the agent","unit":"ms","name":"Total Network time"},"uxtme":{"desc":"Full User Experience time as reported by the browser","unit":"ms","name":"User Experience"}},"limit":100000,"dbtime":3,"dbname":"db_dt_wa_raw_5 ","apiversion":"48.0.0.201709291816.154491-8"},"data":[{"mname":"NM - SearchAndPurchase_Guest - Chrome Agent","monid":22762032,"count":1,"nwtme":37874,"uxtme":118764,"avail":1,"mtime":1510090652753},{"mname":"NM - SearchAndPurchase_Guest - Chrome Agent","monid":22762032,"count":1,"nwtme":33711,"uxtme":120795,"avail":1,"mtime":1510091094753},{"mname":"NM - SearchAndPurchase_Guest - Chrome Agent","monid":22762032,"count":1,"nwtme":44886,"uxtme":134951,"avail":1,"mtime":1510091548753},{"mname":"NM - SearchAndPurchase_Guest - Chrome Agent","monid":22762032,"count":1,"nwtme":32109,"uxtme":114532,"avail":1,"mtime":1510091999753},{"mname":"NM - SearchAndPurchase_Guest - Chrome Agent","monid":22762032,"count":1,"nwtme":31346,"uxtme":116506,"avail":1,"mtime":1510092457753},{"mname":"NM - SearchAndPurchase_Guest - Chrome Agent","monid":22762032,"count":1,"nwtme":34060,"uxtme":126494,"avail":1,"mtime":1510092909753},{"mname":"NM - SearchAndPurchase_Guest - Chrome Agent","monid":22762032,"count":1,"nwtme":34784,"uxtme":120856,"avail":1,"mtime":1510093359753}]}

I would like to remove header until "data":[ and last part of json ]}
and create multiple events of each array
{"mname":"NM - SearchAndPurchase_Guest - Chrome {"mname":"NM - SearchAndPurchase_Guest - Chrome Agent","monid":22762032,"count":1,"nwtme":34784,"uxtme":120856,"avail":1,"mtime":1510093359753}

I have already added below in prop.conf and restarted but it does not work
TIME_PREFIX = \"mtime\":
SHOULD_LINEMERGE = false
LINE_BREAKER = ,{|}(,){
TRUNCATE = 200000
SEDCMD-remove_header = s/{\"meta[^[][//g
SEDCMD-remove_footer = s/]}//g

Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

Aftab_alam — Tue, 29 Sep 2020 16:35:43 GMT

tried changing SEDCMD
[dynatrace_api]
TIME_PREFIX = \"mtime\":
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = ,{
LINE_BREAKER = }(,){
TRUNCATE = 200000
SEDCMD-remove_header = s/{\"meta.*\"data\":[//g
SEDCMD-remove_footer = s/]}//g

it works in search UI but I am still seeing complete json getting indexed
index=main sourcetype=dynatrace_api host="dynatrace-api" | rex field=_raw mode=sed "s/{\"meta.*\"data\":[//g" | rex field=_raw mode=sed "s/]}//g" | table _raw

Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

Aftab_alam — Tue, 29 Sep 2020 16:35:40 GMT

added this in prop.confg /etc/apps/search/local/
[dynatrace_api]
TIME_PREFIX = \"mtime\":
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = ,{
LINE_BREAKER = }(,){
TRUNCATE = 200000
SEDCMD-remove_header = s/{\"meta.*\"data\":[//g
SEDCMD-remove_footer = s/]}//g

it works in search UI
index=main sourcetype=dynatrace_api host="dynatrace-api" | rex field=_raw mode=sed "s/{\"meta.*\"data\":[//g" | rex field=_raw mode=sed "s/]}//g" | table _raw
but I still see complete json is getting indexed

Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

Damien_Dallimor — Wed, 08 Nov 2017 05:31:48 GMT

You should use a custom response handler added to rest_ta/bin/responsehandlers.py for this.

Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

hardikJsheth — Wed, 08 Nov 2017 09:04:59 GMT

I think you should have a python script which is pulling this data. The best way will be to do this modification in your python script.

Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

Damien_Dallimor — Wed, 08 Nov 2017 18:52:23 GMT

The REST API Modular Input App is the defacto standard for getting data from REST API's into Splunk for years now. It has a custom pre processor framework (as detailed in my answer below regarding response handlers) for manipulating the raw received HTTP response into the format that you want to index it in Splunk.

Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

Aftab_alam — Wed, 08 Nov 2017 19:39:27 GMT

Thanks for help. it works perfectly

Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

Aftab_alam — Tue, 29 Sep 2020 16:41:02 GMT

Hi Damien,
how do I increase limit for REST input. I am getting this error. increase truncate limit in prop.conf does not help.
11-09-2017 10:50:53.087 -0800 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 130439 - data_source="rest://Dynatrace Synthetic Monitors", data_host="portal.dynatrace.com", data_sourcetype="dynatrace_api"

Thanks for help

Re: REST API Modular Input: How to remove meta data from JSON and split it in to multiple events

Damien_Dallimor — Thu, 09 Nov 2017 19:02:00 GMT

https://answers.splunk.com/answers/454889/how-to-change-the-the-truncating-limit-in-the-prop-1.html