Breaking New Relic Insights JSON into multiple events

[peteror]

I'm trying to import Insights events from NewRelic into Splunk, using the New Relic add-on. The add-on reads the Insights API every minute and returns multiple events (plus some extra data) in a single JSON file.
I've tried probably every variation of line-breaking I could find on Splunk forums, but nothing seems to work.
We have Splunk on a single server, I don't use a forwarder for this event.

Here is how my props.conf entry looks like now:

[newrelic:insights]
CHARSET=UTF-8
SHOULD_LINEMERGE=false
disabled=false
SEDCMD-remove_header=s/{\"results\":[{\"events\":[//g
SEDCMD-remove_footer=s/]}]\,\"performanceStats\":.//g
LINE_BREAKER=([\r\n,](?:{[^[{]+[)?){"aggregateFacet
TRUNCATE=0
TIME_PREFIX:"timestamp":
MAX_TIMESTAMP_LOOKAHEAD=30
TIME_FORMAT=%s%3N
KV_MODE=json

This removes the header and footer that I don't need, but does not break the events.

Here's how an API response I try to process looks like:

{"results":[{"events":[{"aggregateFacet":"Controller/Grape/sample::Proxy-v18/securitymanagement/login (POST)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.007531404495239258,"duration":0.10342597961425781,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Your login and password don’t match, please try again. (Error code -107)","externalCallCount":1,"externalDuration":0.08628702163696289,"guid":"a140483be3219f64","host":"SAM1","httpResponseCode":"400","port":80,"priority":0.603607,"realAgentId":194100514,"request.headers.accept":"/","request.headers.contentLength":60,"request.headers.contentType":"application/x-www-form-urlencoded; charset=utf-8","request.headers.host":"wsc.example.com","request.headers.userAgent":"sample/2.10.1 (ae.example.example.com; build:524; iOS 12.2.0) Alamofire/4.7.3","request.method":"POST","request.uri":"/securitymanagement/login.json","response.headers.contentLength":98,"response.headers.contentType":"application/json","sampled":false,"timestamp":1562838279585,"traceId":"a140483be3219f64","transactionName":"Controller/Grape/sample::Proxy-v18/securitymanagement/login (POST)","transactionUiName":"v18: /securitymanagement/login (POST)"},{"aggregateFacet":"Controller/Grape/sample::Proxy-v14/products/current (GET)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.00493168830871582,"duration":0.043544769287109375,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Sorry, you have been logged out of the App. Do you want to try logging in again? (Error code -27)","externalCallCount":1,"externalDuration":0.028984785079956055,"guid":"db96b40ce081f9c4","host":"SAM1","httpResponseCode":"400","port":80,"priority":1.8128410000000001,"realAgentId":194100514,"request.headers.accept":"application/json","request.headers.host":"wsc.example.com","request.headers.userAgent":"example/2.3.2(410) - (Android 6.0.1; API Level 23)","request.method":"GET","request.uri":"/products/current","response.headers.contentLength":120,"response.headers.contentType":"application/json","sampled":true,"timestamp":1562838275935,"traceId":"db96b40ce081f9c4","transactionName":"Controller/Grape/sample::Proxy-v14/products/current (GET)","transactionUiName":"v14: /products/current (GET)"},{"aggregateFacet":"Controller/Grape/sample::Proxy-v18/products/current (GET)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.005002737045288086,"duration":0.05406689643859863,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Sorry, you have been logged out of the App. Do you want to try logging in again? (Error code -27)","externalCallCount":1,"externalDuration":0.040181636810302734,"guid":"bb29b6a4bcd32d1f","host":"SAM1","httpResponseCode":"400","port":80,"priority":0.886961,"realAgentId":194100514,"request.headers.accept":"application/json","request.headers.host":"wsc.example.com","request.headers.userAgent":"example/2.11.1(508) - (Android 6.0.1; API Level 23)","request.method":"GET","request.uri":"/products/current","response.headers.contentLength":120,"response.headers.contentType":"application/json","sampled":false,"timestamp":1562838273623,"traceId":"bb29b6a4bcd32d1f","transactionName":"Controller/Grape/sample::Proxy-v18/products/current (GET)","transactionUiName":"v18: /products/current (GET)"},{"aggregateFacet":"Controller/Grape/sample::Proxy-v18/payments/confirm_payment (POST)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.005837678909301758,"duration":0.7262988090515137,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Sorry your payment couldn’t be processed. Please try again or contact your bank for more help. Need help? Call 800165 (Error code -10012)","externalCallCount":1,"externalDuration":0.7115018367767334,"guid":"1443a206b85191cc","host":"SAM1","httpResponseCode":"400","port":80,"priority":1.5844930000000002,"realAgentId":194100514,"request.headers.accept":"/","request.headers.contentLength":73,"request.headers.contentType":"application/x-www-form-urlencoded; charset=utf-8","request.headers.host":"wsc.example.com","request.headers.userAgent":"sample/2.11.1 (ae.example.example.com; build:553; iOS 12.3.1) Alamofire/4.8.2","request.method":"POST","request.uri":"/payments/confirm_payment.json","response.headers.contentLength":165,"response.headers.contentType":"application/json","sampled":true,"timestamp":1562838268402,"traceId":"1443a206b85191cc","transactionName":"Controller/Grape/sample::Proxy-v18/payments/confirm_payment (POST)","transactionUiName":"v18: /payments/confirm_payment (POST)"},{"aggregateFacet":"Controller/Grape/sample::Proxy-v18/products/current (GET)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.02594304084777832,"duration":0.06380271911621094,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Sorry, you have been logged out of the App. Do you want to try logging in again? (Error code -27)","externalCallCount":1,"externalDuration":0.027713537216186523,"guid":"765156b6b3809fa8","host":"SAM1","httpResponseCode":"400","port":80,"priority":1.357329,"realAgentId":194100514,"request.headers.accept":"application/json","request.headers.host":"wsc.example.com","request.headers.userAgent":"example/2.11.1(508) - (Android 6.0; API Level 23)","request.method":"GET","request.uri":"/products/current","response.headers.contentLength":120,"response.headers.contentType":"application/json","sampled":true,"timestamp":1562838249748,"traceId":"765156b6b3809fa8","transactionName":"Controller/Grape/sample::Proxy-v18/products/current (GET)","transactionUiName":"v18: /products/current (GET)"}]}],"performanceStats":{"fileReadCount":1,"decompressionCount":0,"decompressionCacheEnabledCount":0,"filesSkippedByHeader":0,"inspectedCount":25932,"omittedCount":0,"matchCount":5,"processCount":1,"rawBytes":3507705,"decompressedBytes":3507705,"ioBytes":3507705,"decompressionOutputBytes":0,"responseBodyBytes":6548,"fileProcessingTime":2,"mergeTime":0,"ioTime":0,"decompressionTime":0,"decompressionCacheGetTime":0,"decompressionCachePutTime":0,"wallClockTime":17,"fullCacheHits":0,"partialCacheHits":0,"cacheMisses":0,"cacheSkipped":1,"maxInspectedCount":25932,"minInspectedCount":25932,"slowLaneFiles":0,"slowLaneFileProcessingTime":0,"slowLaneWaitTime":0,"sumSubqueryWeight":1.0,"sumFileProcessingTimePercentile":0.0,"subqueryWeightUpdates":0,"sumSubqueryWeightStartFileProcessingTime":58,"runningQueriesTotal":4,"ignoredFiles":0},"metadata":{"eventTypes":["TransactionError"],"eventType":"TransactionError","openEnded":true,"beginTime":"2019-07-11T09:43:58Z","endTime":"2019-07-11T09:44:58Z","beginTimeMillis":1562838238719,"endTimeMillis":1562838298719,"rawSince":"1 MINUTES AGO","rawUntil":"NOW","rawCompareWith":"","guid":"c5b08940-3cc0-8240-4f97-4b06c860e527","routerGuid":"aab8af67-a175-729b-1643-d3aad4a95e3d","messages":[],"contents":[{"function":"events","limit":100,"order":{"column":"timestamp","descending":true}}]}}

[peteror · Accepted solution]

Finally managed to resolve this.
I have no idea what's going on inside the NewRelic add-on: I got the JSON output from calling the NewRelic API directly, saved it as a text file and used Splunk's data preview function - it broke up the file perfectly using my settings.

So I built a custom add-on using the add-on builder, that calls this API (which I suspect is the same the NR add-on does) and filter the input there - works perfectly.

[mbonsack_splunk]

Can you post a link to the app you created? Thanks!