All Apps and Add-ons

Breaking New Relic Insights JSON into multiple events

peteror
Engager

I'm trying to import Insights events from NewRelic into Splunk, using the New Relic add-on. The add-on reads the Insights API every minute and returns multiple events (plus some extra data) in a single JSON file.
I've tried probably every variation of line-breaking I could find on Splunk forums, but nothing seems to work.
We have Splunk on a single server, I don't use a forwarder for this event.

Here is how my props.conf entry looks like now:

[newrelic:insights]
CHARSET=UTF-8
SHOULD_LINEMERGE=false
disabled=false
SEDCMD-remove_header=s/{\"results\":[{\"events\":[//g
SEDCMD-remove_footer=s/]}]\,\"performanceStats\":.//g
LINE_BREAKER=([\r\n,]
(?:{[^[{]+[)?){"aggregateFacet
TRUNCATE=0
TIME_PREFIX:"timestamp":
MAX_TIMESTAMP_LOOKAHEAD=30
TIME_FORMAT=%s%3N
KV_MODE=json

This removes the header and footer that I don't need, but does not break the events.

Here's how an API response I try to process looks like:

{"results":[{"events":[{"aggregateFacet":"Controller/Grape/sample::Proxy-v18/securitymanagement/login (POST)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.007531404495239258,"duration":0.10342597961425781,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Your login and password don’t match, please try again. (Error code -107)","externalCallCount":1,"externalDuration":0.08628702163696289,"guid":"a140483be3219f64","host":"SAM1","httpResponseCode":"400","port":80,"priority":0.603607,"realAgentId":194100514,"request.headers.accept":"/","request.headers.contentLength":60,"request.headers.contentType":"application/x-www-form-urlencoded; charset=utf-8","request.headers.host":"wsc.example.com","request.headers.userAgent":"sample/2.10.1 (ae.example.example.com; build:524; iOS 12.2.0) Alamofire/4.7.3","request.method":"POST","request.uri":"/securitymanagement/login.json","response.headers.contentLength":98,"response.headers.contentType":"application/json","sampled":false,"timestamp":1562838279585,"traceId":"a140483be3219f64","transactionName":"Controller/Grape/sample::Proxy-v18/securitymanagement/login (POST)","transactionUiName":"v18: /securitymanagement/login (POST)"},{"aggregateFacet":"Controller/Grape/sample::Proxy-v14/products/current (GET)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.00493168830871582,"duration":0.043544769287109375,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Sorry, you have been logged out of the App. Do you want to try logging in again? (Error code -27)","externalCallCount":1,"externalDuration":0.028984785079956055,"guid":"db96b40ce081f9c4","host":"SAM1","httpResponseCode":"400","port":80,"priority":1.8128410000000001,"realAgentId":194100514,"request.headers.accept":"application/json","request.headers.host":"wsc.example.com","request.headers.userAgent":"example/2.3.2(410) - (Android 6.0.1; API Level 23)","request.method":"GET","request.uri":"/products/current","response.headers.contentLength":120,"response.headers.contentType":"application/json","sampled":true,"timestamp":1562838275935,"traceId":"db96b40ce081f9c4","transactionName":"Controller/Grape/sample::Proxy-v14/products/current (GET)","transactionUiName":"v14: /products/current (GET)"},{"aggregateFacet":"Controller/Grape/sample::Proxy-v18/products/current (GET)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.005002737045288086,"duration":0.05406689643859863,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Sorry, you have been logged out of the App. Do you want to try logging in again? (Error code -27)","externalCallCount":1,"externalDuration":0.040181636810302734,"guid":"bb29b6a4bcd32d1f","host":"SAM1","httpResponseCode":"400","port":80,"priority":0.886961,"realAgentId":194100514,"request.headers.accept":"application/json","request.headers.host":"wsc.example.com","request.headers.userAgent":"example/2.11.1(508) - (Android 6.0.1; API Level 23)","request.method":"GET","request.uri":"/products/current","response.headers.contentLength":120,"response.headers.contentType":"application/json","sampled":false,"timestamp":1562838273623,"traceId":"bb29b6a4bcd32d1f","transactionName":"Controller/Grape/sample::Proxy-v18/products/current (GET)","transactionUiName":"v18: /products/current (GET)"},{"aggregateFacet":"Controller/Grape/sample::Proxy-v18/payments/confirm_payment (POST)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.005837678909301758,"duration":0.7262988090515137,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Sorry your payment couldn’t be processed. Please try again or contact your bank for more help. Need help? Call 800165 (Error code -10012)","externalCallCount":1,"externalDuration":0.7115018367767334,"guid":"1443a206b85191cc","host":"SAM1","httpResponseCode":"400","port":80,"priority":1.5844930000000002,"realAgentId":194100514,"request.headers.accept":"/","request.headers.contentLength":73,"request.headers.contentType":"application/x-www-form-urlencoded; charset=utf-8","request.headers.host":"wsc.example.com","request.headers.userAgent":"sample/2.11.1 (ae.example.example.com; build:553; iOS 12.3.1) Alamofire/4.8.2","request.method":"POST","request.uri":"/payments/confirm_payment.json","response.headers.contentLength":165,"response.headers.contentType":"application/json","sampled":true,"timestamp":1562838268402,"traceId":"1443a206b85191cc","transactionName":"Controller/Grape/sample::Proxy-v18/payments/confirm_payment (POST)","transactionUiName":"v18: /payments/confirm_payment (POST)"},{"aggregateFacet":"Controller/Grape/sample::Proxy-v18/products/current (GET)::sample::API::Error::APIError","appId":31710528,"appName":"FrontEnd Proxy (Production)","databaseCallCount":9,"databaseDuration":0.02594304084777832,"duration":0.06380271911621094,"entityGuid":"MTQ3MjY1NnxBUE18QVBQTElDQVRJT058MzE3MTA1Mjg","error.class":"sample::API::Error::APIError","error.expected":false,"error.message":"Sorry, you have been logged out of the App. Do you want to try logging in again? (Error code -27)","externalCallCount":1,"externalDuration":0.027713537216186523,"guid":"765156b6b3809fa8","host":"SAM1","httpResponseCode":"400","port":80,"priority":1.357329,"realAgentId":194100514,"request.headers.accept":"application/json","request.headers.host":"wsc.example.com","request.headers.userAgent":"example/2.11.1(508) - (Android 6.0; API Level 23)","request.method":"GET","request.uri":"/products/current","response.headers.contentLength":120,"response.headers.contentType":"application/json","sampled":true,"timestamp":1562838249748,"traceId":"765156b6b3809fa8","transactionName":"Controller/Grape/sample::Proxy-v18/products/current (GET)","transactionUiName":"v18: /products/current (GET)"}]}],"performanceStats":{"fileReadCount":1,"decompressionCount":0,"decompressionCacheEnabledCount":0,"filesSkippedByHeader":0,"inspectedCount":25932,"omittedCount":0,"matchCount":5,"processCount":1,"rawBytes":3507705,"decompressedBytes":3507705,"ioBytes":3507705,"decompressionOutputBytes":0,"responseBodyBytes":6548,"fileProcessingTime":2,"mergeTime":0,"ioTime":0,"decompressionTime":0,"decompressionCacheGetTime":0,"decompressionCachePutTime":0,"wallClockTime":17,"fullCacheHits":0,"partialCacheHits":0,"cacheMisses":0,"cacheSkipped":1,"maxInspectedCount":25932,"minInspectedCount":25932,"slowLaneFiles":0,"slowLaneFileProcessingTime":0,"slowLaneWaitTime":0,"sumSubqueryWeight":1.0,"sumFileProcessingTimePercentile":0.0,"subqueryWeightUpdates":0,"sumSubqueryWeightStartFileProcessingTime":58,"runningQueriesTotal":4,"ignoredFiles":0},"metadata":{"eventTypes":["TransactionError"],"eventType":"TransactionError","openEnded":true,"beginTime":"2019-07-11T09:43:58Z","endTime":"2019-07-11T09:44:58Z","beginTimeMillis":1562838238719,"endTimeMillis":1562838298719,"rawSince":"1 MINUTES AGO","rawUntil":"NOW","rawCompareWith":"","guid":"c5b08940-3cc0-8240-4f97-4b06c860e527","routerGuid":"aab8af67-a175-729b-1643-d3aad4a95e3d","messages":[],"contents":[{"function":"events","limit":100,"order":{"column":"timestamp","descending":true}}]}}

0 Karma
1 Solution

peteror
Engager

Finally managed to resolve this.
I have no idea what's going on inside the NewRelic add-on: I got the JSON output from calling the NewRelic API directly, saved it as a text file and used Splunk's data preview function - it broke up the file perfectly using my settings.

So I built a custom add-on using the add-on builder, that calls this API (which I suspect is the same the NR add-on does) and filter the input there - works perfectly.

View solution in original post

0 Karma

peteror
Engager

Finally managed to resolve this.
I have no idea what's going on inside the NewRelic add-on: I got the JSON output from calling the NewRelic API directly, saved it as a text file and used Splunk's data preview function - it broke up the file perfectly using my settings.

So I built a custom add-on using the add-on builder, that calls this API (which I suspect is the same the NR add-on does) and filter the input there - works perfectly.

0 Karma

mbonsack_splunk
Splunk Employee
Splunk Employee

Can you post a link to the app you created? Thanks!

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...