Getting Data In

How do you remove a header from JSON?

stevenbutterwor
Path Finder

Hi all

I'm ingesting some JSON via REST API, but the events are all squashed into one large event. I'm pretty sure it's because there is a header at the top of the file that needs to be removed for the JSON to be parsed correctly.

I think, again, I need to add SEDCMD to the props, but I'm unsure of the regex required. Any help would be appreciated — below is a sample of the JSON. The bold section is the header.

{"location":{"name":"Adlington (Lancashire)","crs":"ADL","tiploc":"ADNL"},"filter":null,"services":[{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2006","gbttBookedDeparture":"2006","origin":[{"tiploc":"BLCKPLN","description":"Blackpool North","workingTime":"191600","publicTime":"1916"},{"tiploc":"WMER","description":"Windermere","workingTime":"183000","publicTime":"1830"}],"destination":[{"tiploc":"MNCRPIC","description":"Manchester Piccadilly","workingTime":"204500","publicTime":"2045"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2007","realtimeArrivalActual":false,"realtimeDeparture":"2008","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y53223","runDate":"2019-02-11","trainIdentity":"2K54","runningIdentity":"2K54","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2043","gbttBookedDeparture":"2044","origin":[{"tiploc":"MNCRVIC","description":"Manchester Victoria","workingTime":"200500","publicTime":"2005"}],"destination":[{"tiploc":"PRST","description":"Preston","workingTime":"211100","publicTime":"2113"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2043","realtimeArrivalActual":false,"realtimeDeparture":"2044","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y54186","runDate":"2019-02-11","trainIdentity":"2P44","runningIdentity":"2P44","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2103","gbttBookedDeparture":"2104","origin":[{"tiploc":"BLCKPLN","description":"Blackpool North","workingTime":"201400","publicTime":"2014"}],"destination":[{"tiploc":"MNCRPIC","description":"Manchester Piccadilly","workingTime":"214300","publicTime":"2143"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2103","realtimeArrivalActual":false,"realtimeDeparture":"2104","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y53227","runDate":"2019-02-11","trainIdentity":"2K55","runningIdentity":"2K55","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2127","gbttBookedDeparture":"2127","origin":[{"tiploc":"PRST","description":"Preston","workingTime":"210800","publicTime":"2108"}],"destination":[{"tiploc":"MNCROXR","description":"Manchester Oxford Road","workingTime":"220400","publicTime":"2204"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2126","realtimeArrivalActual":false,"realtimeDeparture":"2127","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y50887","runDate":"2019-02-11","trainIdentity":"1Y68","runningIdentity":"1Y68","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true}

Tags (2)
0 Karma

chrisyounger
SplunkTrust
SplunkTrust

If you are using the "REST Modular Input" you might be able to set "Response Handler" to be "JSONArrayHandler" and it should split it into individual events properly.

If that doesn't work, then i think you should set LINE_BREAKER to split correctly by events:

LINE_BREAKER = (^[^\[]+\[|,){"locationDetail"

This will discard the header as well as split events correctly.

Good luck!

0 Karma

stevenbutterwor
Path Finder

Hello, thanks for the advice but none of it worked 😞

Any ideas about removing that line that's not needed as it's still being indexed.

Thanks!

0 Karma

stevenbutterwor
Path Finder

Here is the props

[realtimetrains:basic]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = json
LINE_BREAKER = (^[^[]+[|,){"locationDetail"
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false

0 Karma

rashid47010
Communicator

did you find the solution

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...