How do you remove a header from JSON?

[stevenbutterwor]

Hi all

I'm ingesting some JSON via REST API, but the events are all squashed into one large event. I'm pretty sure it's because there is a header at the top of the file that needs to be removed for the JSON to be parsed correctly.

I think, again, I need to add SEDCMD to the props, but I'm unsure of the regex required. Any help would be appreciated — below is a sample of the JSON. The bold section is the header.

{"location":{"name":"Adlington (Lancashire)","crs":"ADL","tiploc":"ADNL"},"filter":null,"services":[{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2006","gbttBookedDeparture":"2006","origin":[{"tiploc":"BLCKPLN","description":"Blackpool North","workingTime":"191600","publicTime":"1916"},{"tiploc":"WMER","description":"Windermere","workingTime":"183000","publicTime":"1830"}],"destination":[{"tiploc":"MNCRPIC","description":"Manchester Piccadilly","workingTime":"204500","publicTime":"2045"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2007","realtimeArrivalActual":false,"realtimeDeparture":"2008","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y53223","runDate":"2019-02-11","trainIdentity":"2K54","runningIdentity":"2K54","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2043","gbttBookedDeparture":"2044","origin":[{"tiploc":"MNCRVIC","description":"Manchester Victoria","workingTime":"200500","publicTime":"2005"}],"destination":[{"tiploc":"PRST","description":"Preston","workingTime":"211100","publicTime":"2113"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2043","realtimeArrivalActual":false,"realtimeDeparture":"2044","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y54186","runDate":"2019-02-11","trainIdentity":"2P44","runningIdentity":"2P44","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2103","gbttBookedDeparture":"2104","origin":[{"tiploc":"BLCKPLN","description":"Blackpool North","workingTime":"201400","publicTime":"2014"}],"destination":[{"tiploc":"MNCRPIC","description":"Manchester Piccadilly","workingTime":"214300","publicTime":"2143"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2103","realtimeArrivalActual":false,"realtimeDeparture":"2104","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y53227","runDate":"2019-02-11","trainIdentity":"2K55","runningIdentity":"2K55","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2127","gbttBookedDeparture":"2127","origin":[{"tiploc":"PRST","description":"Preston","workingTime":"210800","publicTime":"2108"}],"destination":[{"tiploc":"MNCROXR","description":"Manchester Oxford Road","workingTime":"220400","publicTime":"2204"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2126","realtimeArrivalActual":false,"realtimeDeparture":"2127","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y50887","runDate":"2019-02-11","trainIdentity":"1Y68","runningIdentity":"1Y68","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true}

[chrisyounger]

If you are using the "REST Modular Input" you might be able to set "Response Handler" to be "JSONArrayHandler" and it should split it into individual events properly.

If that doesn't work, then i think you should set LINE_BREAKER to split correctly by events:

LINE_BREAKER = (^[^\[]+\[|,){"locationDetail"

This will discard the header as well as split events correctly.

Good luck!

[stevenbutterwor]

Hello, thanks for the advice but none of it worked 😞

Any ideas about removing that line that's not needed as it's still being indexed.

Thanks!

[stevenbutterwor]

Here is the props

[realtimetrains:basic]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = json
LINE_BREAKER = (^[^[]+[|,){"locationDetail"
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false

[rashid47010]

did you find the solution