How do you remove a header from JSON?

stevenbutterwor — Mon, 11 Feb 2019 20:45:00 GMT

Hi all

I'm ingesting some JSON via REST API, but the events are all squashed into one large event. I'm pretty sure it's because there is a header at the top of the file that needs to be removed for the JSON to be parsed correctly.

I think, again, I need to add SEDCMD to the props, but I'm unsure of the regex required. Any help would be appreciated — below is a sample of the JSON. The bold section is the header.

{"location":{"name":"Adlington (Lancashire)","crs":"ADL","tiploc":"ADNL"},"filter":null,"services":[{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2006","gbttBookedDeparture":"2006","origin":[{"tiploc":"BLCKPLN","description":"Blackpool North","workingTime":"191600","publicTime":"1916"},{"tiploc":"WMER","description":"Windermere","workingTime":"183000","publicTime":"1830"}],"destination":[{"tiploc":"MNCRPIC","description":"Manchester Piccadilly","workingTime":"204500","publicTime":"2045"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2007","realtimeArrivalActual":false,"realtimeDeparture":"2008","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y53223","runDate":"2019-02-11","trainIdentity":"2K54","runningIdentity":"2K54","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2043","gbttBookedDeparture":"2044","origin":[{"tiploc":"MNCRVIC","description":"Manchester Victoria","workingTime":"200500","publicTime":"2005"}],"destination":[{"tiploc":"PRST","description":"Preston","workingTime":"211100","publicTime":"2113"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2043","realtimeArrivalActual":false,"realtimeDeparture":"2044","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y54186","runDate":"2019-02-11","trainIdentity":"2P44","runningIdentity":"2P44","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2103","gbttBookedDeparture":"2104","origin":[{"tiploc":"BLCKPLN","description":"Blackpool North","workingTime":"201400","publicTime":"2014"}],"destination":[{"tiploc":"MNCRPIC","description":"Manchester Piccadilly","workingTime":"214300","publicTime":"2143"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2103","realtimeArrivalActual":false,"realtimeDeparture":"2104","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y53227","runDate":"2019-02-11","trainIdentity":"2K55","runningIdentity":"2K55","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2127","gbttBookedDeparture":"2127","origin":[{"tiploc":"PRST","description":"Preston","workingTime":"210800","publicTime":"2108"}],"destination":[{"tiploc":"MNCROXR","description":"Manchester Oxford Road","workingTime":"220400","publicTime":"2204"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2126","realtimeArrivalActual":false,"realtimeDeparture":"2127","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y50887","runDate":"2019-02-11","trainIdentity":"1Y68","runningIdentity":"1Y68","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true}

Re: How do you remove a header from JSON?

chrisyounger — Mon, 11 Feb 2019 21:16:36 GMT

If you are using the "REST Modular Input" you might be able to set "Response Handler" to be "JSONArrayHandler" and it should split it into individual events properly.

If that doesn't work, then i think you should set LINE_BREAKER to split correctly by events:

LINE_BREAKER = (^[^\[]+\[|,){"locationDetail"

This will discard the header as well as split events correctly.

Good luck!

Re: How do you remove a header from JSON?

stevenbutterwor — Tue, 12 Feb 2019 12:25:22 GMT

Hello, thanks for the advice but none of it worked 😞

Any ideas about removing that line that's not needed as it's still being indexed.

Thanks!

Re: How do you remove a header from JSON?

stevenbutterwor — Tue, 29 Sep 2020 23:12:56 GMT

Here is the props

[realtimetrains:basic]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = json
LINE_BREAKER = (^[^[]+[|,){"locationDetail"
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false

Re: How do you remove a header from JSON?

rashid47010 — Tue, 14 Dec 2021 11:53:40 GMT

did you find the solution

topic How do you remove a header from JSON? in Getting Data In

How do you remove a header from JSON?

Re: How do you remove a header from JSON?

Re: How do you remove a header from JSON?

Re: How do you remove a header from JSON?

Re: How do you remove a header from JSON?