Getting Data In

How do you remove a header from JSON?

stevenbutterwor
Path Finder

Hi all

I'm ingesting some JSON via REST API, but the events are all squashed into one large event. I'm pretty sure it's because there is a header at the top of the file that needs to be removed for the JSON to be parsed correctly.

I think, again, I need to add SEDCMD to the props, but I'm unsure of the regex required. Any help would be appreciated — below is a sample of the JSON. The bold section is the header.

{"location":{"name":"Adlington (Lancashire)","crs":"ADL","tiploc":"ADNL"},"filter":null,"services":[{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2006","gbttBookedDeparture":"2006","origin":[{"tiploc":"BLCKPLN","description":"Blackpool North","workingTime":"191600","publicTime":"1916"},{"tiploc":"WMER","description":"Windermere","workingTime":"183000","publicTime":"1830"}],"destination":[{"tiploc":"MNCRPIC","description":"Manchester Piccadilly","workingTime":"204500","publicTime":"2045"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2007","realtimeArrivalActual":false,"realtimeDeparture":"2008","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y53223","runDate":"2019-02-11","trainIdentity":"2K54","runningIdentity":"2K54","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2043","gbttBookedDeparture":"2044","origin":[{"tiploc":"MNCRVIC","description":"Manchester Victoria","workingTime":"200500","publicTime":"2005"}],"destination":[{"tiploc":"PRST","description":"Preston","workingTime":"211100","publicTime":"2113"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2043","realtimeArrivalActual":false,"realtimeDeparture":"2044","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y54186","runDate":"2019-02-11","trainIdentity":"2P44","runningIdentity":"2P44","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2103","gbttBookedDeparture":"2104","origin":[{"tiploc":"BLCKPLN","description":"Blackpool North","workingTime":"201400","publicTime":"2014"}],"destination":[{"tiploc":"MNCRPIC","description":"Manchester Piccadilly","workingTime":"214300","publicTime":"2143"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2103","realtimeArrivalActual":false,"realtimeDeparture":"2104","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y53227","runDate":"2019-02-11","trainIdentity":"2K55","runningIdentity":"2K55","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true},{"locationDetail":{"realtimeActivated":true,"tiploc":"ADNL","crs":"ADL","description":"Adlington (Lancashire)","gbttBookedArrival":"2127","gbttBookedDeparture":"2127","origin":[{"tiploc":"PRST","description":"Preston","workingTime":"210800","publicTime":"2108"}],"destination":[{"tiploc":"MNCROXR","description":"Manchester Oxford Road","workingTime":"220400","publicTime":"2204"}],"isCall":true,"isPublicCall":true,"realtimeArrival":"2126","realtimeArrivalActual":false,"realtimeDeparture":"2127","realtimeDepartureActual":false,"displayAs":"CALL"},"serviceUid":"Y50887","runDate":"2019-02-11","trainIdentity":"1Y68","runningIdentity":"1Y68","atocCode":"NT","atocName":"Northern","serviceType":"train","isPassenger":true}

Tags (2)
0 Karma

chrisyounger
SplunkTrust
SplunkTrust

If you are using the "REST Modular Input" you might be able to set "Response Handler" to be "JSONArrayHandler" and it should split it into individual events properly.

If that doesn't work, then i think you should set LINE_BREAKER to split correctly by events:

LINE_BREAKER = (^[^\[]+\[|,){"locationDetail"

This will discard the header as well as split events correctly.

Good luck!

0 Karma

stevenbutterwor
Path Finder

Hello, thanks for the advice but none of it worked 😞

Any ideas about removing that line that's not needed as it's still being indexed.

Thanks!

0 Karma

stevenbutterwor
Path Finder

Here is the props

[realtimetrains:basic]
DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = json
LINE_BREAKER = (^[^[]+[|,){"locationDetail"
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false

0 Karma

rashid47010
Communicator

did you find the solution

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...