I know there's a ton of these questions out here but I've got one of my own. I've looked at the other questions out there and between them and some initial help from Bert gotten a good start but I can't seem to get this to work right.
We have a single JSON package being received via HEC - this package contains anywhere from 1 to 500 events. Our users would like those events broken out into individual events within Splunk. Here's what the initial package looks like:
{"batch_id":"0-39-1490386204359","sampling_rate":1,"n":500,"events":[{"earlyAccess":null,"profileId":"d037d7da-bd83-11e6-80d6-0a2c5cb56663","ip":"74.000.000.238","deviceVersion":[{"component":"app","build":"oncue#1.19.37.108208#mallard2_GA.0","otherDetail":null},{"component":"service","build":"qa-a.stb.fios.tv","otherDetail":null},{"component":"location","build":"1000.00,1000.00","otherDetail":null},{"component":"Client","build":"108208#release_signed","otherDetail":null}],"__lane":"prod","__host":"unknown","navigationStack":[{"name":"Playback","index":0,"type":"LINEAR"}],"platformVersion":"108208#release_signed","assetType":"LinearAsset","assetId":"50281370fd523ecbaafd8f4d8145e006-adf51f1b5b403b6daa88b62d9c8567fa-2017-03-24-1","restrictedBy":null,"programStartTime":1490380200000,"attributionId":"8cca77ff-1480-b037-b0b2-015b01e09338","assetSessionId":"1490386121819","__sourceType":"device","serviceTimestamp":null,"appVersion":"1.19.37.108208","playType":"tuneIn","collectionId":"","deviceType":"501","accountId":"1004249","playRate":1000,"maestro":{"vhoId":"","host":"qa-a-aws.stb.fios.tv","userAgent":"Mozilla/5.0 (STB; CPU 501 OS 108208) OnCue/1.19.37","version":"4.4.3851","inHome":true,"ipAddress":"74.000.000.238"},"encoderDelay":124000,"__eventId":"s9a98hR/sMGKvAFbAe90tQ==","__timestamp":1490386121909,"sessionId":"617F6EA8-1490385232","__eventName":"1","programEndTime":1490390100000,"programId":"adf51f1b5b403b6daa88b62d9c8567fa","__source":"unknown","liveTuneType":"live","deviceTimestamp":1490386121819,"deviceId":"617F6EA8","__eventVersion":16,"recordingId":null,"channelId":"50281370fd523ecbaafd8f4d8145e006","timeZone":"America/New_York","eventProgramPoint":1490386121714},{"earlyAccess":null,"profileId":"45737761-ac2b-11e6-80d6-0a2c5cb56663","ip":"68.000.000.133","deviceVersion":[{"component":"app","build":"oncue#1.19.40.108253#mallard2_GA.0","otherDetail":null},{"component":"service","build":"qa-a.stb.fios.tv","otherDetail":null},{"component":"location","build":"1000.00,1000.00","otherDetail":null},{"component":"Client","build":"108253#release","otherDetail":null}],"__lane":"prod","__host":"unknown","navigationStack":[{"name":"","index":0,"type":""}],"platformVersion":"108253#release","assetType":"LinearAsset","assetId":"2732f41bdecc33aca2a23146eabd0954-5e4c3aaa6ef7312b8104c94c842d6a3f-2017-03-24-1","restrictedBy":null,"programStartTime":1490385600000,"attributionId":"ffffffff-ffff-ffff-ffff-fffffffffff","assetSessionId":"1490386010685","__sourceType":"device","serviceTimestamp":null,"appVersion":"1.19.40.108253","playType":"tuneOut","collectionId":"","deviceType":"501","accountId":"1003469","playRate":0,"maestro":{"vhoId":"","host":"qa-a-aws.stb.fios.tv","userAgent":"Mozilla/5.0 (STB; CPU 501 OS 108253) OnCue/1.19.40","version":"4.4.3851","inHome":true,"ipAddress":"68.000.000.133"},"encoderDelay":49000,"__eventId":"uXvQcxR/sMGKvAFbAe6R2w==","__timestamp":1490386063835,"sessionId":"617F7743-1490378565","__eventName":"1","programEndTime":1490387400000,"programId":"5e4c3aaa6ef7312b8104c94c842d6a3f","__source":"unknown","liveTuneType":"live","deviceTimestamp":1490386063730,"deviceId":"617F7743","__eventVersion":16,"recordingId":null,"channelId":"2732f41bdecc33aca2a23146eabd0954","timeZone":"America/New_York","eventProgramPoint":1490387400000}]}
So far, I'm apply the following props.conf to this data:
CHARSET=UTF-8
SHOULD_LINEMERGE=false
disabled=false
SEDCMD-removeheader=s/^(\{[\w\W]+\[{"earlyAccess":)/{"earlyAccess":/g
SEDCMD-removeeventcommas=s/},{"earlyAccess":/}{"earlyAccess":/g
SEDCMD-fixfooter=s/\]\}//g
LINE_BREAKER={"earlyAccess
TRUNCATE=0
TIME_PREFIX="deviceTimestamp":
TIME_FORMAT=%s%3N
KV_MODE=json
That gives me this output but doesn't break between events:
{"earlyAccess":null,"profileId":"d037d7da-bd83-11e6-80d6-0a2c5cb56663","ip":"74.000.000.238","deviceVersion":[{"component":"app","build":"oncue#1.19.37.108208#mallard2_GA.0","otherDetail":null},{"component":"service","build":"qa-a.stb.fios.tv","otherDetail":null},{"component":"location","build":"1000.00,1000.00","otherDetail":null},{"component":"Client","build":"108208#release_signed","otherDetail":null}],"__lane":"prod","__host":"unknown","navigationStack":[{"name":"Playback","index":0,"type":"LINEAR"}],"platformVersion":"108208#release_signed","assetType":"LinearAsset","assetId":"50281370fd523ecbaafd8f4d8145e006-adf51f1b5b403b6daa88b62d9c8567fa-2017-03-24-1","restrictedBy":null,"programStartTime":1490380200000,"attributionId":"8cca77ff-1480-b037-b0b2-015b01e09338","assetSessionId":"1490386121819","__sourceType":"device","serviceTimestamp":null,"appVersion":"1.19.37.108208","playType":"tuneIn","collectionId":"","deviceType":"501","accountId":"1004249","playRate":1000,"maestro":{"vhoId":"","host":"qa-a-aws.stb.fios.tv","userAgent":"Mozilla/5.0 (STB; CPU 501 OS 108208) OnCue/1.19.37","version":"4.4.3851","inHome":true,"ipAddress":"74.000.000.238"},"encoderDelay":124000,"__eventId":"s9a98hR/sMGKvAFbAe90tQ==","__timestamp":1490386121909,"sessionId":"617F6EA8-1490385232","__eventName":"1","programEndTime":1490390100000,"programId":"adf51f1b5b403b6daa88b62d9c8567fa","__source":"unknown","liveTuneType":"live","deviceTimestamp":1490386121819,"deviceId":"617F6EA8","__eventVersion":16,"recordingId":null,"channelId":"50281370fd523ecbaafd8f4d8145e006","timeZone":"America/New_York","eventProgramPoint":1490386121714}{"earlyAccess":null,"profileId":"45737761-ac2b-11e6-80d6-0a2c5cb56663","ip":"68.000.000.133","deviceVersion":[{"component":"app","build":"oncue#1.19.40.108253#mallard2_GA.0","otherDetail":null},{"component":"service","build":"qa-a.stb.fios.tv","otherDetail":null},{"component":"location","build":"1000.00,1000.00","otherDetail":null},{"component":"Client","build":"108253#release","otherDetail":null}],"__lane":"prod","__host":"unknown","navigationStack":[{"name":"","index":0,"type":""}],"platformVersion":"108253#release","assetType":"LinearAsset","assetId":"2732f41bdecc33aca2a23146eabd0954-5e4c3aaa6ef7312b8104c94c842d6a3f-2017-03-24-1","restrictedBy":null,"programStartTime":1490385600000,"attributionId":"ffffffff-ffff-ffff-ffff-fffffffffff","assetSessionId":"1490386010685","__sourceType":"device","serviceTimestamp":null,"appVersion":"1.19.40.108253","playType":"tuneOut","collectionId":"","deviceType":"501","accountId":"1003469","playRate":0,"maestro":{"vhoId":"","host":"qa-a-aws.stb.fios.tv","userAgent":"Mozilla/5.0 (STB; CPU 501 OS 108253) OnCue/1.19.40","version":"4.4.3851","inHome":true,"ipAddress":"68.000.000.133"},"encoderDelay":49000,"__eventId":"uXvQcxR/sMGKvAFbAe6R2w==","__timestamp":1490386063835,"sessionId":"617F7743-1490378565","__eventName":"1","programEndTime":1490387400000,"programId":"5e4c3aaa6ef7312b8104c94c842d6a3f","__source":"unknown","liveTuneType":"live","deviceTimestamp":1490386063730,"deviceId":"617F7743","__eventVersion":16,"recordingId":null,"channelId":"2732f41bdecc33aca2a23146eabd0954","timeZone":"America/New_York","eventProgramPoint":1490387400000}
The actual event break should be taking place at:
{"earlyAccess":
I've tried LINE_BREAKER in various formats as well as trying combinations of BREAK_ONLY_BEFORE and MUST_BREAK_AFTER but haven't had any luck getting the breaks to happen - Splunk still processes it all as a single event. Everything else is working fine with it - it's just not breaking. Any assistance on how to get these darn things to break right would be greatly appreciated...
... View more