Getting Data In

Valid JSON not being broken up into individual events

BongoTheWhippet
Path Finder

I've checked a number of threads about breaking JSON files and I've tried a number of offered solutions and none seem to work.

I'm running 8.1.0 and I don't remember seeing this as much of an issue in previous versions.

The snort (ids-u2json) JSON is lint-valid as follows:

 

{"type": "event", "event": {"msg": "ET INFO Microsoft Connection Test", "classification": "Potentially Bad Traffic", "sensor-id": 0, "event-id": 581, "event-second": 1607588446, "event-microsecond": 790456, "signature-id": 2031071, "generator-id": 1, "signature-revision": 2, "classification-id": 3, "priority": 2, "sport-itype": 63591, "dport-icode": 80, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "192.168.1.125", "destination-ip": "13.107.4.52"}}
{"type": "event", "event": {"msg": "ET POLICY PE EXE or DLL Windows file download HTTP", "classification": "Potential Corporate Privacy Violation", "sensor-id": 0, "event-id": 582, "event-second": 1607588467, "event-microsecond": 769440, "signature-id": 2018959, "generator-id": 1, "signature-revision": 4, "classification-id": 33, "priority": 1, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}
{"type": "event", "event": {"msg": "ET INFO Packed Executable Download", "classification": "Misc activity", "sensor-id": 0, "event-id": 583, "event-second": 1607588467, "event-microsecond": 769340, "signature-id": 2014819, "generator-id": 1, "signature-revision": 1, "classification-id": 29, "priority": 3, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}

 

props.conf on the UF is as follows:

 

[sourcetype=json]
KV_MODE=json
AUTO_KV_JSON=true
NO_BINARY_CHECK = true
disabled = false
SHOULD_LINEMERGE = false
TIME_FORMAT = "event-second": %s, "event-microsecond": %6N
LINE_BREAKER = }}(^s)

 

 and props.conf on the indexer/search head as follows:

 

[stanza]
TZ = UTC
SHOULD_LINEMERGE = false

[_json]
DATETIME_CONFIG =
LINE_BREAKER = }}
NO_BINARY_CHECK = true
disabled = false
KV_MODE = json

[json_no_timestamp]
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
disabled = false

 

According to what I've told the UF to do in props.conf, the JSON events should be splitting up the JSON events using the double braces LINE_BREAKER }} as follows:

 

{"type": "event", "event": {"msg": "ET INFO Microsoft Connection Test", "classification": "Potentially Bad Traffic", "sensor-id": 0, "event-id": 581, "event-second": 1607588446, "event-microsecond": 790456, "signature-id": 2031071, "generator-id": 1, "signature-revision": 2, "classification-id": 3, "priority": 2, "sport-itype": 63591, "dport-icode": 80, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "192.168.1.125", "destination-ip": "13.107.4.52"}}
{"type": "event", "event": {"msg": "ET POLICY PE EXE or DLL Windows file download HTTP", "classification": "Potential Corporate Privacy Violation", "sensor-id": 0, "event-id": 582, "event-second": 1607588467, "event-microsecond": 769440, "signature-id": 2018959, "generator-id": 1, "signature-revision": 4, "classification-id": 33, "priority": 1, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}
{"type": "event", "event": {"msg": "ET INFO Packed Executable Download", "classification": "Misc activity", "sensor-id": 0, "event-id": 583, "event-second": 1607588467, "event-microsecond": 769340, "signature-id": 2014819, "generator-id": 1, "signature-revision": 1, "classification-id": 29, "priority": 3, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}

 

but it doesn't.

Instead, the UF clumps them together as a single event and only reports on the first JSON stanza. Nothing I've tried for LINE_BREAKER seems to work - the UF seems to ignore it. Many thanks

Labels (3)
0 Karma
1 Solution

BongoTheWhippet
Path Finder

Okay. This looks like a bug. There's no way the JSON events should be clumped like this.

View solution in original post

0 Karma

BongoTheWhippet
Path Finder

Okay. This looks like a bug. There's no way the JSON events should be clumped like this.

0 Karma
Get Updates on the Splunk Community!

Splunk at Cisco Live 2025: Learning, Innovation, and a Little Bit of Mr. Brightside

Pack your bags (and maybe your dancing shoes)—Cisco Live is heading to San Diego, June 8–12, 2025, and Splunk ...

Splunk App Dev Community Updates – What’s New and What’s Next

Welcome to your go-to roundup of everything happening in the Splunk App Dev Community! Whether you're building ...

The Latest Cisco Integrations With Splunk Platform!

Join us for an exciting tech talk where we’ll explore the latest integrations in Cisco + Splunk! We’ve ...