Valid JSON not being broken up into individual events

BongoTheWhippet — Thu, 10 Dec 2020 09:56:36 GMT

I've checked a number of threads about breaking JSON files and I've tried a number of offered solutions and none seem to work.

I'm running 8.1.0 and I don't remember seeing this as much of an issue in previous versions.

The snort (ids-u2json) JSON is lint-valid as follows:

{"type": "event", "event": {"msg": "ET INFO Microsoft Connection Test", "classification": "Potentially Bad Traffic", "sensor-id": 0, "event-id": 581, "event-second": 1607588446, "event-microsecond": 790456, "signature-id": 2031071, "generator-id": 1, "signature-revision": 2, "classification-id": 3, "priority": 2, "sport-itype": 63591, "dport-icode": 80, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "192.168.1.125", "destination-ip": "13.107.4.52"}} {"type": "event", "event": {"msg": "ET POLICY PE EXE or DLL Windows file download HTTP", "classification": "Potential Corporate Privacy Violation", "sensor-id": 0, "event-id": 582, "event-second": 1607588467, "event-microsecond": 769440, "signature-id": 2018959, "generator-id": 1, "signature-revision": 4, "classification-id": 33, "priority": 1, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}} {"type": "event", "event": {"msg": "ET INFO Packed Executable Download", "classification": "Misc activity", "sensor-id": 0, "event-id": 583, "event-second": 1607588467, "event-microsecond": 769340, "signature-id": 2014819, "generator-id": 1, "signature-revision": 1, "classification-id": 29, "priority": 3, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}

props.conf on the UF is as follows:

[sourcetype=json] KV_MODE=json AUTO_KV_JSON=true NO_BINARY_CHECK = true disabled = false SHOULD_LINEMERGE = false TIME_FORMAT = "event-second": %s, "event-microsecond": %6N LINE_BREAKER = }}(^s)

and props.conf on the indexer/search head as follows:

[stanza] TZ = UTC SHOULD_LINEMERGE = false [_json] DATETIME_CONFIG = LINE_BREAKER = }} NO_BINARY_CHECK = true disabled = false KV_MODE = json [json_no_timestamp] NO_BINARY_CHECK = true SHOULD_LINEMERGE = false disabled = false

According to what I've told the UF to do in props.conf, the JSON events should be splitting up the JSON events using the double braces LINE_BREAKER }} as follows:

{"type": "event", "event": {"msg": "ET POLICY PE EXE or DLL Windows file download HTTP", "classification": "Potential Corporate Privacy Violation", "sensor-id": 0, "event-id": 582, "event-second": 1607588467, "event-microsecond": 769440, "signature-id": 2018959, "generator-id": 1, "signature-revision": 4, "classification-id": 33, "priority": 1, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}

{"type": "event", "event": {"msg": "ET INFO Packed Executable Download", "classification": "Misc activity", "sensor-id": 0, "event-id": 583, "event-second": 1607588467, "event-microsecond": 769340, "signature-id": 2014819, "generator-id": 1, "signature-revision": 1, "classification-id": 29, "priority": 3, "sport-itype": 80, "dport-icode": 63676, "protocol": 6, "impact-flag": 0, "impact": 0, "blocked": 0, "mpls-label": null, "vlan-id": null, "pad2": null, "source-ip": "205.185.216.10", "destination-ip": "192.168.1.125"}}

but it doesn't.

Instead, the UF clumps them together as a single event and only reports on the first JSON stanza. Nothing I've tried for LINE_BREAKER seems to work - the UF seems to ignore it. Many thanks

Re: Valid JSON not being broken up into individual events

BongoTheWhippet — Mon, 14 Dec 2020 11:23:30 GMT

Okay. This looks like a bug. There's no way the JSON events should be clumped like this.

topic Re: Valid JSON not being broken up into individual events in Getting Data In

Valid JSON not being broken up into individual events

Re: Valid JSON not being broken up into individual events