topic Data is not getting parsed on HEC in Getting Data In

Data is not getting parsed on HEC

rahulg — Wed, 20 Oct 2021 12:53:09 GMT

I have props.conf

[source::tcp:7660]
TRUNCATE=10000000
LINE_BREAKER = {\"time
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
pulldown_type = true
KV_MODE = json
#TZ = America/Chicago
TZ=UTC
=====================================

I see some of events are not parsed in json format

Re: Data is not getting parsed on HEC

somesoni2 — Wed, 20 Oct 2021 14:01:13 GMT

Is the HEC configured on Heavy forwarder/indexer? Check if events which are not parsed as json is in pure JSON format. Did you setup KV_MODE=JSON on your search head(s)?

Re: Data is not getting parsed on HEC

PickleRick — Wed, 20 Oct 2021 14:12:18 GMT

Which endpoint are you send your events to?

Re: Data is not getting parsed on HEC

rahulg — Mon, 25 Oct 2021 13:55:38 GMT

HEC configured on Heavy forwarder and i dont have KV_MODE=JSON on search head(s) will that help?

Re: Data is not getting parsed on HEC

PickleRick — Mon, 25 Oct 2021 14:11:14 GMT

Firstly, I think you don't need line breaking settings since you supply whole single events to the /event endpoint.

Secondly - KV_MODE is a search-time setting so yes, you need it on search-heads, not on indexers/HF's

Re: Data is not getting parsed on HEC

rahulg — Tue, 02 Nov 2021 06:18:21 GMT

Here is sample events which is working fine

{ [-]
command: C:\Windows\System32\sdfhlsdhjfjsnsdf
company_1: Microsoft Corporation
company_2: Microsoft Corporation
connection_count: 0
created: Mon Nov 1 07:52:10 2021
created_1: Sun Jun 6 1
created_2: Sun Jun 6 14:52:03.721 2021
desc_1: Runtime Broker
desc_2: Host Process for Windows Services
exists_1: yes
exists_2: yes
file_1: C:\Windows\System32\kjksnkfhskf
file_2: C:\Windows\System32\svchost.exe
firstbytes_1: jhsfkszhkfhnkkllks.ndklfsf
firstbytes_2: hkdhfkgkdhfgknzdlfgnl.sdflgndlkfgnld
hostname: nkdnf.ks
imphash_1: .nsdlkfnlszknflsNLfnzslkdnfksnkfnskfn
imphash_2: nsdnfknfaksnfdksnflnfknskdfnksnafdks
legal_copyright_1: © Microsoft Corporation. All rights reserved.
legal_copyright_2: © Microsoft Corporation. All rights reserved.
level: Info
listen_ports:
md5: nbzkdfnkzshdkfjskJnfkznfksnk
md5_1: ksndlfn.ksndfknsakf
md5_2: nKSndkfksdfnksandfknsak
message: Process info
module: ProcessCheck
name: RuntimeBroker.exe
owner: NM\JOIN4029
owner_1: NT SERVICE\TrustedInstaller
owner_2: NT SERVICE\TrustedInstaller
parent: C:\Windows\System32\svchost.exe
path: C:\Windows\System32\RuntimeBroker.exe
pid: 24080
ppid: 1264
scanid: S-bszkdbfksnbdfkjs
sha1_1: kndfnkzdnfkdnakgnkfgnxkdzn
sha1_2: ndxnfvkznfnkmzfxbvkzdbfvkbzkbxdv
sha256_1: oiajsosfu094ursjofjlsjdflk
sha256_2: knsldkflzsdjflkslkf
size_1: 8679890
size_2: 567890
time: 2021-11-01T14:18:26Z
type_1: EXE
type_2: EXE
}

if it has file_1 and file_2 works fine and if addition file_3 or similar sha256_3 or any _etc field ect gets added it doesnt shows ja\son format

Re: Data is not getting parsed on HEC

PickleRick — Tue, 02 Nov 2021 09:16:50 GMT

Check if the fields are present in raw event. Then you'll know if it's a parsing problem or ingestion one.