topic Re: JSON auto extraction not occuring for a single field, `src` but occurring for all other fields in Knowledge Management

JSON auto extraction not occuring for a single field, `src` but occurring for all other fields

mbrownoutside — Sat, 13 Nov 2021 11:04:24 GMT

Me and another engineer were taking a look at `index=corelight sourcetype=corelight_notice signature="Scan::*"`.

We noticed that `src` was not properly parsed given `kv_mode=auto`.

We've attempted the follwing four course of action:
1. performed an EXTRACT on _raw as : "src":"(?<src>[^"]+)",
2. performed a REPORT as: corelight_notice_src
* with a transform as `"src":"(?<src>[^"]+)",` on _raq
3. perform an EXTRACT on _raw as : \"src\":\"(?<src>[^\"]+)\",
4. * performed a REPORT as: corelight_notice_src
* with a transform as `* \"src\":\"(?<src>[^\"]+)\",`

Note that performing the `| rex field=_raw "\"src\":\"(?<src>[^\"]+)\","` at search time works fine.

We also attempted with `AUTO_KV_JSON = false` with the above tests 3 and 4, which failed.

We also attempted with `AUTO_KV_JSON = false` and `KV_MODE = none` with the above tests 3 and 4, which failed

Note that the following works:

```
index=corelight sourcetype=corelight_notice signature="Scan::*"
| spath output=src path=src
```

When AUTO_KV_JSON=true, then most JSON fields are extracted (except for src).

When AUTO_KV_JSON=true and KV_MODE=json, then most JSON fields are extracted (except for src).

Any ideas on what the problem is?

```

{"_path":"notice","_system_name":"zEEK01","_write_ts":"2021-11-12T23:22:24.722517Z","ts":"2021-11-12T23:22:24.722517Z","note":"Scan::Address_Scan","msg":"kk: 192.168.0.1 scanned at least 27 unique hosts on ports 443/tcp, 80/tcp in 42m29s","sub":"local","src":"192.168.0.1","peer_descr":"proxy-01","actions":["Notice::ACTION_LOG"],"suppress_for":1,"severity.level":3,"severity.name":"error"}

```

Thanks,

Matt

Re: JSON auto extraction not occuring for a single field, `src` but occurring for all other fields

PickleRick — Sat, 13 Nov 2021 06:39:13 GMT

Are you sure it's a well-formed json?

The value for "suppress_for" field doesn't seem right.

Re: JSON auto extraction not occuring for a single field, `src` but occurring for all other fields

mbrownoutside — Sat, 13 Nov 2021 11:05:38 GMT

I apologize. I meant to use "N" as a replacement for a number, but didn't test the JSON.

This is usually a number, and I have corrected the above.

This isn't it, unfortunately.

Re: JSON auto extraction not occuring for a single field, `src` but occurring for all other fields

PickleRick — Sat, 13 Nov 2021 14:05:21 GMT

Well, what can I say. "Works for me".

But this is my home instance of splunk free so I don't have multiple users and access rights. Are you sure nothing overwrites/blocks search-time parsing settings for your user?

Re: JSON auto extraction not occuring for a single field, `src` but occurring for all other fields

mbrownoutside — Wed, 17 Nov 2021 20:19:17 GMT

Yes `AUTO_KV_JSON` being set to true then this should work without issue.

Very abnormal to me as `rex` is working to hit and extract via a named group match.

Re: JSON auto extraction not occuring for a single field, `src` but occurring for all other fields

scelikok — Thu, 18 Nov 2021 08:56:04 GMT

Hi @mbrownoutside,

I see FIELDALIAS for src on TA-CorelightForSplunk props.conf . It is trying to alias id.orig_h AS src.

Could you please try deleting this FIELDALIAS to test? id.orig_h null field may be overwriting the src field as null.