How to achieve Crowdsec json logs fields extraction?

NEHS — Wed, 01 Feb 2023 18:41:07 GMT

Hello Splunk's community,

I got some difficulty for the fields extraction in crowdsec's logs which are format with JSON (using the crowdsec plugin dedicated to this task). I know that there is a lot of post on this forum about json fields extraction but i didn't find any case that could helped me on this.

Firstly here is a sample of an events:

{ [-] capacity: 40 decisions: [ [-] { [-] duration: 4h origin: crowdsec scenario: crowdsecurity/http-crawl-non_statics scope: Ip type: ban value: confidential } ] events: [ [-] { [-] meta: [ [-] { [+] } { [+] } { [-] key: IsInEU value: true } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } ] timestamp: 2023-02-01T15:22:29+01:00 } { [+] } { [+] } { [+] } { [+] } { [+] } ] events_count: 52 labels: null leakspeed: 500ms machine_id: confidential-2@172.18.218.4 message: Ip confidential performed 'crowdsecurity/http-crawl-non_statics' (52 events over 22.814207421s) at 2023-02-01 14:22:29.975537808 +0000 UTC remediation: true scenario: crowdsecurity/http-crawl-non_statics scenario_hash: f0fa40870cdeea7b0da40b9f132e9c6de5e32d584334ec8a2d355faa35cde01c scenario_version: 0.3 simulated: false source: { [-] as_name: confidential as_number: confidential cn: FR ip: confidential latitude: confidential longitude: confidential range: 176.128.0.0/11 scope: Ip value: confidential } start_at: 2023-02-01T14:22:07.161331449Z stop_at: 2023-02-01T14:22:29.97553887Z

I successfully accessed to the fields under 'source' with something like (source.ip, source.as_name) but i can not find a solution for accessing to the value of a field in 'events.meta.IsInEU'. I tried different things with the spath command but unfortunately none of these things worked. I think that the issue is because the fields in meta do not have the same format as in source:

events: [ [-] { [-] meta: [ [-] { [+] } { [+] } {<shoud be a name here>: [-] key: IsInEU value: true }

As you can see above, i think that it would be much easier if there was a name here so i can access to the under key and value (events.meta.should_be_a_name_here.key|value). I don't know if there is some kind of index which i could put to access the data like events{}.meta{0}.key|value. Also i didn't expand the other fields that are aligned with meta because they're all named 'meta' and structure under them is the same than the one which you can see for the first one.

The purpose for all of this would be to make operation such as 'stats count by <value of the key IsInEU'

Thanks in advance for all your answers

Best Regards

Re: How to achieve Crowdsec json logs fields extraction?

sbs2001 — Thu, 02 Feb 2023 06:59:54 GMT

Hi ! Shivam from CrowdSec here. Although I'm not very familiar with Splunk, you can simplify the JSON pushed by CrowdSec to Splunk. This would make your data extraction logic simpler too.

To do this you'd need to override the "format" parameter at "/etc/crowdsec/notifications/splunk.yaml" . The "format" parameter is gotemplate which receives an alert object .

Let us know if you need help here or on our discord

topic Re: How to achieve Crowdsec json logs fields extraction? in Splunk Search

How to achieve Crowdsec json logs fields extraction?

Re: How to achieve Crowdsec json logs fields extraction?