How to achieve Crowdsec json logs fields extraction?

[NEHS]

Hello Splunk's community,

I got some difficulty for the fields extraction in crowdsec's logs which are format with JSON (using the crowdsec plugin dedicated to this task). I know that there is a lot of post on this forum about json fields extraction but i didn't find any case that could helped me on this.

Firstly here is a sample of an events:

	
{ [-]
   capacity: 40
   decisions: [ [-]
     { [-]
       duration: 4h
       origin: crowdsec
       scenario: crowdsecurity/http-crawl-non_statics
       scope: Ip
       type: ban
       value: confidential
     }
   ]
   events: [ [-]
     { [-]
       meta: [ [-]
         { [+]
         }
         { [+]
         }
         { [-]
           key: IsInEU
           value: true
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
         { [+]
         }
       ]
       timestamp: 2023-02-01T15:22:29+01:00
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
     { [+]
     }
   ]
   events_count: 52
   labels: null
   leakspeed: 500ms
   machine_id: confidential-2@172.18.218.4
   message: Ip confidential performed 'crowdsecurity/http-crawl-non_statics' (52 events over 22.814207421s) at 2023-02-01 14:22:29.975537808 +0000 UTC
   remediation: true
   scenario: crowdsecurity/http-crawl-non_statics
   scenario_hash: f0fa40870cdeea7b0da40b9f132e9c6de5e32d584334ec8a2d355faa35cde01c
   scenario_version: 0.3
   simulated: false
   source: { [-]
    as_name: confidential
    as_number: confidential
    cn: FR
    ip: confidential
    latitude: confidential
    longitude: confidential
    range: 176.128.0.0/11
    scope: Ip
    value: confidential 
   }
   start_at: 2023-02-01T14:22:07.161331449Z
   stop_at: 2023-02-01T14:22:29.97553887Z 

 I successfully accessed to the fields under 'source' with something like (source.ip, source.as_name) but i can not find a solution for accessing to the value of a field in 'events.meta.IsInEU'. I tried different things with the spath command but unfortunately none of these things worked. I think that the issue is because the fields in meta do not have the same format as in source:

events: [ [-]
     { [-]
       meta: [ [-]
         { [+]
         }
         { [+]
         }
         {<shoud be a name here>: [-]
           key: IsInEU
           value: true
         }

As you can see above, i think that it would be much easier if there was a name here so i can access to the under key and value (events.meta.should_be_a_name_here.key|value). I don't know if there is some kind of index which i could put to access the data like events{}.meta{0}.key|value. Also i didn't expand the other fields that are aligned with meta because they're all named 'meta' and structure under them is the same than the one which you can see for the first one.

The purpose for all of this would be to make operation such as 'stats count by <value of the key IsInEU'

Thanks in advance for all your answers

Best Regards

[sbs2001]

Hi ! Shivam from CrowdSec here. Although I'm not very familiar with Splunk, you can simplify the JSON pushed by CrowdSec to Splunk. This would make your data extraction logic simpler too.

To do this you'd need to override the "format" parameter at "/etc/crowdsec/notifications/splunk.yaml" . The "format" parameter is gotemplate which receives an alert object .

Let us know if you need help here or on our discord