topic Re: Rex field extraction in Splunk Search

Rex field extraction

zacksoft — Mon, 02 Jul 2018 13:23:33 GMT

Could someone help me extract the two bold words from the following sample
SAMPLE EVENT 1
2018-07-02 08:51:44,648 https-nsse-nio-8663-exec-18 LRQ9923 531x698404x16 1kvc79 99.103.154.114,30.128.209.1 /best/madget/1.0/login The user 'LRQ9923' has PASSED authentication.
Could someone help me extract the three bold words from the following sample
SAMPLE EVENT 2
2018-07-02 09:18:44,761 https-nsse-nio-8663-exec-90 anonymous 558x723020x25 5lqwk7 88.128.203.123,30.118.254.78 /best/madget/1.0/login The user 'JRA3620' has FAILED authentication. Failure count equals 3

Re: Rex field extraction

493669 — Mon, 02 Jul 2018 13:34:00 GMT

Hi @zacksoft,
try this:

|rex "user\s\'(?<User>[^']+)'\shas\s(?<Result>\w+)\sauthentication.*(count\sequals\s(?<Count>\d+))?"

try it in regex101: https://regex101.com/r/vtbCOg/1

Re: Rex field extraction

zacksoft — Mon, 02 Jul 2018 14:02:29 GMT

@493669
For the event of following type.
It won't give me the 'Failure Count'

2018-07-02 09:18:44,761 https-nsse-nio-8663-exec-90 anonymous 558x723020x25 5lqwk7 88.128.203.123,30.118.254.78 /best/madget/1.0/login The user 'JRA3620' has FAILED authentication. Failure count equals 4

Re: Rex field extraction

woodcock — Mon, 02 Jul 2018 14:06:31 GMT

Try this:

user\s+'?(?<user>[^\s']+)'?\shas\s(?<outcome>\S+)\sauthentication.(?:\s+Failure count equals\s+(?<failure_count>\d+))?

Re: Rex field extraction

zacksoft — Mon, 02 Jul 2018 14:14:40 GMT

In the events like the following , it won't give me the Failure Count

Re: Rex field extraction

woodcock — Mon, 02 Jul 2018 14:26:09 GMT

Quite correct, I updated my answer.

Re: Rex field extraction

niketn — Mon, 02 Jul 2018 14:41:46 GMT

@zacksoft this should work with the sample data you have provided. Following is a run anywhere search based on the data provided:

| makeresults
| eval data="2018-07-02 08:51:44,648 https-nsse-nio-8663-exec-18 LRQ9923 531x698404x16 1kvc79 99.103.154.114,30.128.209.1 /best/madget/1.0/login The user 'LRQ9923' has PASSED authentication.;2018-07-02 09:18:44,761 https-nsse-nio-8663-exec-90 anonymous 558x723020x25 5lqwk7 88.128.203.123,30.118.254.78 /best/madget/1.0/login The user 'JRA3620' has FAILED authentication. Failure count equals 3;2018-07-02 09:18:44,761 https-nsse-nio-8663-exec-90 anonymous 558x723020x25 5lqwk7 88.128.203.123,30.118.254.78 /best/madget/1.0/login The user 'JRA3620' has FAILED authentication. Failure count equals 4"
| makemv data delim=";"
| mvexpand data
| rename data as _raw
| rex  "user\s+'?(?<user>[^\s']+)'?\shas\s(?<outcome>\S+)\sauthentication.(?:\s+Failure count equals\s+(?<failure_count>\d))?"

Re: Rex field extraction

zacksoft — Tue, 03 Jul 2018 07:56:53 GMT

Thank you @woodcock

Re: Rex field extraction

FrankVl — Tue, 03 Jul 2018 08:03:38 GMT

Make sure to change that last \d to a \d+ if the failure count can be higher than 9.

Re: Rex field extraction

zacksoft — Tue, 03 Jul 2018 08:06:37 GMT

Sure @FrankVI

Would you happen to know any way to automatically generate Splunk rex extraction commands by selecting what fields we want to fetch from the event?

Re: Rex field extraction

FrankVl — Tue, 03 Jul 2018 09:19:17 GMT

When you unfold an event in Splunk, there is a button called "Event Actions", which has an action "Extract Fields".

This brings you to Splunk's Field Extractor GUI, and that does allow you to mark fields in the event and let Splunk generate the regex. You can then either store that as an automatic field extraction, or copy paste the resulting regex into a search query.

But generally this doesn't really result in the best quality regexes. It is really worthwhile investing a little bit of time in learning how regular expressions work and then writing them yourself, with the help of tools like regex101.com. That way you keep it in your control and don't rely on magic you don't fully understand.

Re: Rex field extraction

woodcock — Tue, 03 Jul 2018 12:40:40 GMT

Updated again