Splunk Search

multiple field extraction with Regex

dwear
Explorer

I'm trying to use the Extract fields wizard to pull a field out of a log, but running into an issue. Here a portion of an example log.

username1 (WT_AD/RSA) [Webtop I,Webtop III] - 2017/03/14 15:09:57

What I'm interested in is the value in [ ] which is the user's "Role". The problem I have is, that value may be a single value (ie. [Webtop I] ), or possibly multiple values separated by commas. In this example, the user has been assigned 2 "Roles", "Webtop I" and "Webtop III". I was able to get everything inside the [ ] to appear as a single field (i.e. Role="Webtop I, Webtop III"), but I'd like to be able to extract them each as a value (ie. Role="Webtop I" AND Role="Webtop III". This way when I search for Role="Webtop I", I get results when a user was assigned multiple roles as well.

I'm not very good with Regular Expressions, but here is what is working with them as a single value.

^[^)\n]*)[(?P[^]]+)

Any help would be appreciated. Thanks in advance.

Tags (1)
0 Karma

adayton20
Contributor

I had to convert this comment into another answer (so I could insert pictures) to address your other questions. I am assuming you have admin rights to your Splunk.

1.) For the first part, if you are presented with a value of spaces between the brackets, you could simply add a | search yourfield!=" " at the end of your search, which filters out any value with a space in it.

2.) Yes, you can turn this into an extracted field without using props.conf and/or transforms.conf by doing it entirely in the GUI... and for the second part (making a multivalue field split) you can create a calculated field.

Go to Settings > Fields > Field Extractions > New

If you don't have a specific app you're working in, leave that as its default value.

Insert a name for your extraction, the sourcetype where the data resides, and insert the regex from the search excluding the double quotes, like this:

alt text

Save it.

When you save it, you'll be taken back to a section where you can search through other field extractions. Search the name of the field extraction you just created. You will need to change the permissions on it for other apps and/or for other people to see/use it.

Under the sharing column, click permissions.

Where it says "Object should appear in", either change it to "This App" or "Global". If you choose This App, the object will only be available to use in the current app (which is probably the Search app). If you choose Global, it will become a global object available in all apps. Use your discretion as to which roles you'd like to assign read or write permissions to. I usually select read for everyone, and admin for write, but I don't know what kind of environment you're working in.

Once you save it, now you'll need to go to Settings > Fields > Calculated fields

Click New, follow the similar format to the field extractions, and fill in the values.

This time, you're going to insert the eval expression split(yourfield,",")

alt text

Click save, make sure you change your permissions of the calculated field to the same permissions you gave your field extraction.

From here, to apply your extractions and make them visible for use, you can either:

Restart your search head

or

type into the search bar |extract reload=true and then in your address bar of your browser do a debug refresh SearchHeadURL:8000/en-US/debug/refresh and press refresh button - wait for it to complete. I usually do both because for whatever reason one or the other doesn't always work by itself. What this does is force a refresh on splunkd resources.

Now, you should be able go to where your data resides and simply type in index=yourIndex sourcetype=yourSourcetype and see yourfield under "interesting fields"

Lastly, as with this topic and any other question, make sure to accept the answer(s) so others know the question was answered 🙂

Hope that helps!

0 Karma

adayton20
Contributor

Try this:

| rex field=_raw "\[(?P<yourfield>.[^\]]*)"
| makemv delim="," yourfield 

Worked for me:
alt text

dwear
Explorer

So this is awesome, except I left out one piece which I think is breaking the Regex. I accidentally left out the beginning of the log which contains another [ ]. Its identifying the IP address instead of the Role. Here is the log from beginning...

info - [10.34.234.251] - username (WT_AD/RSA)[Webtop I,Webtop III] - 2017/03/14 15:09:57

0 Karma

adayton20
Contributor

EDIT: I just noticed your log from your original post has a space between the closed parenthesis and the open bracket, but the log you posted in your comment does not.

If your log does NOT have a space between the ) and [ try this:

 | rex field=_raw "\)\[(?P<yourfield>.[^\]]*)"

If your log DOES have one or more spaces between the ) and [ try this:

| rex field=_raw "\)\s+\[(?P<yourfield>.[^\]]*)"

If your logs switch between having and not having spaces between the ) and [ try this:

| rex field=_raw "(\)\s+|\))\[(?P<yourfield>.[^\]]*)"

dwear
Explorer

Awesome! Thanks! 2 last questions. Any way to exclude blank [ ]. In the logs it may be blank before the user has been assigned a role and those are showing up. I can always filter the search before hand.

2) Can I turn this into an extracted field for easy future use? Or should I create field extraction from transform as asimagu suggested? Looked through that, but not completely sure I understand what needs to be done.

Thanks for all your help!

0 Karma

adayton20
Contributor

You're very welcome! I went ahead and turned my reply into another post so I could insert pictures for you.

0 Karma

asimagu
Builder

You could also take a look at this command: makemv

First, you could extract your field as single value with all the commas and then use makemv to convert it to multivalue

| makemv delim="," <yourfieldname>

http://docs.splunk.com/Documentation/Splunk/6.5.2/SearchReference/Makemv

0 Karma

asimagu
Builder

Hi dwear

The wizard will not help you with multivalue fields. You need to play with props.conf and transforms.conf

An alternative using the UI would be:

1- Create the extraction with the wizard and copy the regular expression it generates to get your first value
2- Go to Fields -> Create a Tranform
3- Go to Fields -> Create a Field extraction "based on a transform" (the one you just created)

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...