Splunk Search

How do I split a log entry into tags?

Communicator

Good evening all,

I was hoping to get an idea of the best practices in breaking out a custom field.

My log records have the usual stuff--the level, the date, etc. Then, after a variable number of characters, is a field that looks something like this:

[TAGS: A, B, C]

[TAGS: A, D]

[TAGS: D, C, G]

[TAGS: E, F]

Ideally, I'd like to be able to specify the source, and then select the tag values without a lot of complicated regexes for the users.

source=/mysource.log TAGS=A OR TAGS=D

I was looking through the information on extracting new fields and thought that I'd ask the forum on the best way to do this.

Thanks in advance.

Edited to add:
OK, I found an even better way, if I can get it to work. A drop down for just this source would be the best--no typing! 😄 Is this a best practice? Can you get the drop down to populate from every value in the TAGS field so that the drop down lists A, B, C, D , E , F, G?

0 Karma
1 Solution

Motivator

Extracting tags

One way is to extract the full list of tags into a field, then separately split the list into individual values.

In transforms.conf:

[extract-tags]
# Extract everything between the two brackets into a field
REGEX = \[TAGS: ([^\]]+)\]
FORMAT = mytags::$1

[split-tags]
# Take each section that does not include a comma as a separate value
SOURCE_KEY = mytags
MV_ADD = True
REGEX = ([^,]+)
FORMAT = mytag::$1

In props.conf:

[mysourcetype]
# The two field extractions must run in the correct order, since split-tags
# only works if there is already a value in the 'mytags' field.
REPORT-0-tags = extract-tags
REPORT-1-tags = split-tags

Also, DON'T try to name your resulting field tag. The word tag has special meaning in the Splunk search language; doing so may produce strange results when searching.



Dropdown menus

It's not completely clear what you're asking for, but it should be doable.

If you extract the fields as above, you will get the usual built-in menus in Field Discovery at the left, and under field values when you show the field in search results.

You can also create dropdown menus as part of search dashboards (form search). You can either populate them from a fixed list or from a search. Using a hard-coded list will be faster if you know all possible tag values ahead of time. The exact process may vary depending on which version of Splunk you're using.

For Splunk 4.2 and Simple XML, try: http://docs.splunk.com/Documentation/Splunk/4.2.5/Developer/AddDropDowns. For Advanced XML, you can use a SearchSelectLister or StaticSelectLister module.

View solution in original post

Motivator

Extracting tags

One way is to extract the full list of tags into a field, then separately split the list into individual values.

In transforms.conf:

[extract-tags]
# Extract everything between the two brackets into a field
REGEX = \[TAGS: ([^\]]+)\]
FORMAT = mytags::$1

[split-tags]
# Take each section that does not include a comma as a separate value
SOURCE_KEY = mytags
MV_ADD = True
REGEX = ([^,]+)
FORMAT = mytag::$1

In props.conf:

[mysourcetype]
# The two field extractions must run in the correct order, since split-tags
# only works if there is already a value in the 'mytags' field.
REPORT-0-tags = extract-tags
REPORT-1-tags = split-tags

Also, DON'T try to name your resulting field tag. The word tag has special meaning in the Splunk search language; doing so may produce strange results when searching.



Dropdown menus

It's not completely clear what you're asking for, but it should be doable.

If you extract the fields as above, you will get the usual built-in menus in Field Discovery at the left, and under field values when you show the field in search results.

You can also create dropdown menus as part of search dashboards (form search). You can either populate them from a fixed list or from a search. Using a hard-coded list will be faster if you know all possible tag values ahead of time. The exact process may vary depending on which version of Splunk you're using.

For Splunk 4.2 and Simple XML, try: http://docs.splunk.com/Documentation/Splunk/4.2.5/Developer/AddDropDowns. For Advanced XML, you can use a SearchSelectLister or StaticSelectLister module.

View solution in original post

Motivator

Something like sourcetype=XXX TAGS | head 1000 | dedup tag | sort tag should help with the dropdown search.

Depending on what your data looks like, you may need to adjust for performance, and 1000 records may or may not be enough to see all tags. There are some things you can do if it's too slow (scheduled searches in advance or summary indexing), but basics first...

Communicator

LOL....

OK, you got me there! It would be helpful to probably include the correct name of the log file, huh?

I'm still working on the search to get just the tags to populate the dropdown list, but this has helped a lot.

0 Karma

Motivator

Make sure that what you have in props.conf for [mysourcetype] matches the sourcetype field on your events exactly.

Also try running a search for source=/mysource.log TAGS | kv extract-tags, and see if you get the mytags field. If it works that way, Splunk isn't running the extractions automatically, typically because props.conf is wrong.

mytag won't show up until you can see mytags, since the process to get the individual tags relies on the value of the whole tag section being extracted first.

It's ok if some events don't have any tags; the fields will just be missing for those.

Communicator

I've put in the transforms and props changes, but I'm not seeing how they've worked. The configuration changes are being applied--I threw in a sourcetype change and that seems to have carried through to the web pages when I view the logs.

So I'm left with searching that's being problematic.

The entire log line looks like this:
(date)::[Log Level such as "INFO"]::[[TAGS: A, B, C]] Log entry text. Log may or may not have TAGS at all.

I'm not seeing in the Field Discovery anything looking like "mytags" or "mytag".

Thank you for your continuing help. I'm learning a lot from you.

0 Karma

Motivator

In the example XML, it first runs a search and fills the dropdown with the results. The search string is what goes in the 'what goes here' bit. After the search is run, it will fill the dropdown from the resulting values. You need to also replace the 'suser' from the example with 'mytag' or whatever you call your extracted field.

Then, <choice> tells it to append a hard-coded option to the menu as well.

IF you can be sure of knowing all possible tags ahead of time, just use the hardcoded option and put them in manually.

Either way, you still need to extract the fields first.

Communicator

Good afternoon,

Thanks. I'm trying this now.

As far as the drop down, I was looking for something along those line, but I'm not doing too good on the query to get the tags to appear in the dropdown.

Using:



<![CDATA[*Not sure what goes here***]]>
nagrin
<choice value="">Any

Like I said, I'm trying to your pattern to work, so I'll let you know how it goes.

Thanks a bunch!

0 Karma