Splunk Search

Extracted Fields working inconsistently in Search

Ossian
Explorer

I'm rather new to Splunk. One of the things I have been tasked with is the tracking of API commands sent in URLs to us by our customers. I have some fields that I have extracted that appear correctly in the left Fields part of the screen and work in stats break downs, but when I attempt to search with them the results are inconsistent.

Much of the important information I need to extract is in the "options" portion of the URL.

options=ORD%3DAU+TEXT%3DN+PS%3DN+FormatOnHost%3DN+UNIX%3DY+SENDDATA%3DN+PA%3DGN+REVL%3DY+REVF%3DAU

I want to extract values for each field name that precedes the %3d with the values after it (i.e. "ORD" would be the field and "IV" would be the value). I used the field extractor with a regex similar to the following for each field:

(?i)ORD(%3D|=)(?P<ORD>\w+)

There is one of these for each of the fields in the options. Each of the fields seem to capture the correct values when looked at in the Fields section of Splunk and when using the stats command. However if I search with the following I get no results:

host="mywebserver" ORD=AU :0 events from 1 AM through 7 AM Thursday, March 17, 2011

There is obviously data out there with that value because if I do the following I get the expected results and the fields show as extracted with the AU value:

host="mywebserver" ORD%3dAU :174 events from 1 AM through 7 AM Thursday, March 17, 2011

The really odd thing is that almost the opposite happens when I search for a different value (all the requests have "ORD%3dIV" on them and none have literally "ORD=IV").

host="mywebserver" ORD=IV :2,138 events from 1 AM through 7 AM Thursday, March 17, 2011

host="mywebserver" ORD%3dIV :705 events from 1 AM through 7 AM Thursday, March 17, 2011

Is there something I'm doing wrong with my extraction or possibly something wrong in our Splunk environment?

Tags (2)
1 Solution

Ossian
Explorer

I worked with splunk support and they were able to provide some work arounds for this issue. First off they pointed me to another answer here: http://answers.splunk.com/questions/7093.

Here is the answer they provided:

  1. instead of running your searches as: host= ORD=IV you can run them as host= ORD=*IV - this will work
  2. instead of running your search as host= ORD=IV you can run them as host= | search ORD=IV - this will work
  3. instead of doing the above, you can keep running your searches exactly the way you are running them, however you need to add the following configuration within a fields.conf (say under /etc/apps/search/local/fields.conf) or wherever you are collecting your configs. This third workaround requires a Splunk Restart. Here is the exact stanza you need (cat fields.conf) :

    [ORD] INDEXED_VALUE=false

I have confirmed that the first two do work correctly and provide the expected results. We have not had a chance to restart or splunk app as it is under near constant use.

View solution in original post

0 Karma

Ossian
Explorer

I worked with splunk support and they were able to provide some work arounds for this issue. First off they pointed me to another answer here: http://answers.splunk.com/questions/7093.

Here is the answer they provided:

  1. instead of running your searches as: host= ORD=IV you can run them as host= ORD=*IV - this will work
  2. instead of running your search as host= ORD=IV you can run them as host= | search ORD=IV - this will work
  3. instead of doing the above, you can keep running your searches exactly the way you are running them, however you need to add the following configuration within a fields.conf (say under /etc/apps/search/local/fields.conf) or wherever you are collecting your configs. This third workaround requires a Splunk Restart. Here is the exact stanza you need (cat fields.conf) :

    [ORD] INDEXED_VALUE=false

I have confirmed that the first two do work correctly and provide the expected results. We have not had a chance to restart or splunk app as it is under near constant use.

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

When you run the searches that doesn't use the field in the search (host="mywebserver" ORD%3dIV) what is the value of the extract ORD field?

Can you have multiple ORDs defined in your events? If so, then this could be causing the problem you're seeing. By default we'll extract only the first value, however you can modify the extractor to extract all occurences of the field and create a multivalued field see MV_ADD in the documentation

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Can you please provide: (a) some sample data, (b) props/transforms.conf responsible for the field extractions?

0 Karma

Ossian
Explorer

Thanks for the quick response. When I run with ORD%3dIV, the value of ORD is IV in 100% of the events.

ORD only appears in a single place in these events.

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...