I'm rather new to Splunk. One of the things I have been tasked with is the tracking of API commands sent in URLs to us by our customers. I have some fields that I have extracted that appear correctly in the left Fields part of the screen and work in stats break downs, but when I attempt to search with them the results are inconsistent.
Much of the important information I need to extract is in the "options" portion of the URL.
I want to extract values for each field name that precedes the %3d with the values after it (i.e. "ORD" would be the field and "IV" would be the value). I used the field extractor with a regex similar to the following for each field:
There is one of these for each of the fields in the options. Each of the fields seem to capture the correct values when looked at in the Fields section of Splunk and when using the stats command. However if I search with the following I get no results:
host="mywebserver" ORD=AU :0 events from 1 AM through 7 AM Thursday, March 17, 2011
There is obviously data out there with that value because if I do the following I get the expected results and the fields show as extracted with the AU value:
host="mywebserver" ORD%3dAU :174 events from 1 AM through 7 AM Thursday, March 17, 2011
The really odd thing is that almost the opposite happens when I search for a different value (all the requests have "ORD%3dIV" on them and none have literally "ORD=IV").
host="mywebserver" ORD=IV :2,138 events from 1 AM through 7 AM Thursday, March 17, 2011 host="mywebserver" ORD%3dIV :705 events from 1 AM through 7 AM Thursday, March 17, 2011
Is there something I'm doing wrong with my extraction or possibly something wrong in our Splunk environment?
When you run the searches that doesn't use the field in the search (host="mywebserver" ORD%3dIV) what is the value of the extract ORD field?
Can you have multiple ORDs defined in your events? If so, then this could be causing the problem you're seeing. By default we'll extract only the first value, however you can modify the extractor to extract all occurences of the field and create a multivalued field see MV_ADD in the documentation
Thanks for the quick response. When I run with ORD%3dIV, the value of ORD is IV in 100% of the events.
ORD only appears in a single place in these events.
Can you please provide: (a) some sample data, (b) props/transforms.conf responsible for the field extractions?
I worked with splunk support and they were able to provide some work arounds for this issue. First off they pointed me to another answer here: http://answers.splunk.com/questions/7093.
Here is the answer they provided:
instead of doing the above, you can keep running your searches exactly the way you are running them, however you need to add the following configuration within a fields.conf (say under /etc/apps/search/local/fields.conf) or wherever you are collecting your configs. This third workaround requires a Splunk Restart. Here is the exact stanza you need (cat fields.conf) :
I have confirmed that the first two do work correctly and provide the expected results. We have not had a chance to restart or splunk app as it is under near constant use.