Hi,
I have a field extraction situaton that I've never come across before, and hoping someone can help me.
We have a number of fields setup to do search-time extractions and transformations. One of the fields is named "action", which looks at the values in the field and transforms them. The transformation works when you do a query that doesn't directly query that field, but if you query the field directly, it isn't found. However, if you wildcard it, the field is found.
Here's my transforms.conf:
[stonesoft_action_blocked]
REGEX = |(Connection_Discarded)|
FORMAT = action::blocked
[stonesoft_action_teardown]
REGEX = |(Connection_Closed(?:-Abnormally)?)|
FORMAT = action::teardown
[stonesoft_action_allowed]
REGEX = action=(Allow|Permit)
FORMAT = action::allowed
If I query "index=myIndex", then the field "action" field appears under "Interesting Fields", with each option — teardown, allowed, and blocked". However, if I click on any of these values, and they get added to the search, it now comes back with zero events. So, "index=myIndex action=blocked" returns nothing. If I enter that directly in the search (rather than clicking on it from the event), it also returns zero events.
If I wildcard the search, and type "index=myIndex action=*blocked*"
, then I get events returned.
Hope this makes sense. Appreciate any advise.
You have to tell the Search Head that these fields are not indexed values (they do not fall between to major/minor breakers) by adding this to fields.conf
:
[action]
INDEXED_VALUE = false
See details here:
https://www.splunk.com/blog/2011/10/07/cannot-search-based-on-an-extracted-field.html
You have to tell the Search Head that these fields are not indexed values (they do not fall between to major/minor breakers) by adding this to fields.conf
:
[action]
INDEXED_VALUE = false
See details here:
https://www.splunk.com/blog/2011/10/07/cannot-search-based-on-an-extracted-field.html
Also, would this all be resolved if he didn't do this type of indexed field approach and used props.conf with a case or if eval to calculate the field?
It isn't indexed, it is search time already (REPORT in props.conf). Just the way it is being done in transforms.conf, with those static values, has some disadvantages that require that setting that woodcock mentioned in order for Splunk to properly handle this field in searches.
Thanks. Can you elaborate? If there a better alternative?
If you can do it from an automatic lookup, that is best, otherwise my solution is your best option. Don't forget to UpVote
any helpful answers and click Accept
, @a212830.
Thanks. Answer should be accepted now. When you say lookup, I'm assuming that you mean some sort of eval?
No, with lookup he means lookup, not eval.
Check these (and related) docs pages:
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/DefineanautomaticlookupinSplunkWeb
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addfieldsfromexternaldatasources
Word. (and thanks for keeping me straight that it is NOT indexed). Check the props.conf details on EVAL-, FIELDALIAS-, EXTRACT-, and so forth. A ton of options.
Honestly, I would suggest going through the UI and creating the field extraction with the Interactive Field Extractor and seeing the config it produces. That's a great way to learn.
BTW: Since we know each other, I recall you used to be primary on the backend while your peer handled more of this UI stuff. Given your new role, might be a good time to get the boss to send you for some of those courses like https://www.splunk.com/en_us/training/learning-path/courses-for-users/advanced-searching-and-reporti...
Thanks. Good idea.
And yes, the fields.conf entry fixed the issue. That said, I'll revist how this was done.
Typically people will just extract the 'raw' action as defined in the event into some field (e.g. vendor_action) and then use a lookup, or an EVAL with an if/case statement to calculate the normalized 'action' field.
Hey Burch,
My intent is not to make these indexed fields. Is there something in this setup that is making these appear as indexed fields?
So just for my understanding, that is because he uses a REPORT transform which assigns a static value which doesn't occur in the event itself?
When you would use an eval in props.conf instead, that wouldn't be an issue, right?
More than that, actually; it is because the string does not occur inside the event OR is in the event but not bounded by segmenters. If the field came from EVAL or LOOKUP, yes, splunk would then understand it. Also, you did not supply your props.conf so I don't know if these are tied to TRANSFORMS-
or REPORT-
. If the former, you could use action::blocked
syntax and it would work.
Thanks 🙂
He did share his props in one of the nested comments: https://answers.splunk.com/comments/702388/view.html
It uses REPORT.
Hello,
Just started to use splunk and i dont have many knowledge about programing language.
Anyone can help me extragting latitude and Longitude so i can use it in geolocation?
Here is a sample of na event:
"2018-07-26 15:29:59 192.168.1.5 GET /igeoearcweb/igeoesig/ExecCmd.asp Lat=34.224167&Lng=-118.063111&SC=WGS84&Scale=25000&cmd=Center 81 - 192.168.1.5 Mozilla/5.0+(compatible;+SemrushBot/2~bl;++http://www.semrush.com/bot.html) - 404 0 2 0"
Thanks
Hi Carlos. I think your post might have been better suited as a new question rather than an answer to the question posted on this page. If so, this page may be effective for getting you help: http://docs.splunk.com/Documentation/Splunkbase/latest/Answers/Questions
Are you literally searching with those double quotes around your entire query? Have you tried not doing that?
Normally you only put quotes around field values and such, not around the entire search query. I've tried searching like that on my own splunk box and that indeed gives weird behavior with no results when you have multiple search criteria (just "index=bla" works fine, "index=bla field=foo" doesn't work).
No, those quotes are not part of the search.
What does the props.conf look like for this?
[stonesoft]
KV_MODE = none
REPORT-leef = stonesoft_transport_id,stonesoft_dest_port,stonesoft_src_port,stonesoft_dest,stonesoft_src,stonesoft_sender,stonesoft_dvc,stonesoft_action_al
lowed,stonesoft_action_blocked,stonesoft_action_teardown
LOOKUP-transport_protocols = transport_protocols transport_id