I've seen similar questions to mine asked, but none of the advice has solved my issue.
I created a new field extraction (which correctly pulled the data in the 10,000 event sampling) and it shows up in the field extraction list under "Settings":
API Server Logs - 03-12 : EXTRACT-MW_ErrorMessage Inline (Error Code=").+>(?P<MW_ErrorMessage>[^<]+) mwalser search Private | Permissions Enabled
After creating the field extraction, I attempted to reload the search:
index="cisres_events" sourcetype="API Server Logs - 03-12" | extract reload=T
But the newly created extraction "MW_ErrorMessage" does not show up in the selectable list of "All Fields".
What else might be causing the field extraction to not show up in the list? I've attempted to rebuild this extraction several times with different naming conventions and even tried modifying the permissions to no avail. Any suggestions?
So, it was a small UI foible that was hiding my newly created fields. In the field list there is a "coverage" drop down. "Coverage 1% or more" was selected value, but the extracts I had created only showed up in 0.8% of the logs I was exploring. It appears 1% is the default since I hadn't changed this value prior to discovering my issue.
So, it was a small UI foible that was hiding my newly created fields. In the field list there is a "coverage" drop down. "Coverage 1% or more" was selected value, but the extracts I had created only showed up in 0.8% of the logs I was exploring. It appears 1% is the default since I hadn't changed this value prior to discovering my issue.
Is your search mode set to Verbose
(not Fast
, not Smart
)?
A field is only interesting
if it occurs in 95ish% (I forget the exact number) of events that are returned in the search. If your field is very rare, it is not interesting
and will not show up. You can however select it from the Field Picker
and that makes it show up as Selected
.
Also, have you hit the _bump
endpoint to refresh your session?
it's been a much longer day than I originally thought
It turns out the the Field Picker was showing "Coverage 1% or more" and not "All Fields". These extractions were only occurring in about 0.8% of records, so they were not showing up on the 1% list.
I have added this as the answer but it has not been moderated yet.
So I got you 90+% to the answer!
The breadcrumbs got me there eventually. 🙂 I had not made the connection between the "interesting" metric and the coverage in the drop down. Honestly, I had never really noticed that "coverage" values before because other extractions I had created showed up in the majority of records, so it was never an issue until I started hunting for rare instances.
Fields are relative to sourcetype, are you searching on the same sourcetype you used when creating the field?
Another thing to note, field extractions take a minute or two to appear after creating them
What is the sharing permission on the field extractions? Is it private, This app or Global? If it's private or This App, when you're searching, are you in the same app context?
It was previously "private" but I have modified it to "App (search)" and in both instances it has not shown up. How do I know if I am in the same "app context"?
When I experienced this before, I went directly from the field creation back to the Search by clicking on the link that reads "Explore using this field in search" (or words to that effect). I.E. - I had used the back link that takes one back to the search from which the new extraction is created, and it still did not show up.
If you created the field extraction in search, then if it's private OR this App, then you should be searching from Search app only. Also, did you verify that the regex works fine? May be try to run like this to see if the regex is correct.
index="cisres_events" sourcetype="API Server Logs - 03-12" | rex "(Error Code=").+>(?P<MW_ErrorMessage>[^<]+)"
Not sure if the regex got changed when you posted here, but it seems off. Try this for regex.
(Error Code=\").+\>(?P<MW_ErrorMessage>[^\<]+)
For more accurate suggestions, share your sample event from where you're trying to extract the field.
Thanks for the regex heads up. Yeah, I did have an error from a bad edit, but that wasn't the issue. I posted the answer below,
Thanks again for your help!