Extracted field not displaying in Interesting Fields sidebar

[BB_MW]

Hi, I appreciate that there are numerous questions on here for similar problems but, after reading quite a few of them, nothing seems to quite fit my scenario / issue.

I am trying to extract a field from an event and call it 'action'. The entry in the props.conf looks like :
EXTRACT-pam_action = (Action\: (?P<action>\[[^:\]]+]) )

I know that the extraction is working as there is a field alias later in the props.conf :
FIELDALIAS-aob_gen_syslog_alias_32 = action AS change_type

When I run a basic generating search on the index & sourcetype, the field 'action' does not appear in the 'Interesting Fields' but the 'change_type' alias does appear!

The regex is fine as I can create the 'action' field OK if I add the rex to the search.

I have also added the exact same regex to the props.conf file but called the field 'action1' and that field is displayed OK.

Another test I tried is to create a field alias for the action1 field name called 'action' :
FIELDALIAS-aob_gen_syslog_alias_30 = action1 AS action
FIELDALIAS-aob_gen_syslog_alias_32 = action1 AS change_type
'change_type' is visible but, again 'action' is not visible.

Finally my search
"index=my_index action=*" produces 0 results whereas
"index=my_index change_type-*" produces an accurate output.

I have looked in the props and transforms configs across my searchhead and can't see anything that might be 'removing' my field extraction but, I guess my question is..... how can I debug the creation ( or not ) of a field name?

I have a deep suspicion that it is something to do with one one the Windows TA's apps that we have installed but am struggling to locate the offending configuration 😞

Many thanks for any help.
Mark

[BB_MW · Accepted solution]

I found the issue - there was a rogue local/props.conf in a completely unrelated app that had all sorts of EXTRACTS, FIELDALIASES, etc but, crucially, no stanza spec! One of the extractions defined a field called 'action' with a regex that didn't match anything in my raw event and, because the rogue extract had a higher precendance, my extract didn't get populated.

Rather than ignoring the defined props statements, Splunk applied the EXTRACTS, etc to everything! 

Removing the props.conf solved my issue and everything is good.

[gcusello]

Hi @BB_MW ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[isoutamo]

Hi

one additional hints to look where this action field is defined. 

You could use btool to look that. You will need a cli access to check this. Another option was some addtional app from splunkbase.

splunk btool props list --debug | egrep action

Above command (run it as splunk service user) should show where (which conf file contains that definition) it has defined and how. Especially if you have some TAs etc. which also use same name for field that could be the reason. 

There are some other parameter which you may need if you need to run this in some user + app contex. You should check those from docs or use help option on cli.

r. Ismo

[KrishatSplunk]

That is an interesting issue since  you will find everything ok  related to configuration and best guess is that you are hitting the maximum limit of knowledge bundle replication and max_content_length  

Below is the new recommended setting as per splunk document 

on  search head distesearch.conf

[replicationSettings] 
maxBundleSize = 4096

you must also increase max_content_length in server.conf on the indexers (search peers) to at least 4GB and also on search head and deployer. 

[httpServer]
max_content_length =  4294967296

[gcusello]

Hi @BB_MW 

in the interesting fields you have fields contained at least in 20% of the events, if you add action=* to you search do you see the field?

Then, did youchecked if the sourcetype that you associated to the field extraction is in the vents that you're searching?

the new field extraction is related only tho the declared sourcetype.

Ciao.

Giuseppe

[PickleRick]

Don't bother with the Interesting Fields sidebar. It contains only _extracted fields_ (so if you're searching in fast mode you'll get just the basic metadata fields or the ones explicitly used) which are present in at least 20% of the results.

So this is not the way to verify if the field is properly extracted.

Also remember that when using fast mode only the fields explicitly used are extracted.

BTW, try your search with

| stats count count(action) count(change_type)

[BB_MW]

Hi there,

Sorry, I should also have added that I'm searching in Smart Mode. The results, though, are the same for Verbose Mode.

I hadn't thought of doing a stats on the fields but I can confirm that count(action) is still 0 and the count(change_type) has a positive value.