Knowledge Management

Extracted field not displaying in Interesting Fields sidebar

BB_MW
Explorer

Hi, I appreciate that there are numerous questions on here for similar problems but, after reading quite a few of them, nothing seems to quite fit my scenario / issue.

I am trying to extract a field from an event and call it 'action'. The entry in the props.conf looks like :
EXTRACT-pam_action = (Action\: (?P<action>\[[^:\]]+]) )

I know that the extraction is working as there is a field alias later in the props.conf :
FIELDALIAS-aob_gen_syslog_alias_32 = action AS change_type

When I run a basic generating search on the index & sourcetype, the field 'action' does not appear in the 'Interesting Fields' but the 'change_type' alias does appear!

The regex is fine as I can create the 'action' field OK if I add the rex to the search.

I have also added the exact same regex to the props.conf file but called the field 'action1' and that field is displayed OK.

Another test I tried is to create a field alias for the action1 field name called 'action' :
FIELDALIAS-aob_gen_syslog_alias_30 = action1 AS action
FIELDALIAS-aob_gen_syslog_alias_32 = action1 AS change_type
'change_type' is visible but, again 'action' is not visible.

Finally my search
"index=my_index action=*" produces 0 results whereas
"index=my_index change_type-*" produces an accurate output.

I have looked in the props and transforms configs across my searchhead and can't see anything that might be 'removing' my field extraction but, I guess my question is..... how can I debug the creation ( or not ) of a field name?

I have a deep suspicion that it is something to do with one one the Windows TA's apps that we have installed but am struggling to locate the offending configuration 😞

Many thanks for any help.
Mark

Labels (1)
0 Karma
1 Solution

BB_MW
Explorer

I found the issue - there was a rogue local/props.conf in a completely unrelated app that had all sorts of EXTRACTS, FIELDALIASES, etc but, crucially, no stanza spec! One of the extractions defined a field called 'action' with a regex that didn't match anything in my raw event and, because the rogue extract had a higher precendance, my extract didn't get populated.

Rather than ignoring the defined props statements, Splunk applied the EXTRACTS, etc to everything! 

Removing the props.conf solved my issue and everything is good.

 

View solution in original post

0 Karma

BB_MW
Explorer

I found the issue - there was a rogue local/props.conf in a completely unrelated app that had all sorts of EXTRACTS, FIELDALIASES, etc but, crucially, no stanza spec! One of the extractions defined a field called 'action' with a regex that didn't match anything in my raw event and, because the rogue extract had a higher precendance, my extract didn't get populated.

Rather than ignoring the defined props statements, Splunk applied the EXTRACTS, etc to everything! 

Removing the props.conf solved my issue and everything is good.

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @BB_MW ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

one additional hints to look where this action field is defined. 

You could use btool to look that. You will need a cli access to check this. Another option was some addtional app from splunkbase.

splunk btool props list --debug | egrep action

Above command (run it as splunk service user) should show where (which conf file contains that definition) it has defined and how. Especially if you have some TAs etc. which also use same name for field that could be the reason. 

There are some other parameter which you may need if you need to run this in some user + app contex. You should check those from docs or use help option on cli.

r. Ismo

0 Karma

KrishatSplunk
Loves-to-Learn

That is an interesting issue since  you will find everything ok  related to configuration and best guess is that you are hitting the maximum limit of knowledge bundle replication and max_content_length  

Below is the new recommended setting as per splunk document 

on  search head distesearch.conf

[replicationSettings] 
maxBundleSize = 4096

 

you must also increase max_content_length in server.conf on the indexers (search peers) to at least 4GB and also on search head and deployer. 

[httpServer]
max_content_length =  4294967296
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @BB_MW 

in the interesting fields you have fields contained at least in 20% of the events, if you add action=* to you search do you see the field?

Then, did youchecked if the sourcetype that you associated to the field extraction is in the vents that you're searching?

the new field extraction is related only tho the declared sourcetype.

Ciao.

Giuseppe

PickleRick
SplunkTrust
SplunkTrust

Don't bother with the Interesting Fields sidebar. It contains only _extracted fields_ (so if you're searching in fast mode you'll get just the basic metadata fields or the ones explicitly used) which are present in at least 20% of the results.

So this is not the way to verify if the field is properly extracted.

Also remember that when using fast mode only the fields explicitly used are extracted.

BTW, try your search with

| stats count count(action) count(change_type)

 

0 Karma

BB_MW
Explorer

Hi there,

Sorry, I should also have added that I'm searching in Smart Mode. The results, though, are the same for Verbose Mode.

I hadn't thought of doing a stats on the fields but I can confirm that count(action) is still 0 and the count(change_type) has a positive value.

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...