Cannot search for field but field is shown in field list

[afx]

I have a totally weird case...
I have field extractions defined in props.conf either individually or all in one extraction, no difference.
The fields show up in the "Interesting Fields" list.

When I then select a field and pick a value, the result is empty.
For example

index=myindex sourcetype=mysrc myfield=A

Shows nothing even though the popup told me there are 100 entries of myfield with contents A.
But if I then flip the search to exclude A I get results...

index=myindex sourcetype=mysrc myfield!=A

This happens for several fields defined for that index/source type but not all and I see no pattern why some work and some do not (position seems irrelvant).
If I use the fieldsummary filter, I find things just fine.
Splunk 7.4.2, 2 indexers, one SH

So why can't I select field contents that Splunk clearly knows about?

I am totally lost.

thx
afx

[jnudell_2]

Does it work when you do:

index=myindex sourcetype=mysrc message_id=*

| search message_id = "AUK"

[afx]

Nope, that does not work.

[jnudell_2]

Can you setup a "dummy" search head (new instance) and install only that one SAP TA/Add-on, and see if you get the same behavior? Also, if there are different versions of the TA, try the older version to see if the results differ.

You could also disable the TA (this will turn off all the field extractions) and then create just the one specific EXTRACT line in the Search & Reporting App for the field extraction and see if that works?

[afx]

Copied one sample file and the TA do a dedicated test box.
No change.
Used rex inline. Then it works. But why?
thx
afx

[jnudell_2]

Very weird, check this blog post out, and see if it helps:
https://www.splunk.com/blog/2011/10/07/cannot-search-based-on-an-extracted-field.html

[afx]

I've seen that one, but it stats this issue is fixed since 4.3.
Would think it is still valid?

[jnudell_2]

Probably not, but it seems like it matches your situation regardless. There's a ton of troubleshooting that can be done, but I think that would require a better medium than Splunk Answers.

[jnudell_2]

You could also try @FrankVl 's suggestion about bloomfilters. I'm not sure if that's the right direction, but let us know if it works out.

[afx]

Neither the bloomfilters nor the fields.conf file help.
Time to go up the corporate chain to get a ticket openend I guess ;-(

Thanks for all the suggestions!

[jnudell_2]

What do you get when you try:

| ... [ your search without message_id= ] ...

| stats dc(message_id) as message_id_count values(message_id) as message_id_values

Do you get a numeric value for the count and a list of message_id values?

[afx]

Yes, that shows me a count of 15 and 15 different Ids.

[FrankVl]

Can you share the props/transforms for how that specific field is being extracted?

[afx]

This is the props.conf file:

[sap:sal]
EXTRACT-sal = ^(?<message_id>.{3})(?<date>.{8})(?<time>.{6})(\w\w)(?<process_id>.{5})(?<task>.{5})(?<proctype>.{2})(?<term>.{8})(?<user>.{12})(?<transaction>.{20})(?<app>.{40})(?<client>.{3})(?<message>.{64})(?<src>.{20})

LOOKUP-auto_sap_sm20 = sap_sm20 message_id AS message_id OUTPUTNEW audit_class AS sap_audit_class event_class AS sap_event_class message AS sap_message new_in_release AS sap_new_in_release

transforms.conf:

[sap_sm20]
batch_index_query = 0
case_sensitive_match = 1
filename = SAP_SM20.csv

[FrankVl]

And what field(s) do you have issues with? The ones from the lookup perhaps?

[afx]

That would be easy 😉
The ones that work are user, app,src, sap_audit_class.
They don't seem to follow a pattern.
Even weirder, sap_audit_class is based on message_id and works, but message_id itself does not...

[FrankVl]

And you're not overlooking whitespace in the field values or so?

[afx]

Nope.
I select values from the dropdown menus that splunk offers me for the fields.

[FrankVl]

Yeah, I thought that's what you meant with how you described your testing in the question post, just wanted to double check.

Well then I'm running out of ideas...

[jnudell_2]

What results do you see when you table the events:

index=myindex sourcetype=mysrc 

| table index sourcetype myfield

| sort myfield

Do you see values in myfield?

[afx]

Yes, they show up there.
I also did this:

 index=myindex sourcetype=mysource 
    | fieldsummary 
    | search field="myfield" 
    | table field count distinct_count values

And the result is just perfectly fine.

[jnudell_2]

And if you do:

index=myindex sourcetype=mysrc 

| table index sourcetype myfield

| search myfield="A"

What do you see?