Splunk Search

Cannot search for field but field is shown in field list

afx
Contributor

I have a totally weird case...
I have field extractions defined in props.conf either individually or all in one extraction, no difference.
The fields show up in the "Interesting Fields" list.

When I then select a field and pick a value, the result is empty.
For example

index=myindex sourcetype=mysrc myfield=A

Shows nothing even though the popup told me there are 100 entries of myfield with contents A.
But if I then flip the search to exclude A I get results...

index=myindex sourcetype=mysrc myfield!=A

This happens for several fields defined for that index/source type but not all and I see no pattern why some work and some do not (position seems irrelvant).
If I use the fieldsummary filter, I find things just fine.
Splunk 7.4.2, 2 indexers, one SH

So why can't I select field contents that Splunk clearly knows about?

I am totally lost.

thx
afx

0 Karma
1 Solution

afx
Contributor
0 Karma

afx
Contributor

Ok, as promised, here is my SAP Security Audit Log solution:
https://answers.splunk.com/answers/757714/how-to-splunk-the-sap-security-audit-log.html

cheers
afx

0 Karma

afx
Contributor

Ok, meanwhile I have stumbled across this: Fixed Width data streamsfield extractions

This resulted in me trying to put delimiters in via a transform on_raw and then define the fields via the delimiter method. So far it seem successful, I'll do some more tests and will report back.

cheers
afx

0 Karma

woodcock
Esteemed Legend

If your values are not separated by major/minor-breakers (segmenters.conf), then they will not appear as indexed values in your tsidx so you have to tell the Search Head that these fields are not indexed fields by adding this to fields.conf:

[myfield]
INDEXED_VALUE = false

See details here:
https://www.splunk.com/blog/2011/10/07/cannot-search-based-on-an-extracted-field.html

BE CAREFUL! This is a GLOBAL setting and will effect all fields named myfield so you would be best off naming this field something that is uncommon (unlikely to be used by anybody else anywhere else).

afx
Contributor

Tried that already ;-(
No change.
And from reading the first line of that blog post I would assume that this is outdated anyway.

0 Karma

woodcock
Esteemed Legend

Nope. It is still exactly the way that it works and I believe that is almost certainly your problem. Did you deploy this to your Search Heads and either restart Splunk there or debug/refresh?

0 Karma

afx
Contributor

I did and it makes no change at all.
See last answer post, looks like I have a solution now.
Will post once testing is finished and the thing is a bit more polished.

0 Karma

FrankVl
Ultra Champion

This type of issue was actually covered in today's Splunk Troubleshooting class. The training material suggests this solution (also in fields.conf):

[message_id]
INDEXED_VALUE=<VALUE>*

This automatically replaces the term "AUK" with "AUK*" in the lispy search.

0 Karma

FrankVl
Ultra Champion

And when you search for "AUK" or "AUK*" (so without the field name). And if you get results, what part of the event is getting highlighted?

I'm guessing this has something to do with bloom filters failing due to your event being one long string.

0 Karma

jnudell_2
Builder

Searching for just "AUK" won't work, because there is no space after the AUK value. That's why AUK* works.

0 Karma

FrankVl
Ultra Champion

Yeah, and I'm guessing that is also why message_id="AUK" fails, since Splunk also applies its bloom filters to such searches as far as I know.

0 Karma

afx
Contributor

It is not the one long string, as I wrote in the first post, this also happens with individual definitions.

Ok, tried just the string and I do not get any results...
Which is weird.

I do get results for AUK* or "AUK*" which then highlights everything until the first space.

0 Karma

FrankVl
Ultra Champion

And that is probably your issue. When searching for field="foo", Splunk uses bloom filters to exclude buckets where the keyword "foo" does not exist. Since your event doesn't get broken up into keywords properly, the bloom filter will cause your search to not return any results, because the keyword "AUK" does not occur anywhere.

Might be best to switch off bloomfilters for this index I guess. Unless anyone else has any brilliant ideas of how to fix this.

0 Karma

afx
Contributor

Ok how do I turn them off?

0 Karma

afx
Contributor

I've put createBloomfilter = false into the indexes.conf file in the appropriate stanza.
Then pushed the masteter config.
So far no change.

0 Karma

jnudell_2
Builder

Hi @afx ,
Based upon your answers in the comments, you are probably not using the right search term(s). Can you provide a sample of what an event looks like that contains the data you're looking for? Highlight the value that is supposed to be the myfield value you're looking for.

Here's another thing you might try (seems to match your case):
https://www.splunk.com/blog/2011/10/07/cannot-search-based-on-an-extracted-field.html

0 Karma

afx
Contributor

Why would I not use the right search terms?
Splunk is offering me the terms in the popup menus for the fields.
For example

AUK20190614155300000387400019D110.42.24S_ESA_ANG SAPMSSY1 1001ARFC&&ARFC_DEST_CONFIRM 10.42.242.200

Should be found when I select message_id=AUK from the interesting fields sidebar.
But I get no results
(The copy and paste did collapse spaces in this record. The extract definition above shows where things are. Just Imagine plenty of spaces to fill the fixed size records where you see a single space, this line is always exactly 200 characters wide).

0 Karma

jnudell_2
Builder

What happens when you try message_id="*AUK*"?

0 Karma

afx
Contributor

No results found ;-(

0 Karma

jnudell_2
Builder

I would also try this:

... [ your search without message_id= ] ...
| rex "^(?<message_id_test>...)"
| search message_id_test="AUK"

If that works, then I think there's something wonky with your field extractions, and it showing up on the interesting fields is definitely odd.

0 Karma

afx
Contributor

This works just fine.
If Splunk would not show me the fields in the selection list, then this would all be simply my regex incompetence.
But as it shows the fields jus as defined, I am stumped.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...