Why is one field not been extracted as expected?

[glpadilla_sol]

Hello everyone,

I have an issue with one field let say foo

These are the scenarios:

1. If I run a search just with the index that contains the logs I can see the field foo at the fields bar perfectly and also I can see the values.

2. If I select the field and is added to the search for example index=bar foo="hello" the results are ZERO even though I select that value from the previous search (where I saw the field and the values at the field bar).

3. If I add the sourcetype at the search example index=bar sourcetype=net foo="hello" I can see results but not the expected results, usually I get less than the real number and the number of results are random in the same interval of time.

Configuration:

I am using the automatic key-value field extraction KV_MODE=json to try to extract the fields of a source. The sources is sending the logs in JSON format.  

#props.conf - SH configuration and indexers

[net]
CHARSET=UTF-8
KV_MODE=json
TRUNCATE = 99999
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=true

#inputs.conf - forwarder

[monitor:///pat/*.json]
sourcetype= net
index = bar
disabled = false
crcSalt = <SOURCE>
ignoreOlderThan = 1d

Version 8.2.2

Cluster enviroment.

Notes:

Also I tried with another sourcetype and data not using the KV_MODE instead I used the EXTRACT-foo and I had the same results, the field doesn't show results when is added to the search.

The rest all the fields are not having this issue, they work perfectly.

Thank you for the help.

[glpadilla_sol · Accepted solution]

I found the root cause, updating in case someone else is facing the same issue:

https://www.splunk.com/en_us/blog/tips-and-tricks/cannot-search-based-on-an-extracted-field.html?_gl...

The solution

$SPLUNK_HOME/etc/system/local/fields.conf
[MyField]
INDEXED_VALUE = false

[cpetterborg]

It kind of looks like you might have created an indexed field extraction using EXTRACT-foo. If that is the case, try the following search:

index=bar foo::"*hello*"

[VatsalJagani]

@glpadilla_sol - Try searching like this:

index=bar foo="*hello*" | where foo="hello"

(added wildcard(*) in the search command.)

I hope this helps!!!

[glpadilla_sol]

Thank you again.

I tried with that search and no results, if I add the sourcetype, 

index=bar sourcetype="test" foo="*hello*" | where foo="hello"

I can see results but not all the results, it's a partial view, that also happens when I run this 

index=bar sourcetype ="test" foo="hello" 

Both scenarios I got the same results.

Also I notice at the search.log that the search is running against some extractions that are not part of that sourcetype. I mean we have the same field been extracted into another sourcetype, but at indexing time using props.conf and transforms.conf.

Can this for some reason be related? 

Event though are different sourcetypes?

Thank you for your inputs.

[glpadilla_sol]

Hello @VatsalJagani thank you for the answer, but the issue is not the search. I mean no matter if I use * or not we are not getting results.

The issue is that I can see the field and the values at the field bar, but no when I add it at the search.

Kind Regards.

[isoutamo]

Maybe this explains it https://community.splunk.com/t5/Splunk-Search/Why-is-it-when-you-don-t-put-a-wild-card-when-searchin... ?

[VatsalJagani]

@glpadilla_sol - It's not because of the search you need to add *. 

But there is some concept about minor and major segmentation that sometimes will not allow you to get results even when you search the extracted value.

[isoutamo]

Can you post a sample from your json file? Preferably from source as raw events inside <> -block.

[glpadilla_sol]

Hello,

Unluckily I cannot share that content because is confidential. 

But is a normal JSON like this

{"backendhost": "2.11.27.94", "backendport": 1001, "console_device": "unknown", "console_name": "sdj934", "domain": "domainname", "frontendport": 8347, "frontendprotocol": "tcp"}

Regards.