Search filter returning odd results

[jansvensen]

Hey everyone!

Hope you are doing alright and my question is in the right place here. 

For a few days, i am seeing a strange issue with my searches. I do mainly use text fields and filter them in my dashboards. Nothing too wild going on. Worked for a while, does not anymore, now. 

This example describes the issue quite well:

I am search from an index called "dt_statusreport". Configuring a filter "Country"="*" will give me all entries in the index:

If i now change the filter to "Country"="CA", no results will be fould:

The same occurs if i add the filter by clicking it from the results from search 1.

Even more odd things happen, if I filter for "Country"="DE". Magically, the count reduces from 101 to 12.

I have no darn idea where this might come from. If you can push me to any direction, i'll be very glad!

Cheers,

Sven

[jansvensen]

New findings: I played around with test data a bit more. Getting an interesting outcome. splunk version is 8.2.0 in this case. 

Input field: "Country"

Calculated field: ctry = tostring(Country)

Filter by "Country" --> filter does not work / results as described here

Filter by "ctry" --> filter does work / results are displayed properly

It seems i can work around it with setting the field data type to "string". Is there a way to define all fields of an index?

[jansvensen]

8.2.0 seems to break my searches on windows and ubuntu. interesting.

[jansvensen]

Just rolled back my Splunk on Windows to 8.1.3 --> works perfectly. with the same input script.

8.2.0 on Ubuntu still no chance. Fresh installed system.

[jansvensen]

Chaps, thank you for your support so far. I did not get the time to try your recommendations, but will do so next week. I'll update then! 

[fredchase]

Under the green search magnifying glass, click on the "Fast Mode" and when the menu opens up, choose "Verbose Mode". It might fix the problem.

[yuanliu]

Try

index=dt_statusreport Country=*CA*

 ?

[jansvensen]

Adding the asterisks actually does do the trick in terms of filtering. It does bring new perks, of course.

 Data is created via a simple script. Just sending json over. The script i am testing with just now is:

$Counter = 1

do {


    $TestValueArray=@(
        "Yes",
        "No",
        "No",
        "No",
        "No",
        "Yesa",
        "No",
        "Yesa",
        "No",
        "No",
        "Yesb",
        "No",
        "No"
    )

    $TestValue = $TestValueArray[(Get-Random –Minimum 0 –Maximum 12)]

$json = @"
{
    "host": "Test",
    "index": "dt_test",
    "event": "Status Report Log Entry created for Test",
    "fields": {
        "Test Value": "$TestValue"
    }
}
"@
    
    $Headers = @{
        Authorization = "Splunk c39425b6-380b-46f0-b705-f381b046a031"
        ContentType = "application/json"
    }

    #write-host "Invoke-WebRequest -Uri "http://10.10.10.39:8088/services/collector" -Method "POST" -Body $json -Headers $Headers"
    Invoke-WebRequest -Uri "http://10.10.10.39:8088/services/collector" -Method "POST" -Body $json -Headers $Headers  | Out-Null

    $counter = $counter + 1
    
}

until ($counter -gt 1000)

[jansvensen]

< index = "dt_test" "Test value"=*Yes* > does include "Yes", "Yesa" and "Yesb" now, of course. 
< index = "dt_test" "Test value"=Yes > does not include anything

[yuanliu]

If this test data set results in different search behavior in 8.1.3 and 8.2.0, I suspect that the upgrade has some errors.   Did you test  in a newly installed 8.2.0 on the same platform that has this problem?

[jansvensen]

Hi!

My Ubuntus were 8.2.0 from scratch, Windows was updated. I'll retry this week and give an update here.

[yuanliu]

From your test data generator, I have another suspicion: Is it possible that your 8.2 installations somehow collapse multiple events into one event so some "Test Value" become multivalued?  If "Test Value" = ["Yes", "Yesa"], it will match *Yes*, but not "Yes".

Since this data is synthetic, maybe you can share some sample events as ITWhisperer suggested?  In your real data, if some events have Country = ["DE"], some have Contry = ["DE", "CA"], Country = "DE" will only return those with single valued "DE".

You can also sanitize your real data using "index=dt_statusreport | table Country" and share in this forum.  This will only show Country codes, nothing else in your data.

Alternatively, share multi-value test output using this

index=dt_statusreport
| eval mv_country = mvcount(Country)
| stats count by mv_country

 If every event is single-valued, you should see one count with mv_country == 1.

[jansvensen]

it does seem, all values are singles:

still seeing the issue. This is splunk 7.2.0 now.

[yuanliu]

---

@jansvensen wrote:

it does seem, all values are singles:
...

still seeing the issue. This is splunk 7.2.0 now.

---

This is quite confusing.

• First sentence.  Your screenshot showing mv_country are all 1, is it tested in 7.2 or in 8.2?  If the problem is 8.2 specific, you need to test in 8.2.
• Second sentence.  Do you mean that 7.2 also starts to show the bad behavior?

[jansvensen]

Hi! 

Sorry, if i was not clear.

I do have different versions in my lab. All are basic Ubuntu with splunk installed the same way. I'm using the same version of test script in every VM. So far my findings are:

8.2.0: issue exists

8.1.4: working

8.1.3: working

8.0.0: working

7.2.0: issue exists

[isoutamo]

If you haven’t create a support ticket, you should do it now.

[jansvensen]

Thanks! I'll do so!

[isoutamo]

You should check from job inspector what is the real search what splunk has combined for you based on your search and possible searchFilters etc.

Job -> Inspect Job -> search.log or Search Job Properties.

r. Ismo

[ITWhisperer]

Can you share some suitable anonymised events?