Splunk search fails with content that contains a hyphen (-)?

[itbetter]

We're running into something weird where searches may fail. We think it is due to dashes

index="kubernetes" pod="podname-3220973768-tr295" - This fails

index="kubernetes" pod="podname*" - this works fine

Another example
index="kubernetes" fix-something-here - this finds a bunch of entries

index="kubernetes" fix-something-here namespace="kube-system" - This fails

The actual message for the last example is

Found 10 events related to fix-something-here-r2hnp pod in prod namespace

We suspect it is due to the dashes but we're not a hundred percent sure. Does anyone have some tips?

[mattymo]

Hi itbetter!

This experience can be remedied by deploying a fields.conf to the search heads for the index time fields Splunk Connect for Kubernetes sends. You can confirm this by searching with the syntax field::foo and you should see the results you expect.

Default Indexed fields Splunk Connect for Kubernetes sends:
https://github.com/splunk/splunk-connect-for-kubernetes/blob/522a5e57c69d06caef24268737fb65863fdbbad...

More on the fluent-hec plugin fields option here:
https://github.com/splunk/fluent-plugin-splunk-hec#fields-section-optional-single

More on Index time field configs:
https://docs.splunk.com/Documentation/Splunk/7.2.5/Data/Configureindex-timefieldextraction#Where_to_...

What should go in fields.conf :

[namespace]
INDEXED = true

[pod]
INDEXED = true

[container_name]
INDEXED = true

[container_id]
INDEXED = true

[cluster_name]
INDEXED = true

https://docs.splunk.com/Documentation/Splunk/7.2.5/admin/Fieldsconf

Just be aware of any conflicts with any other apps that extract these same field names at search time, as this config would interfere with that.

This has to do with the concept of "Major Breakers" which our good friend @martin_mueller can explain much better than I ever could:

https://conf.splunk.com/files/2017/recordings/fields-indexed-tokens-and-you.mp4
https://conf.splunk.com/files/2017/slides/fields-indexed-tokens-and-you.pdf

[jawaharas]

Thanks a lot. The config change in 'fields.conf' file saved my day.

[mattymo]

@itbetter be sure to accept the answer if it got ya rocking!

[burwell]

1) I am able to search for hosts with hyphens in their names

 host="abc-def"

2) I have data where my field values have hyphens in them, e.g.

 2018-07-28 21:00:00  db_dt="2018-07-28 09:20:00"

I am able to search for

index=main db_dt="2018-07-28 09:20:00"

3) I have seen a different issues with hyphens in fieldnames

If I have keyword=value and my keyword has a hyphen or a period in the name, Splunk converts these to underscores.

For example events like text.abc-foo.disabled="0" will get me a field named text_abc_foo_disabled with value 0

4) Only thing I can think of are your hyphens doubled? e.g. podname followed by two hyphens and you are searching for one?

[PowerPacked]

Hi @itbetter

Can be a possibility that the field values are having invisible spaces,

remove those spaces in the field values by

index="kubernetes" | eval pod=trim(pod) | search pod="podname-3220973768-tr295"

give a try with above search, hope it helps

Thanks

[itbetter]

No joy. Still didn't work. We've learned that splunk uses the dash or hyphen as a wildcard. Is there a way to escape it?