topic Re: Splunk search fails with content that contains a hyphen (-)? in Splunk Search

Splunk search fails with content that contains a hyphen (-)?

itbetter — Thu, 26 Jul 2018 20:09:20 GMT

We're running into something weird where searches may fail. We think it is due to dashes

index="kubernetes" pod="podname-3220973768-tr295" - This fails

index="kubernetes" pod="podname*" - this works fine

Another example
index="kubernetes" fix-something-here - this finds a bunch of entries

index="kubernetes" fix-something-here namespace="kube-system" - This fails

The actual message for the last example is

Found 10 events related to fix-something-here-r2hnp pod in prod namespace

We suspect it is due to the dashes but we're not a hundred percent sure. Does anyone have some tips?

Re: Splunk search fails with content that contains a hyphen (-)?

PowerPacked — Thu, 26 Jul 2018 21:06:52 GMT

Hi @itbetter

Can be a possibility that the field values are having invisible spaces,

remove those spaces in the field values by

index="kubernetes" | eval pod=trim(pod) | search pod="podname-3220973768-tr295"

give a try with above search, hope it helps

Thanks

Re: Splunk search fails with content that contains a hyphen (-)?

itbetter — Fri, 27 Jul 2018 14:07:09 GMT

No joy. Still didn't work. We've learned that splunk uses the dash or hyphen as a wildcard. Is there a way to escape it?

Re: Splunk search fails with content that contains a hyphen (-)?

burwell — Sat, 28 Jul 2018 21:35:37 GMT

1) I am able to search for hosts with hyphens in their names

 host="abc-def"

2) I have data where my field values have hyphens in them, e.g.

 2018-07-28 21:00:00  db_dt="2018-07-28 09:20:00"

I am able to search for

index=main db_dt="2018-07-28 09:20:00"

3) I have seen a different issues with hyphens in fieldnames

If I have keyword=value and my keyword has a hyphen or a period in the name, Splunk converts these to underscores.

For example events like text.abc-foo.disabled="0" will get me a field named text_abc_foo_disabled with value 0

4) Only thing I can think of are your hyphens doubled? e.g. podname followed by two hyphens and you are searching for one?

Re: Splunk search fails with content that contains a hyphen (-)?

mattymo — Tue, 02 Apr 2019 18:32:29 GMT

Hi itbetter!

This experience can be remedied by deploying a fields.conf to the search heads for the index time fields Splunk Connect for Kubernetes sends. You can confirm this by searching with the syntax field::foo and you should see the results you expect.

Default Indexed fields Splunk Connect for Kubernetes sends:
https://github.com/splunk/splunk-connect-for-kubernetes/blob/522a5e57c69d06caef24268737fb65863fdbbade/manifests/splunk-kubernetes-logging/configMap.yaml#L217

More on the fluent-hec plugin fields option here:
https://github.com/splunk/fluent-plugin-splunk-hec#fields-section-optional-single

More on Index time field configs:
https://docs.splunk.com/Documentation/Splunk/7.2.5/Data/Configureindex-timefieldextraction#Where_to_put_the_configuration_changes_in_a_distributed_environment

What should go in fields.conf :

[namespace]
INDEXED = true

[pod]
INDEXED = true

[container_name]
INDEXED = true

[container_id]
INDEXED = true

[cluster_name]
INDEXED = true

https://docs.splunk.com/Documentation/Splunk/7.2.5/admin/Fieldsconf

Just be aware of any conflicts with any other apps that extract these same field names at search time, as this config would interfere with that.

This has to do with the concept of "Major Breakers" which our good friend @martin_mueller can explain much better than I ever could:

https://conf.splunk.com/files/2017/recordings/fields-indexed-tokens-and-you.mp4
https://conf.splunk.com/files/2017/slides/fields-indexed-tokens-and-you.pdf

Re: Splunk search fails with content that contains a hyphen (-)?

jawaharas — Tue, 09 Jul 2019 06:19:47 GMT

Thanks a lot. The config change in 'fields.conf' file saved my day.

Re: Splunk search fails with content that contains a hyphen (-)?

mattymo — Mon, 15 Jul 2019 19:15:35 GMT

@itbetter be sure to accept the answer if it got ya rocking!