Why do I need to search metadata in a "fields" argument of an HEC with '::'?

[barkery]

I have some logging being sent into an HTTP Event Collector like this:
Endpoint: https://myeventcollector.com:8088/services/collector/event
Request Payload: { "sourcetype": "$filename", "fields": { "ip_address": "$ip_address", "file_path": "$log" }, "time": $timestamp, "event": $line }

According to this document fields, "Specifies a JSON object that contains explicit custom fields to be defined at index time. Requests containing the "fields" property must be sent to the /collector/event endpoint, or they will not be indexed".
http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/FormateventsforHTTPEventCollector

I am seeing the "ip_address" and "file_path" fields being correctly extracted, but am experiencing strange behavior when I try to filter them. If I specify ip_address="10.119.32.165" in the search, I get partial results from my search. However, when I specify ip_address::10.119.32.165 in the search, all events return.

I was able to find this similar question, but why does Splunk consider data from the "fields" argument of a HEC metadata when the documentation literally calls them fields? Also, I'm familiar with how a Splunk forwarder works and know that a deployment app uses '::' to specify metadata during indexing, but why can't I use = to search search metadata in the first place?
https://answers.splunk.com/answers/562832/add-to-search-returns-no-results.html

Just looking to understand what is happening here better, as this will be very confusing for my users.

Thank you,

Ryan

[rpendleton]

Did you ever figure this out? I'm seeing the same thing and am wondering what I'm missing, since the documentation seems misleading to me as well. Even using the web UI to search for the "fields" uses = and produces no results, but changing the search to :: works fine.

[ifeldshteyn]

In the same boat as you. What is the point of providing this parameter if you can't search this data?

After a lot of searching through the docs and other posts, for anyone else who is stuck, here is a workaround.

You need to go to your $SPLUNK_HOME/etc/system/local/fields.conf (create it if you don't have it) and then for each field you want to search with an equal sign create a stanza that says...

[FIELDNAME1]
INDEXED_VALUE = false

[FIELDNAME2]
INDEXED_VALUE = false

...

[FIELDNAMEN]
INDEXED_VALUE = false

If you have only a few fields that should be fine, if you have 100s, I hope you can automate this file creation.

After a reboot the fields become searchable with = sign instead of :: .

Credit where it is due --> https://answers.splunk.com/answers/666826/in-splunk-hec-what-should-you-check-if-you-cannot.html