Why is my Splunk index showing "0"?

[tcsec2user]

I push the logs to splunk using hec  method  using this end point "/services/collector" that index data showing in 1 MB in index manger but im search through the index the events are always showing "0". only default configtracker events are showing.

[richgalloway]

If you can see configtracker events then index search is working.  The more likely problem is data is not being sent to HEC correctly.  Tell us more about how HEC is being used.  What is the format of the data?  Do you get a 200 response code?  Is the specified index one of those allowed by the HEC token you're using?  Have you checked the logs for relevant messages?

[tcsec2user]

yes im getting the response 200 

{

    "text": "Success",

    "code": 0

}

like and im using same index token. i have checked the index manager the event count is zero and tha data is not stored in db.what are required changes i need to do ?

[richgalloway]

If you're using the services/collector endpoint then the data must be in JSON format with specific fields specified.  If the data is not in JSON format then you should use the services/collector/raw endpoint.  See https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/FormateventsforHTTPEventCollector#Even...

Have you seen any messages in splunkd.log about this problem?

[tcsec2user]

i have tried with /services/collector/raw also but no luck the data is not correct indexed.i checked with  /services/collector/ack.

[tcsec2user]

this is my splunkd.log

09-16-2022 15:56:08.184 +0000 ERROR ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/phantom/bin/scripts/phantom_retry.py" File "/opt/splunk/etc/apps/phantom/bin/phantom_splunk.py", line 190, in rest
09-16-2022 15:56:08.184 +0000 ERROR ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/phantom/bin/scripts/phantom_retry.py" raise splunk.AuthorizationFailed('Error talking to Splunk: {} {}: {}'.format(method, path, str(e)))
09-16-2022 15:56:08.184 +0000 ERROR ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/phantom/bin/scripts/phantom_retry.py" splunk.AuthorizationFailed: [HTTP 403] Error talking to Splunk: GET /servicesNS/nobody/phantom/configs/conf-phantom: [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/servicesNS/nobody/phantom/configs/conf-phantom?count=-1&output_mode=json
09-16-2022 15:56:22.605 +0000 INFO ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:228] [get_server_roles] [26822] Fetched server roles, roles=['universal_forwarder', 'license_master', 'license_manager']
09-16-2022 15:56:22.611 +0000 INFO ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:256] [get_cluster_mode] [26822] Fetched cluster mode, mode=disabled
09-16-2022 15:56:22.611 +0000 INFO ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:30] [should_run] [26822] should run test, sh=False
09-16-2022 15:56:37.433 +0000 INFO ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:228] [get_server_roles] [26832] Fetched server roles, roles=['universal_forwarder', 'license_master', 'license_manager']
09-16-2022 15:56:37.445 +0000 INFO ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:256] [get_cluster_mode] [26832] Fetched cluster mode, mode=disabled
09-16-2022 15:56:37.445 +0000 INFO ExecProcessor [25121 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:30] [should_run] [26832] should run test, sh=False

[richgalloway]

I see nothing relevant in those log entries.  I'm afraid I'm out of ideas.