topic Re: Missing data on ingest in Splunk Search

Why is there Missing data on ingestion?

ReginaP — Fri, 18 Aug 2023 18:06:48 GMT

Brand news servers. Not receiving all data from the UF.
Confirmed connectivity.
Confirmed inputs via "/opt/splunkforwarder/bin/splunk btool inputs list | grep bc_ | grep "\["",
Only getting 2 sourcetypes when there should be at least 16 for the index.

Getting this error message:
Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).

Getting this when starting splunkd:

Splunk> Take the sh out of IT.

Checking prerequisites...
        Management port has been set disabled; cli support for this configuration is currently incomplete.
        Checking conf files for problems...
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.3-dd0128b1f8cd-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

Re: Missing data on ingest

gcusello — Fri, 18 Aug 2023 07:21:24 GMT

Hi @ReginaP,

did you disabled the local firewall on the system?

did you checked the connectivity between the forwarder and the Indexer using telnet on the port you configure for receiving?

did you enabled receiving on the Indexers?

the "Invalid key" error isn't relevant, is instead very strange the message "Management port has been set disabled; cli support for this configuration is currently incomplete".

for the second message, you could see the solution at https://community.splunk.com/t5/Getting-Data-In/Unable-to-start-Splunk-forwarder/m-p/386916 but anyway, iy shouldn't have effects on the log forwarding.

Ciao.

Giuseppe

Ciao.

Giuseppe

Re: Missing data on ingest

isoutamo — Fri, 18 Aug 2023 09:20:50 GMT

which user you are running splunkd on UF? You should use separate non root user. But this means that you must give access to this user to access those files which you want to read. You should try to check if user have access those e.g.

sudo -s root bash sudo -s <your splunk user> bash less /path/to/log/file

If you cannot see the content of that file you need to give access by e.g. "setfacl" command.

r. Ismo

Re: Missing data on ingest

ReginaP — Fri, 18 Aug 2023 12:51:56 GMT

We have root running splunkd on all our UFs with no issues.

Re: Missing data on ingest

ReginaP — Fri, 18 Aug 2023 12:58:26 GMT

I checked the connection via telnet to the correct port.

Re: Missing data on ingest

isoutamo — Fri, 18 Aug 2023 13:17:30 GMT

Actually this is huge security issue, but it shouldn't affect to your current issue.

When you do that "splunk btool inputs ...." you get 16+ sourcetypes of inputs, but on splunk there is only 2 of them?

What you get with

splunk list inputstatus

For your missing bc_* inputs? There should be something like

/your/input/file/bc_something file position = 631 file size = 631 parent = /var/log percent = 100.00 type = finished reading

Re: Missing data on ingest

ReginaP — Fri, 18 Aug 2023 18:04:59 GMT

It was a RHEL8 python issue. Thank you for your responses