topic Re: TailingProcessor File Status without port 8089? in Splunk Search

TailingProcessor File Status without port 8089?

vcarbona — Thu, 27 Mar 2014 14:16:00 GMT

We would like to have forwarders run as root in order to overcome file permissions. However, we also will be security hardening it as much as possible. One of these measures is to stop port 8089 on the forwarder. I assume this will not give us the ability to read the REST endpoint https://hostname:8089/services/admin/inputstatus/TailingProcessor%3AFileStatus. Are there any other ways to gather this data without the REST endpoint being available?

Re: TailingProcessor File Status without port 8089?

somesoni2 — Thu, 27 Mar 2014 16:21:18 GMT

How about using search?
index=_internal component=TailingProcessor

Re: TailingProcessor File Status without port 8089?

martin_mueller — Thu, 27 Mar 2014 16:36:08 GMT

You can call the endpoint directly from the CLI:

./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

Re: TailingProcessor File Status without port 8089?

vcarbona — Thu, 27 Mar 2014 16:43:24 GMT

That would have been cool, but I get this:

ps -ef|grep splunk

splunk 5604 1 53 11:39 ? 00:00:13 splunkd -p 8089 restart
splunk 5605 5604 0 11:39 ? 00:00:00 [splunkd pid=5604] splunkd -p 8089 restart [process-runner]

./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus
QUERYING: 'https://127.0.0.1:8089/services/admin/inputstatus/TailingProcessor:FileStatus'
This command [GET /services/admin/inputstatus/TailingProcessor:FileStatus] needs splunkd to be up, and splunkd is down.

Re: TailingProcessor File Status without port 8089?

martin_mueller — Thu, 27 Mar 2014 17:19:14 GMT

So you've made port 8089 unavailable even from localhost? Then it might indeed be tough to call the REST API.

Re: TailingProcessor File Status without port 8089?

vcarbona — Thu, 27 Mar 2014 18:45:22 GMT

Thanks! This actually helped us to identify some problem areas. But it appears conditions like "ignored file (crc conflict, needs crcSalt)" or "File did not match whitelist ..." do not show up.

Re: TailingProcessor File Status without port 8089?

vcarbona — Mon, 28 Sep 2020 17:00:28 GMT

I noticed the btprobe command shows some interesting data about the file status. It appears it is able to retrieve the modtime and seek pointer. Is it correct to assume that sptr (or seek pointer) is where the forwarder left off reading the file?

splunk cmd btprobe -d /opt/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/messages

Using logging configuration at /opt/splunkforwarder/6.0.3-204106/etc/log-cmdline.cfg.
key=0xf4e82f9f021c429d scrc=0xc6e25d94afc02135 sptr=871 fcrc=0x452905a167cf4509 flen=0 mdtm=1404740503 wrtm=1404740504

Re: TailingProcessor File Status without port 8089?

vcarbona — Tue, 08 Jul 2014 17:50:46 GMT

Actually "ignored file (crc conflict, needs crcSalt)" does appear, it just has a different error. I run it as "index=_internal component=TailingProcessor ERROR salt". It shows up as an ERROR alert level. The "File did not match whitelist ..." appears when I set TailingProcessor to DEBUG level.