TailingProcessor File Status without port 8089?

[vcarbona]

We would like to have forwarders run as root in order to overcome file permissions. However, we also will be security hardening it as much as possible. One of these measures is to stop port 8089 on the forwarder. I assume this will not give us the ability to read the REST endpoint https://hostname:8089/services/admin/inputstatus/TailingProcessor%3AFileStatus. Are there any other ways to gather this data without the REST endpoint being available?

[vcarbona]

Actually "ignored file (crc conflict, needs crcSalt)" does appear, it just has a different error. I run it as "index=_internal component=TailingProcessor ERROR salt". It shows up as an ERROR alert level. The "File did not match whitelist ..." appears when I set TailingProcessor to DEBUG level.

[vcarbona]

Thanks! This actually helped us to identify some problem areas. But it appears conditions like "ignored file (crc conflict, needs crcSalt)" or "File did not match whitelist ..." do not show up.

[martin_mueller · Accepted solution]

You can call the endpoint directly from the CLI:

./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

[vcarbona]

I noticed the btprobe command shows some interesting data about the file status. It appears it is able to retrieve the modtime and seek pointer. Is it correct to assume that sptr (or seek pointer) is where the forwarder left off reading the file?

splunk cmd btprobe -d /opt/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/messages

Using logging configuration at /opt/splunkforwarder/6.0.3-204106/etc/log-cmdline.cfg.
key=0xf4e82f9f021c429d scrc=0xc6e25d94afc02135 sptr=871 fcrc=0x452905a167cf4509 flen=0 mdtm=1404740503 wrtm=1404740504

[martin_mueller]

So you've made port 8089 unavailable even from localhost? Then it might indeed be tough to call the REST API.

[vcarbona]

That would have been cool, but I get this:

ps -ef|grep splunk

splunk 5604 1 53 11:39 ? 00:00:13 splunkd -p 8089 restart
splunk 5605 5604 0 11:39 ? 00:00:00 [splunkd pid=5604] splunkd -p 8089 restart [process-runner]

./splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus
QUERYING: 'https://127.0.0.1:8089/services/admin/inputstatus/TailingProcessor:FileStatus'
This command [GET /services/admin/inputstatus/TailingProcessor:FileStatus] needs splunkd to be up, and splunkd is down.

[somesoni2]

How about using search?
index=_internal component=TailingProcessor