Splunk Cloud Platform

File being monitored by Splunk Universal Forwarder is not being entered in an index

yourknightmares
Explorer

I am installing splunk universal forwarder on an AWS elastic beanstalk environment to forward logs to our new splunk cloud application. Everything sets up correctly and I am able to find data searching the _internal index with the hostname of the instance. The problem is, no data of the file I'm monitoring is actually being forwarded, though I can tail the file and see it being updated when new logs from my web application are being added.

I know the monitor succeeds, because in the AWS logs after a deployment I can see "2021-11-05 20:06:09,416 P3428 [INFO] Added monitor of '/tmp/logs/node.log'.", and I add it with: "/opt/splunkforwarder/bin/splunk add monitor "/tmp/logs/node.log" -hostname "$splunk_logs_hostname" -sourcetype json -index node"

So if I understand this correctly, it should show up in my splunk application under the "node" index. But when I search for it nothing comes up, and if I go to settings > indexes where I created the index, there's no events or current size.

Does anyone have any ideas on how to troubleshoot this issue?

0 Karma

PickleRick
Champion

Any errors in UF's logs? Does forwarder process have access to the files? SELinux?

yourknightmares
Explorer

Looked through the logs and fixed a small issue but the problem persists, I can see in the logs that the file is being watched, as the metrics.log has several lines like: "11-09-2021 16:54:58.991 +0000 INFO Metrics - group=per_source_thruput, series="/tmp/logs/implantbase/node.log", kbps=0.008, eps=0.194, kb=0.254, ev=6, av g_age=0.333, max_age=1"

The forwarder does have access to the file I believe, and yes SELinux (AWS Elastic Beanstalk)

0 Karma

PickleRick
Champion

If SELinux is enabled it may interfere with file access even though "normal" permissions look okay at first glance. Check your auditd logs for problems with accessing the logs.

yourknightmares
Explorer

Looks like it worked, as I can see a log for adding the monitor succeeding:

"Audit:[timestamp=11-10-2021 15:59:57.839, user=admin, action=edit_monitor, info=granted object="/tmp/logs/implantbase/node. log" operation=create]" (From audit.log)

Although I'm not sure why it's an action of "edit_monitor" instead of "add", but it has an operation of "create" so seems intended?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!