File being monitored by Splunk Universal Forwarder is not being entered in an index

[yourknightmares]

I am installing splunk universal forwarder on an AWS elastic beanstalk environment to forward logs to our new splunk cloud application. Everything sets up correctly and I am able to find data searching the _internal index with the hostname of the instance. The problem is, no data of the file I'm monitoring is actually being forwarded, though I can tail the file and see it being updated when new logs from my web application are being added.

I know the monitor succeeds, because in the AWS logs after a deployment I can see "2021-11-05 20:06:09,416 P3428 [INFO] Added monitor of '/tmp/logs/node.log'.", and I add it with: "/opt/splunkforwarder/bin/splunk add monitor "/tmp/logs/node.log" -hostname "$splunk_logs_hostname" -sourcetype json -index node"

So if I understand this correctly, it should show up in my splunk application under the "node" index. But when I search for it nothing comes up, and if I go to settings > indexes where I created the index, there's no events or current size.

Does anyone have any ideas on how to troubleshoot this issue?

[PickleRick]

Any errors in UF's logs? Does forwarder process have access to the files? SELinux?

[yourknightmares]

Looked through the logs and fixed a small issue but the problem persists, I can see in the logs that the file is being watched, as the metrics.log has several lines like: "11-09-2021 16:54:58.991 +0000 INFO Metrics - group=per_source_thruput, series="/tmp/logs/implantbase/node.log", kbps=0.008, eps=0.194, kb=0.254, ev=6, av g_age=0.333, max_age=1"

The forwarder does have access to the file I believe, and yes SELinux (AWS Elastic Beanstalk)

[PickleRick]

If SELinux is enabled it may interfere with file access even though "normal" permissions look okay at first glance. Check your auditd logs for problems with accessing the logs.

[yourknightmares]

Looks like it worked, as I can see a log for adding the monitor succeeding:

"Audit:[timestamp=11-10-2021 15:59:57.839, user=admin, action=edit_monitor, info=granted object="/tmp/logs/implantbase/node. log" operation=create]" (From audit.log)

Although I'm not sure why it's an action of "edit_monitor" instead of "add", but it has an operation of "create" so seems intended?