Why my _audit index search are not returning any results?

[franklinc]

Hi everyone,

I have a Splunk Enterprise standalone instance. It is running on Ubuntu server 14.04.6 LTS.

I recently upgraded from 6.5 to 7.2 and from there to 8.1.0.

There are a few custom apps installed on this Splunk instance.

Like a month ago, I realized that the searches to the _audit index weren't returning any result. 

/opt/splunk/var/log/splunk/audit.log file is receiving data, and permissions are ok.

It seems like the data is not being monitored and injected in the _audit index. I have already checked and compared backup files to find the missing link here, but no luck, and I can't find any significant error in the splunkd.log file.

Please, any suggestions?

[franklinc]

OK, I have found the issue.

A custom-made app (implemented long before I took control of the Splunk Instance) is setting up the environment to enable the forwarding of local instance data and metrics to a SOC layer. That worked just fine before the latest upgrades (from 6.5 to 7.2 to 8.1). When I disable the forwarding, the audit data is injected back in the local _audit index, and it breaks again as soon as I enable the forwarding.

Now that forwarding is necessary, it can't be disabled, but now I'm trying to figure out how to save a copy of the audit data without impacting the rest of the saved searches and forwarded data. I know I could select the option to make a local copy of the forwarded data. Still, I don't know what the consequences could be since it has been working for a while without the option "store a local copy of forwarded events" enabled. 

Any suggestions?

[richgalloway]

Verify your role has access to the _audit index.

Make sure the searches are specifying the index name explicitly rather than depending on _audit to be on the list of default indexes (which can change).

Verify the audit.log file is being monitored.  Use splunk list monitor to check.

[franklinc]

Verify your role has access to the _audit index.

> Yes, the role has access.

Make sure the searches are specifying the index name explicitly rather than depending on _audit to be on the list of default indexes (which can change).

>Yes, search specify the _audit index

Verify the audit.log file is being monitored.  Use splunk list monitor to check

> After running 'splunk list monitor' it shows the ...splunk/audit.log is being monitored.

The last event indexed in _audit was from 16 days ago.

I did try deleting the _audit index data and generating a new one, and it didn't work.