I'm running Splunk Enterprise 9.1.1. It is a relatively fresh installation (done this year). Splunk forwarders are also using version 9.1.1 of the agent. The indexer is also the deployment server....
See more...
I'm running Splunk Enterprise 9.1.1. It is a relatively fresh installation (done this year). Splunk forwarders are also using version 9.1.1 of the agent. The indexer is also the deployment server. Beyond that, I only have forwarders forwarding to it. I have one Linux host (Redhat 8.9) with this problem. I've deployed Splunk_TA_nix and enabled rlog.sh to show info from /var/log/audit/audit.log. Using today as an example (06/05/2024), I don't see entries for 06/05/2024. But I do see logs from today under 05/06/2024. Example from the splunk search page: index="linux_hosts" host=bad_host (last 30 days) 05/06/2024 at left side of events audit data...........(06/05/2024 14:32:12) audit data......... As I mentioned above, I have one deployment server. All forwarders are using the same/centralized. Small environment, I'd say ~25 linux hosts (redhat 7 and 8). This is the only Redhat 8 with this problem. Tried reinstalling splunk forwarder (completely deleted /path/to/splunkforwarder) once I uninstalled it. I knowa little about using props.conf with TIME_FORMAT and have not done so. My logic is if I needed it, I'd see this on all forwarders not just the one i have with this problem. I did localectl and it shows en_US. ausearch -i (same thing rlog.sh does) shows the dates/times as I'd expect. Anything else I should look for from the OS perspective? Any suggestions on what I could do from splunk? Also, noticed that when I go to the _internal index, dates/times are consistent. When I use my normal index (linux_hosts) this is my one RH8 that has this problem. Other Redhat 8 are what I'd expect. A side note here: someone else suspected this host wasn't logging. So they did a manual import of the audit.log files. Mind you, the dates in the file were not parsed since they didn't go through rlog.sh (ausearch -i) first. Could this also be part of the problem? If so, how can I undo what was done? Thanks!