About sarlacc

sarlacc · ‎07-30-2024

After later review, the rebuild of the host did not fix the problem. In fact, this was more widespread of a problem than I had indicated. I'd say 3-5% of my hosts were seeing this problem. I found the releasenotes for the latest version of the Splunk_TA_nix addon. Once I downloaded/installed this version of the addon, new records starting appearing. https://docs.splunk.com/Documentation/AddOns/released/UnixLinux/Releasenotes

sarlacc · ‎06-10-2024

I resolved this by rebuilding the operating system of the affected host. It previously had a Splunk 6.x forwarder and I uninstalled the old version before I installed the new one. I also did a rm -rf on the old forwarder path. I had thought removal of the /opt/path_where_forwarder_was_installed would be enough. Are their other files/directories that I should also include in my uninstall script?

sarlacc · ‎06-07-2024

I had made the cahnge to props.conf but I'm not seeing anything show up today (06/07/24). 06/06/24 logs were fine as I'd expect. I'd like to make sure I did the props.conf right: I put this under etc/system/local/props.conf [auditd] {is this the right text here? I thought I had read this should be the sourcetype. I got the sourcetype from inputs.conf of rlog.sh under the Splunk_TA_nis app.} TIME_FORMAT=%m/%d/%y %T:%3N {I did this to mimic what I see when I do a search from the splunk webserver and this is what shows up at the left before the actual log entry}

sarlacc · ‎06-06-2024

Hi gcusello , My indexer hosts less than 50 hosts. Environment is rather small. Format of the date is American (mm/dd/yyyy). I can also see that when I do an ausearch -i from the audit.log. The date is the format I'd expect as indicated above. In terms of the _internal index, dates are correct. Yesterday's _internal index showed 06/05/2024 and the logs themselves were the same. I'm not sure I'm answering your question, however. Let me know if you needed other information about _*indexes. I will try setting props.conf on this one particular host and reply back with the results tomorrow.

sarlacc · ‎06-05-2024

I'm running Splunk Enterprise 9.1.1. It is a relatively fresh installation (done this year). Splunk forwarders are also using version 9.1.1 of the agent. The indexer is also the deployment server. Beyond that, I only have forwarders forwarding to it. I have one Linux host (Redhat 8.9) with this problem. I've deployed Splunk_TA_nix and enabled rlog.sh to show info from /var/log/audit/audit.log. Using today as an example (06/05/2024), I don't see entries for 06/05/2024. But I do see logs from today under 05/06/2024. Example from the splunk search page: index="linux_hosts" host=bad_host (last 30 days) 05/06/2024 at left side of events audit data...........(06/05/2024 14:32:12) audit data......... As I mentioned above, I have one deployment server. All forwarders are using the same/centralized. Small environment, I'd say ~25 linux hosts (redhat 7 and 8). This is the only Redhat 8 with this problem. Tried reinstalling splunk forwarder (completely deleted /path/to/splunkforwarder) once I uninstalled it. I knowa little about using props.conf with TIME_FORMAT and have not done so. My logic is if I needed it, I'd see this on all forwarders not just the one i have with this problem. I did localectl and it shows en_US. ausearch -i (same thing rlog.sh does) shows the dates/times as I'd expect. Anything else I should look for from the OS perspective? Any suggestions on what I could do from splunk? Also, noticed that when I go to the _internal index, dates/times are consistent. When I use my normal index (linux_hosts) this is my one RH8 that has this problem. Other Redhat 8 are what I'd expect. A side note here: someone else suspected this host wasn't logging. So they did a manual import of the audit.log files. Mind you, the dates in the file were not parsed since they didn't go through rlog.sh (ausearch -i) first. Could this also be part of the problem? If so, how can I undo what was done? Thanks!

sarlacc · ‎10-15-2023

Thanks @richgalloway I'm aware that best practice is indexers are same or later than forwarders (https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjR1urKjfmBAxV3l2oFHZgqAmAQFnoECBsQAQ&url=https%3A%2F%2Fdocs.splunk.com%2FDocumentation%2FVersionCompatibility%2Fcurrent%2FMatrix%2FCompatibilitybetweenforwardersandindexers&usg=AOvVaw1VxTlqkTc48Sr-quUpc-0s&opi=89978449) While I won't leave it this way, would that mean I could leave forwarders at 6.6 while I do the upgrades on the indexers (to 9.1.1) in my environment? And then when I have time I'll upgrade the forwarders? I had in my mind that I would have to upgrade forwarders incrementally while I upgrade the indexer, but seems like that isn't the case.

sarlacc · ‎10-12-2023

I am aware of this site: https://docs.splunk.com/Documentation/Splunk/7.2.10/Forwarding/Compatibilitybetweenforwardersandindexers I have several simple Splunk implementations (all functions run on one server). My indexers are a mixture of 6.5 and 6.6. I plan on upgrading to 7.2.10 with the eventual goal of getting to the latest version. First, I'd like to understand what forwarders can communicate with indexers. The link above relates to 7.0.0 and later. I'm at 6.5/6.6 as stated earlier. Secondly, I know I need to upgrade splunk to various incremental versions before I get to 9.x. What is the recommended path to upgrading to 9.x? Since I'm 6.5 or 6.6, I believe my next is 7.2.10 (is that right?). But what is the path after that? Thanks for the help!

Posts	7
Solutions	1
Karma Given	1
Karma Received	0
Member Since	‎10-12-2023

Online Status	Offline
Date Last Visited	‎07-30-2024 09:52 PM

Only 1 forwarder showing logs in Month, Day, Year ...

Forwarder to Indexer compatibility (legacy version...

Re: Only 1 forwarder showing logs in Month, Day, Y...

Re: Only 1 forwarder showing logs in Month, Day, Y...

Re: Only 1 forwarder showing logs in Month, Day, Y...

Re: Only 1 forwarder showing logs in Month, Day, Y...

Only 1 forwarder showing logs in Month, Day, Year ...

Re: Forwarder to Indexer compatibility (legacy ver...

Forwarder to Indexer compatibility (legacy version...

Join the Conversation