I am trying to look at cpu and mem statistics on my indexers and search heads, but the index only ever goes back 15 days, almost to the hour, but I need to look a a specific date almost a month ago.
Any ideas on why this could be and how can get around it?
When dealing with historical data in Splunk, there are a few factors to consider.
i) Check if your Splunk deployment has custom retention policies configured. You can adjust these policies to retain data for a longer period of time.
I think that you should read at
https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setaretirementandarchivingpolicy
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf
Check the retention settings on the _introspection index. By default, it's 14 days. Change the frozenTimePeriodInSecs setting in indexes.conf to retain data longer.
Okay, I am assuming that will change the retention for future events, but how can I get the old logs back?
If you have backups of the old data, restore them to the thawed folder. See https://docs.splunk.com/Documentation/Splunk/9.2.0/Indexer/Restorearchiveddata for details.
If you don't have backups then the old data is gone forever.