One more document to read https://lantern.splunk.com/Splunk_Platform/Product_Tips/Upgrades_and_Migration/Upgrading_the_Splunk_platform Be sure that you take needed backups (also SPLUNK_DB) to get possibility to do rollback if needed. Full backup of all DB files could be tricky. Also take backups for kvstores. @tej57 already posted link to correct update order which you must follow. If/when you are updating over several versions you must start servers after each step to get done needed conversion tasks. Also before upgrade check that your apps etc. are compliant for next version. Update those before platform update. Fix all issues before you go forward! As Splunk has migrated its documentation from docs to help, there are some situations that you cannot find needed documentation from those. Then wayback machine is your friend, where you could try to found missing docs!   ... View more

What is the real work issue which you are trying to solve? And is this one shot or continuously needed answer to it? ... View more

As metrics reports only metrics on top X, this didn't get real amount as there probably are many periods when another nodes have more traffic. ... View more

Thanks, If you will continue with this and can solve, please inform us what the solution was! ChatGPT proposes the next algorithms for this case Algorithm in MLTKWhen to UseStrengthsWeaknesses StateSpaceForecast(algo=StateSpaceForecast) Small seasonal datasets with trend Captures seasonality & trend Not designed for sparse series with many zeros ARIMA (algo=ARIMA) Strong seasonality, autocorrelation Handles short time series Needs continuous values (zeros can reduce model quality) LLP (Local Linear Projection) Simple, quick Light-weight Limited for complex patterns DenseNNRegressor / LSTM Complex, nonlinear series Can learn patterns over multiple users at once Requires MLTK deep learning toolkit, more tuning One-Class SVM / Isolation Forest (for anomaly detection instead of forecasting) Detecting abnormal future access Robust to sparse data Not a forecasting method per se   Given your case — weekly time series, short history (26 points for training), seasonality (weekly), and sparsity — the StateSpaceForecast algorithm is usually the safest starting point in MLTK for this type of forecasting.   ... View more

There was same kind of discussion on slack side some times ago. Maybe this can leads you into correct way? https://splunkcommunity.slack.com/archives/CD9CL5WJ3/p1727111432487429 ... View more

As those are not generally available in splunkbase and those are made by splunk, you could try to get those by asking from splunk support. ... View more

There are some important differences e.g. USER are different. When running from UF it’s splunkfwd and other cases its root. Have you try to run it commandline as user splunkfwd not with root or sudo root? Also at least libraries have different order to use. It’s easier to check different when you sort those env variables before looking. Also it’s easy to cat those into same sort | uniq -c to check are there something which are missing or different in separate runs. ... View more

You should report this issue on https://splunkcommunity.slack.com/archives/C4HDPSF60 ... View more

Ok. If possible you should participate Data Administration class or something similar. It contains basic stuff how to ingest data into Splunk  https://education.splunk.com/Saba/Web_spf/NA10P2PRD105/app/me/learningeventdetail;spf-url=common%2Fledetail%2Fcours000000000003499%3FfromAutoSuggest%3Dtrue https://www.splunk.com/en_us/pdfs/training/splunk-enterprise-data-administration-course-description.pdf Depending on your needs you could also parse that data also on indexers instead of use separate HFs for that. It's hard to say which option is better for you as we don't know enough well your needs. Also you should think again if you need clustered indexers instead of use separate. That will make your environment more robust than what you have now. Of course it needs more disk space etc. but I'm quite sure that it's worth of those additional costs. You will save those when you have 1st issue/crash with your individual indexer... I think that you should take some courses or learn same information by yourself from net or take some local Splunk company/consultant/architect to help you to define and setup your environment. I'm quite sure that you will save more money that way than starting from scratch w/o enough knowledge and later setup it again. ... View more

Why you want to use HF between indexers and UF? The best practices is send events directly from UF to IDX. If you can do that way just add another outputs.conf to all UF:s and then define used targets in inputs.conf. That's much easier and robust way than using HF between UFs and IDXs. If you must use HF (e.g. security policy) then you should have at least two HW making routing between UFs and IDXs. ... View more

Can you show sample of raw syslog events before those have sent to SC4S? Can you also check the whole event from FortiAnalyzer (local logs) and logs from other FortiGates sent to FA and check if there was some other field where those times are set correctly? It's so long time ago when I have looked those that I cannot remember what all fields there are 😞 If I recall right there are many/some information several times in one event with little bit different format. Maybe there was another field which contains also TZ information? ... View more

I think the in most cases there are no real issues with different versions as long as there is no too big cap with versions. And as/if you are using only HEC to sending events from HF->IDX then it shouldn't be issue. But if you are using also s2s then there could be some challenges. And at least MC gives you a warnings if HFs are added there and those are newer than MC itself. If you will need help from Splunk Support then this could be issue as that combination is not officially supported.  Anyhow you should update at least 9.2.x or 9.3. asap. Here is link to support times for Splunk core https://www.splunk.com/en_us/legal/splunk-software-support-policy.html#core ... View more

With earlier versions the rule was that indexers must be the newest versions. HFs and UFs which were connected can be lower. And same for UF vs HF. This has been changed with 9.x (maybe x was 3 or 2, I cannot remember exact version). After your indexers and cm are that level HFs and UFs can be newer than indexers and CM and other splunk servers.  So in your situation when you have 8.x.x all HFs and UFs should be max same version than those servers are.  Anyhow those versions are already out of support, so you should upgrade those as soon as possible to supported version. Probably 9.4.4 is currently best option. Don’t go to 10.0.0 as it’s too new for production use! ... View more

If you cannot find those events from any indexes, are you defined lastChangeIndex in your indexes.conf? If not then it's time to add it. lastChanceIndex = <index name> * An index that receives events that are otherwise not associated with a valid index. * If you do not specify a valid index with this setting, such events are dropped entirely. * Routes the following kinds of events to the specified index: * events with a non-existent index specified at an input layer, like an invalid "index" setting in inputs.conf * events with a non-existent index computed at index-time, like an invalid _MetaData:Index value set from a "FORMAT" setting in transforms.conf * You must set 'lastChanceIndex' to an existing, enabled index. Splunk software cannot start otherwise. * If set to "default", then the default index specified by the 'defaultDatabase' setting is used as a last chance index. * Default: empty string ... View more

For some reason I'm not surprised this with Forti products 😞 ... View more

I think that you cannot do this with props and transforms. The reason for that is the order how those different processors are done in ingestion phase. See e.g https://www.aplura.com/assets/pdf/props_conf_order.pdf Based on that diagram, ANNOTATE_PUNCT is after Splunk has apply other props and transforms stuff and events cannot go backwards on ingestion pipeline. ... View more

This is incorrect information. You cannot update directly from 8.1.x to 9.4.x. You must do it as @livehybrid told. This rule is also defined on splunk docs. Also you must start your splunk service after each step or otherwise it didn't do needed conversions between old to new version! ... View more

Here is old post where you can link to scripts which could get old versions to you https://community.splunk.com/t5/Installation/Need-Splunk-Universal-Forwarder-7-x/m-p/695726/highlight/true#M14117 https://github.com/ryanadler/downloadSplunk ... View more

Additions to earlier answers. Splunk has https://voc.splunk.com/ Voice of the Customer program, where are some new beta versions which you could test and give feedback to Splunk. Unfortunately currently there is no OIDC related stuff available. Then there is Customer Advisory Programs where customers can give direct feedback to Splunk's PMs. There are some roadmap presentations in every session (kept quarterly or something like that). In those you have possibility to give direct hopes to PMs what and why you are needing this and that. ... View more

As @PickleRick said this is timezone issue. Are all those logs wrongly timed or only some? I mean that if your SC4S is in one TZ and you are collecting syslogs from several different locations.  Also are your splunk servers and SC4S in same TZ? There are at least two common syslog protocols: RFC3164 aka BSD syslog  RFC5424  The newer (RFC5424) contains TZ information on every event, but old one has only date and time, but not TZ information.  Check what those sources are used and if possible use RFC5424 version. If you cannot use that then you must add TZ information to those on SC4S or Splunk HEC side. Here is one instructions for it https://splunk.my.site.com/customer/s/article/Splunk-Connect-for-Syslog-Events ... View more

If you are installing this TA for monitoring DS itself then @richgalloway answer is correct. But if you are installing it for deploy it to some UF's, then you need to something else. ... View more

Here is more about selecting correct user in windows environment https://help.splunk.com/en/splunk-enterprise/get-started/install-and-upgrade/9.4/install-splunk-enterprise-on-windows/choose-the-windows-user-splunk-enterprise-should-run-as ... View more

Can you check on UF side what it has written into local log files? There should be information why it can’t send those into your indexers. ... View more

When you have cluster, then correct method is add nodes to it and after data has spread into those new nodes, then remove old nodes. Here is details how to do it https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platform-Linux-but/m-p/538062 ... View more

Hi as you have an individual servers, you could use this method https://community.splunk.com/t5/Installation/How-to-migrate-indexes-to-new-indexer-instance/m-p/528064/highlight/true When you have several indexers you should consider to migrate into clustered environment. You should read this https://docs.splunk.com/Documentation/SVA/current/Architectures/TopologyGuidance r. Ismo ... View more