Hello @verbal_666  on a Splunk distributed or cluster, did you apply on HF, SHC or IDX side(s) then? In our case we receive json logs from HF. @isoutamo what do you think? 🙂 Thanks for your help! ... View more

Very interesting 👍👍👍 Thanks a lot 👏👏 ... View more

If for any reason you arent able to index locally using selectiveIndexing (e.g. small local disk) but can forward to your indexers then I have found that setting the Deployment Server up to be able to search against the search peers also fixes the UI and allows management of the agents without local indexing. 🌟 Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing ... View more

It's not that I'm bothered by what you did as such. It's just that I don't think that promoting unsupported actions is healthy. And I think that if someone finds this thread they should have the full picture. So while I fully sympathize and understand why you did what you did, the reader deserves a full explanation (and the title of this thread is misleading because the upgrade from 8.2 to 9.1 did _not_ cause any problems on its own). You were trying to salvage a situation you put yourself into. What you shouid have done after upgrade from 9.1 to 9.4 was to either resolve the problem with kvstore (which I admit could have been problematic and non obvious) or - if you weren't able to fix the kvstore - restore from backup. "Downgrading" was the worst thing you could have done because there are things which change between releases (otherwise there would be no need for upgrade) which will not work with earlier releases. At this point you "only" ended up with not working DS gui but you might have ended up with broken indexes, or kvstore if it was an upgrade introducing changes to internal data format. So while in your particular situation this worked, it is not a general solution to the problem of broken upgrade and it should not be promoted as such. Especially if others have encountered similar problems after properly performed upgrade. Your "solution" which is only a walkaround for a problem you created yourself, is not helping others. I'm glad that it worked for you. It might be an interesting piece of information but with the proper context. 1. Your problem was not caused by the upgrade from 8 to 9.1 (as the topic says) 2. Your problem was caused (or at least related to) by an unsupported downgrade. That's it. Again - I'm glad it worked for you but it's not something that should be marked as solution to "broken DS gui after upgrade from 8.2 to 9.1". I hope I'm more clear this time. ... View more

Bah...... that's strange... i tried first to delete completely the Deployer "var", totally reinstall the 9.4.5 SH-Cluster, and the problem has gone 🙄  It seems it was a first cluster notification that was stuck somewhere. I also deleted all the "var" from all SHC and relaunched the SHC bootstrap. Nothing happened. Really do not know 😦 SPLUNK has some very strange behavious sometimes 🤔 ... View more

@verbal_666 there is a built in way directly in the XML without using CSS to hide those buttons from panels <option name="link.inspectSearch.visible">false</option> I use a token in that and set the token from a config file so I can turn on or off simply.  See the docs here. https://help.splunk.com/en/splunk-enterprise/create-dashboards-and-reports/simple-xml-dashboards/10.0/simple-xml-reference/simple-xml-reference#search-inspect-refresh-and-export-options-0   ... View more

Interesting 🙂 The docs for 9.4 also say that it's installed with least priv user. Worth testing and raising a case. ... View more

Upgrade Path -Upgrade from 8.1.2 → 9.0.x -Upgrade from 9.0.x → 9.2.10 9.0.x are no longer present on Splunk repository. It's possible to upgrade, -Upgrade from 8.x.x → 9.1.x [if there's a SH Cluster, follow how to upgrade KVSTORE/MONGODB] -Upgrade from 9.1.x → 9.4.x -Upgrade from 9.4.x → 10.x.x ... View more

And what is you stats wants a _time group? index=* source=/my.log | bin span=5m _time | stats count by _time[,source] 00:00 5 00:05 (no records, skipped) 00:10 10 00:15 20 The result will be, 00:00 5 00:10 10 00:15 20 The only way in this case is to use a timechart 👍 index=* source=/my.log | timechart span=5m count by source 00:00 5 00:05 0 00:10 10 00:15 20 ... View more

As usual - "it depends". During normal indexing a single pipeline engages 4-6CPU. So if you have a host which does nothing but ingestion processing (a HF), you can relatively harmlessly raise your number of pipelines and the performance scales quite well (maybe not straight linearliy but not much worse). But on an indexer you have to remember about two things: 1) You're still limited by the fact that you have to write all that to disk at the end of the pipeline (so the performance improvement will be significantly less than linear). 2) Typically indexers mostly do searching after all. So tying CPUs to ingest processing leaves you with much less left resources for searching. That might lead to problems with long running/delayed/skipped searches. So on a modern reasonably sized box, with a typical use case indeed 1 or 2 parallel ingestion pipelines seem the optimal settings. With a slightly atypical architecture (for example a separate HF layer which does the heavy lifting and indexers only receive the parsed data and write it to disks), you could consider raising the parameter more. ... View more

Final version... obviously inside a script or an interactive menu with parameters should work fine 👍   curl -skL -u 'usr:pwd' 'https://SHC_NODE:8089/servicesNS/-/-/saved/searches' --get -d 'output_mode=json' -d 'count=0' | jq -r ' .entry[] | select(.acl.app == "MYAPP" and .acl.owner == "MYUSER") | .name + " : " + .acl.app + " : " + .author + " : " + .acl.owner + " : " + .acl.sharing + " : " + (.content.disabled|tostring) '   Alternative,   curl -skL -u 'usr:pwd' 'https://SHC_NODE:8089/servicesNS/-/-/saved/searches' --get -d 'output_mode=json' -d 'count=0' | jq -r ' .entry[] | select(.acl.app == "MYAPP" and .acl.owner == "MYUSER") | [.name,.acl.app,.author,.acl.owner,.acl.sharing,.content.disabled] | @csv '   Thanks all 👍 ... View more

Thanks PeteAve! I'll try that and see what happens.....  ... View more

It's not the only factor in captain election. So just because you have raft enabled doesn't mean that your election will work properly. ... View more

https://ideas.splunk.com/ideas/PLECID-I-670 ... View more

Great, thanks 👏👏👏 I took my way, doing so,   |eval earliest_epoch="$time.earliest$",latest_epoch="$time.latest$" |eval earliest_epoch=case(isnum(earliest_epoch),earliest_epoch,earliest_epoch=="now",time(),"earliest_epoch"="",0,true(),relative_time(time(),earliest_epoch)) |eval latest_epoch=case(isnum(latest_epoch),latest_epoch,latest_epoch=="now",time(),true(),relative_time(time(),latest_epoch))     ... View more

True. Sometimes users don't have permissions to run their own crons and the system-wide crontab is fixed. That can be problematic here. Anyway. The (ugly) walkaround to the issue with spawning such stuff from within Splunk itself would be to simply create multiple inputs. If you want to spawn 2-minute long jobs every minute, you can just create two (or better yet - three so that you're sure there's no voverlap) separate inputs. One is running */3. Another one is 1,4,7,10..., and another one is 2,5,8,11... Ugly, but should work. ... View more

Very strange, since also PickleRick says it's the normal behaviour of a DS not deleting custom addons 🤔 ... View more

Clear. So, an event with _time field with "+", in practice, represents a complete _time extraction with all "date_*" underfields inside 👍 Thanks 👏👏 ... View more

I think it's a right analysis 👍 Maybe, just for some tests 😊i'll try playing with "max_fd" in limits to see how System works. Just to stress the System 😂 as said in first post, it's only a test to understand better UF. I just saw how dangerous, sometimes, is to introduce the "..." or "*" or any other wildcard in path inputs, since UF could get crazy 🤷‍♂️like crcSalt, which could ingest x2/x3/x4/... data if not right blacklisted (think about log rotation with maybe gz/zip/bz extensions 🤷‍♂️ ). Anyway, there's something else than only fd. In a stable environment, UF should "leave free" the file (and drop its fd) after "time_before_close" (5 by default), so can process other files in queue. Another strange situation, i can't see any WARN about fd in splunkd.log, as sometimes i saw in other situations 🤔🤔🤔log explicitly said the max_fd was raised 🤔now not! Strange! Maybe this behaviour occurs different on different distros, as should be a System problem, not directly related to UF work 🤷‍♂️ ... View more

Hi. I also have a large size files on some servers, about 10Gb per day in 3 files each server, and those files during the day are very delayed to be ingested, with ACK to true. While those files delay from 1 to also 4 hours to be indexed, other files on same servers are ingested fine in realtime. So, also with UF 8.2.12, i think it's a thruput of Network Infrastructure, or maybe too many datas from those inputs 🤷‍♂️ I also have   [thruput] maxKBps = 0 [general] parallelIngestionPipelines = 2 [queue] maxSize = 100MB [queue=parsingQueue] maxSize = 10MB I don't think there are other methods, since it's a phisiological problem 🤷‍♂️ The only way, maybe, is to add more Indexers in SPLUNK Infra or ask the Applicative Teams to split those file in more servers 🤷‍♂️ ... View more

Hi this could explain that behaviour to you https://docs.splunk.com/Documentation/Splunk/9.1.2/DMC/Configurefeaturemonitoring Based on this instructions you could see what those health messages means. r. Ismo ... View more