In addition, you can add these to the end of a dashboard URL to hide specific items. Example: en-US/app/search/chrometest?earliest=0&latest=&hideChrome=true&hideEdit=true Will make it nice and sparse for a TV display. ... View more

Hi, Data in Splunk is indeed immutable. This doesn't mean that with a little work, that the data can't be cleaned up and made available for search without the PII data in there. 0) You already nailed part of the solution: SEDCMD to keep the problem from getting worse for new data indexed. 1) Create a search that finds all the events with the PII data in it that needs to be cleansed. Run that on the ./splunk CLI and dump the results to a file. 2) Use your favorite text mangling tools (sed, awk, perl, LISP 🙂 ) to sanitize the data on disk. 3) Run the original search again, this time with '... | delete' at the end, to mark the existing entries as unavailable for being included in search results. 4) Use ./splunk add oneshot to re-index the sanitized data file. 5) enjoy a frosty beverage of your choice for a job well done. The original search & |delete may take quite a while, depending on how many events need to be found & extracted. The oneshot will slurp it back in as fast as the forwarder/index can absorb it. Note that oneshot WILL count against the license, so plan accordingly. ... View more

What was the solution? ... View more

Ah, I did figure out how to fix this! In the app that owns the saved search, look for the $app/metadata/local.meta file. In that file, look for the name of the saved search. There is a field called "owner" that has the old AD or LDAP userid in it. Replace that old userid with "nobody" (no quotes). This is a userid that Splunk sets aside to be able to run saved searches when there is no real userid that owns the search anymore. Restart the search head after the change, or run $SPLUNK/bin/splunk btool fix-dangling and wait a few seconds and reload the dashboard. ... View more

After upgrading from 5.x to 6.1.2 yesterday, I now see this problem too. Did you find an answer to this problem? ... View more

Edited my original post, and added a screenshot of both the new search view where the Chart Title is missing, and the flashtimeline view where Chart Title is present as an option. ... View more

It's not nice to remove features that provide operational granularity for making changes. Please put it back! ... View more

An interesting caveat to this -- if you alter the settings above to shorten the search outage when a CM restarts, it also impacts how aggressive the CM is when taking an indexing peer out of rotation. When it rebalances the cluster (making buckets searchable, replicating buckets to a new peer), it takes advantage of these settings. CPU usage goes way up during the rebalance. ... View more

No. This is a server-wide config, and can't be set per app. ... View more

That did it. In the form view, that search needed to be wrapped in <[!CDATA[ ]]> or have the ">" and "<" escaped. Thanks! ... View more

I may have been close, but this doesn't seem to do it...at least not yet. I Added "by myAPI" to eventstats. The final chart output doesn't show the percent_of_all_APIs (which is named poorly, and should be percent_per_API) result. I'm still missing something. ... View more

Hi, I ran into this as well. For some reason, the way I had uploaded the app through the GUI and/or wiped & re-installed it via the command line didn't work. To get this fixed (and to be able to find the setup page), I had to do the following. 1) stop splunk 2) remove the existing Splunk for *nix app, TA and addon 3) untar the splunk for *nix app into the etc/apps dir (either in $SPLUNK or in the SHP space) 4) restart splunk 5) kick off the splunk for *nix app, which brought me to the setup screen 6) kick off the splunk_ta app, which brought me to the setup screen And I now have a working splunk for *nix app. ... View more

That's a splunk docs typo, in my opinion. "1 active user per core (idealy) two". If a user isn't actively searching, then they shouldn't count! 🙂 ... View more

nrjsh1988, I disagree with MuS here. limits.conf: [thruput] maxKBps= will do what you want, if you set it on the indexer itself. If you have multiple indexers, each indexer gets a fraction of the total. Be aware there are caveats to this solution, which I will leave an an exercise to the reader. ... View more

What was the solution? ... View more

You've hit on a touchy problem with Splunk: figuring out how busy the infrastructure is at any point in time. There are two things to look at. 1) How many users are currently using Splunk. This is interesting, but only goes so far. Am I "currently using Splunk" if I have a static dashboard on my screen that has finished loading 10 minutes ago, and I'm either staring at it, or have my head turned talking to someone else? martin_mueller's search in his comment is spot on, and will help you answer this question. 1 hour may be too long of a time frame, as I have found 1m or 5m is more useful for determining how busy Splunk is. 2) How many searches are currently being run. This is a little harder, because searches come and go, sometimes fairly quickly. A couple of ways to see this info. First, concurrent searches by user. Who's exercising Splunk the most? index=_internal source=*metrics.log group="search_concurrency" NOT "system total" | timechart span=1m sum(active_hist_searches) as concurrent_searches by user Interesting patterns emerge per person/group and time of day. Second, is this ad-hoc or scheduled? Too many concurrent scheduled searches can really bring Splunk to its knees. A lot of scheduled searches may be okay, if they are very short duration (like populating summary indexes or report acceleration). `set_sos_index` sourcetype=ps | multikv | `get_splunk_process_type` | search type="searches" | rex field=ARGS "_--user=(?<search_user>.*?)_--" | rex field=ARGS "--id=(?<sid>.*?)_--" | rex field=sid "remote_(?<search_head>[^_]*?)_" | eval is_remote=if(like(sid,"%remote%"),"remote","local") | eval is_scheduled=if(like(sid,"%scheduler_%"),"scheduled","ad-hoc") | eval is_realtime=if(like(sid,"%rt_%"),"real-time","historical") | eval is_subsearch=if(like(sid,"%subsearch_%"),"subsearch","generic") | eval search_type=is_remote.", ".is_scheduled.", ".is_realtime | timechart span=1m dc(sid) AS "Search count" by is_scheduled Props go out to hexx (SoS guru) for these, and hopefully they (or something like it) will show up in SoS in the near future. ... View more

Already done, and fixed. Turns out that the install itself appears to be correct in all instances we could audit, but permissions were just not getting picked up. To fix: stopped both SHs, removed all 3 apps (SA-nix, Splunk_TA, Splunk for nix app), and untarred a fresh copy. Restarted Splunk SH, went through SA-nix first time setup, then Splunk for NIX first time setup, and permissions were correct, and I could see the past 3 months of data in the os index. Thanks to BrianO for verifying the original setup looked right, and the suggestion for the "nuke from orbit" and re-install via tar. ... View more