@VatsalJaganiThanks, precisely my thinking on the topic. I appreciate your swift response - it was waiting for me when I got back to my office. ... View more

@VatsalJagani    I'm surprised by this answer, specifically the statement there "are only two options that Splunk provides to specify when Splunk should move buckets from Warm Bucket to Cold Bucket." Perhaps you can explain a bit further? As I read the specifications, there are other methods. The original question mentions maxHotSpanSecs = positive integer * Upper bound of timespan of hot/warm buckets, in seconds. as well as maxWarmDBCount = nonnegative integer * The maximum number of warm buckets. Now if each warm bucket has a max span of 1 day, and we limit the count of warm buckets to 30, then warm buckets over 30 days old will be placed into the cold db. Of course we may run into issues, as warm buckets are generated for a variety of reasons, such as a Splunk restart (current hot becomes warm) or hot idle (again, current hot becomes warm). But it seems to me that this would be a reasonable proxy, especially if the user were willing to have, e.g., 1.5x more files than needed. If you could explain why this scheme would not work, that would help me understand.   ... View more

See my comments in other places in this discussion, which I've tested up to version 7.3.1. ... View more

My current installation, as of the date of this post, is running Splunk version 7.3.1. In this version, the file to modify is splunk/share/splunk/search_mrsparkle/exposed/build/pages/enterprise/common.js The code to modify with extra page lengths is where we see: splunk/share/splunk/search_mrsparkle/exposed/build/pages/enterprise/common.js and then as previously noted. I was able to increase pages on the statistics output by modifying splunk/share/splunk/search_mrsparkle/exposed/build/pages/enterprise/search.js to read Base.prototype.initialize.apply(this, arguments), this.children.count = new SyntheticSelectControl({ menuWidth: "narrow", className: "btn-group pull-left", items: [ { value: "10", label: _("10 Per Page").t() }, { value: "20", label: _("20 Per Page").t() }, { value: "50", label: _("50 Per Page").t() }, { value: "100", label: _("100 Per Page").t() }, { value: "200", label: _("200 Per Page").t() }, { value: "500", label: _("500 Per Page").t() } ], model: this.model.report.entry.content, modelAttribute: "display.prefs.statistics.count", but my other attempts failed.   The main difference is that display.prefs.statistics.count. It has to be adjacent to the declaration of the number of rows. ... View more

I'd like add a solution detail: In my original notes on this problem,  I stated that trying to add a peer gave me this error message: Encountered the following error while trying to save: Invalid action for this internal handler (handler: distsearch-peer, supported: list|edit|remove|_reload|new|disable|enable|doc, wanted: create). In addition, the GUI did not display the option to add a new peer when I was in the list of search peers. I had to be the "distributed peers menu" to see the "+ search peer" option on that page; and it did not work. As soon as I created a file called private.pem, even though it and its trusted.pem was not acceptable because the key length was too short, the option of "add search peer" appeared on the list of distributed peers. I am afraid that I can't recall the error message I received when I used "add search peer" with an invalid key pair. ... View more

@isoutamoHere's an update on this problem. After intensive work with vendor, I have no resolution, but I did finally find what I think is the problem in my current installation. Before my upgrade to 7.1.3, I had two files in $SPLUNK_HOME/etc/auth/distServerKeys/. One was private.pem, an RSA private key, and another was trusted.pem, a public key. I have been unable to determine from the documentation  (yet) exactly what these keys are supposed to secure, or how they are generated. In my current installation, I have only a "trusted.pem." This file is something I created, IIRC, in order to see if I could get the installation working. This trusted.pem is a full server certificate and *not* a key. It's a multi-part file with a certificate, an encrypted private key, and another certificate. I can probably fix everything is I can find the exact procedure to generate trusted.pem and private.pem from the files I have in $SPLUNK_HOME/etc/auth:  cacert.pem, ca.srl, ca.pem, and a server.pem. (There's a few others there as well, e.g., in audit directory.) I'm now looking for documentation on what does what, without much success. ... View more

@isoutamoThanks for your help. With any luck, later today I will have time to test a different version of a new trusted.pem - or try a labor-intesive re-configuration of the operating environment to see if I can persuade the my original sh to work. I am current waiting on my vendor's Splunk expert to discuss next steps with me. I will update this page (and you directly) when I have further progress. ... View more

@isoutamoThanks for the ideas. I removed the search peers from the sh and restarted the sh. I then removed trusted.pem from the peer and restarted the peer I restarted the peer. I have not managed to get the peers to work correctly. I tried several different variations, e.g., I turned off both the sh and peer simultaneously, and then started the sh and then the peer. I also replaced 'trusted.pem' with a new copy that I created via $splunk/bin/splunk createssl server-cert -d $splunk/etc/auth -n 'trusted.pem' -c <ip address> which did not help -- I thought it might fix the issue. I may have some ideas about that -- perhaps the "-c" should be replaced with a FQDN and the name of the instance, or if possible left out entirely.   ... View more

@isoutamo I'm re-sending this reply  -- I suspect that my previous reply was not properly directed to you. Thank you for your response. Here's the answers: Time is in sync on all nodes. They are synced via NTP and operate in a small cluster in the same datacenter. The splunk versions are the same on all nodes. The search head was the last to be converted. I apologize if I was unclear. If I remove a search peer and then try to re-add it, I cannot. I get an error when I try this via the GUI: Encountered the following error while trying to save: Invalid action for this internal handler (handler: distsearch-peer, supported: list|edit|remove|_reload|new|disable|enable|doc, wanted: create). I have not tried to add via the Splunk command line.   Please feel free to make suggestions or ask further questions. ... View more

Thank you for your response. Here's the answers: Time is in sync on all nodes. They are synced via NTP and operate in a small cluster in the same datacenter. The splunk versions are the same on all nodes. The search head was the last to be converted. I apologize if I was unclear. If I remove a search peer and then try to re-add it, I cannot. I get an error when I try this via the GUI: Encountered the following error while trying to save: Invalid action for this internal handler (handler: distsearch-peer, supported: list|edit|remove|_reload|new|disable|enable|doc, wanted: create). I have not tried to add via the Splunk command line.   Please feel free to make suggestions or ask further questions. ... View more

I admit you are completely correct. Unfortunately, my Splunk installation is tightly bundled with a different application, and I don't (at present) have the time to untangle the two. In fact, as much as I'd like to upgrade to 8.x, I'm don't believe that I can... if I can figure out how to to download the latest Splunk and find time for a test installation, perhaps I can try to get it licensed -- the licenses come through the vendor of the other app -- and see if I can persuade them to work together. Thanks again for the help and advice. ... View more

Thanks! You are entirely correct! I had the wrong port. My particular setup used port 9997 for the GUI, and looking around further I found the value of mgmtHostPort in my local/web.conf. Using that value, I can get the REST API to work from a console. I do get XML responses about passwords, but no passwords. I used, e.g., /servicesNS/-/-/storage/passwords/. I also tried to access /servicesNS/-/-/storage/passwords/general and /servicesNS/-/-/storage/passwords/:general:, but got a "could not find" response. At this point, however, I have to put this aside since I have the answer I was seeking by using the script I reference above. I suspect that part of the problem following the original recipe may be that I am running 6.3.2; the file passwords.conf was only created in 6.5.2 and above. Thanks again, rvany and scheng both, for your invaluable help solving this puzzle. ... View more

This tool works perfectly: https://pypi.org/project/splunksecrets/ ... View more

And now I have HTTPS running, and it makes no difference. I also added list_storage_passwords to authorize.conf -- it was not there before -- but regardless I cannot access the passwords from either search or over the net. I have a strong suspicion that it's been disabled somehow. ... View more

My management port is 9997; I'm running Splunk 6.2.1 that was installed as part of an Aspect install of Prophecy. After a great deal of additional attempts, I now find that the REST API works: | rest /services/storage gives the result author id published splunk_server title updated system https://127.0.0.1/services/storage/collections <deleted> collections 2019-12-03T15:34:19+00:00 system https://127.0.0.1/services/storage/passwords <deleted> passwords 2019-12-03T15:34:19+00:00 but asking for the passwords via "| rest /services/storage/passwords" fails, and I wonder if it 's related to lack of HTTPS access. ... View more

After putting this aside for a while -- some urgent projects -- I have tried again today, and it still does not work for me. I've tried this several times -- on the same server that has the passwords I want to decipher -- and I still can't get it to work. Password refresh does work, that is, debug/refresh gives me a results page, and I do see: Refreshing admin/passwords OK on the page that comes up. When I try "services/storage/passwords" I get the same error as before: an added "/en-US" in the path and a 404. When I try to use the rest API via search, I get no errors but no results. I find that no /services APIs work. E.g., /services/server/health_report and similar innocuous REST API attempts, both via HTTP and via "|rest", all fail. /server/info fails. /search/apps/local does work via '| rest.' I am now trying to figure out why /server and /services both fail. I wonder if it's related to a privilege issue, but I'm logging in as admin. ... View more

First and foremost, thank you for your update. I tried the REST API on the same Splunk instance I had passwords.conf installed, and I got no response at all. In addition, the URL http(s)://server:mgmt_port/services/storage/passwords on my installation immediately add an en-US and does not display anything. I will try some variations on this; I have a suspicion that I may need to run this on the licensing master. If I figure this out I'll post my comments here. ... View more

This requires Splunk version 7.2.2 and above. ... View more

When I try the pre-7.x recovery method on version 6.5.x, step 3 does work but step 4 does not: I get a response of The path '/en-US/services/storage/password' was not found. Note that the '/en-US' was added automatically by Splunk, not me. The alternate version of '/servicesNS/-/-/storage/passwords' gives a similar error. '| rest' does not work either when run on the Splunk index head. I'm trying to recover the password using a non-critical server; I'd rather not have to use the index head. ... View more

In Splunk 6.x, XML no longer works; instead, you must modify the JS. I modified the JS in the following file: share/splunk/search_mrsparkle/exposed/js/build/searchpage.js First I de-minified the JS. Searching the file, I find: this.children.count = new s({ modelAttribute: "display.prefs.events.count", model: this.model.content, items: [{ value: "10", label: _("10 Per Page").t() }, { value: "20", label: _("20 Per Page").t() }, { value: "50", label: _("50 Per Page").t() }], save: !2, elastic: !0, menuWidth: "narrow", toggleClassName: "btn-pill", popdownOptions: { attachDialogTo: "body" } I modify this to read: this.children.count = new s({ modelAttribute: "display.prefs.events.count", model: this.model.content, items: [{ value: "10", label: _("10 Per Page").t() }, { value: "20", label: _("20 Per Page").t() }, { value: "50", label: _("50 Per Page").t() }, { value: "100", label: _("100 Per Page").t() }, { value: "250", label: _("250 Per Page").t() }], save: !4, elastic: !0, menuWidth: "narrow", toggleClassName: "btn-pill", popdownOptions: { attachDialogTo: "body" } In other words, I add 100 and 250 as options, and I make 250 the standard option. I then minify the JS file. (I realize it may be possible to modify files in the views directory, e.g., share/splunk/search_mrsparkle/exposed/js/views/search/results/eventspane/controls/Master.js, in order to add the improved selections, but I have not tested it.) Once I restart splunkd and splunkweb, and of course make certain to flush the cache on the browser, the 250 options appears. I modified my $SPLUNK_HOME/etc/apps/*/local/savesearches.conf as well as the one in $SPLUNK_HOME/etc/system/local to include, at the top, [default] display.prefs.events.count = 250 which seems to have reset searches to use 250 as the default display. I modified individual saved searches as needed, and I have also gone through viewstates.conf and modified the mysterious, apparently-undocumented Count* lines to 250 instead of their default 50. However, to date, I have not been able to get my standard search page to pre-select 250 as the default; it always comes up as the standard 50 per page. At the moment, I use "?display.prefs.events.count=250" appended to the URL of the bookmarked search page to force the page into default display of 250 results per page, and wouldn't mind hearing if someone has managed to get the default search page to work without this workaround. ... View more

As of September 2017, working on a new installation, I have still not found a way to modify the current searches to use 250 lines by default. ... View more

Update: adding display.prefs.events.count = 250 to the ui-prefs.conf stanza did not work. ... View more

Excellent, thank you. FYI I had to create a copy of this in a separate app - that is, in etc/apps/prophecy-premise/local/ because my setup has that app, with specialized searches, built-in by the provider. And "of course" I flushed the browser cache. I'll continue to pursue setting the default for the number of rows per page. I'm guessing that it's possible to set it. ... View more