It should be possible. While there are probably a number of scenarios which could cause this to happen, there's the really simple case of the indexer which held the primary no longer being available. Since the original indexer is no longer available, cluster replication would ensure your replication factor is met. All of the buckets, however, would be replicated buckets and thus have the rb_ prefix. Once you consider that simple case, you can imagine, there are a number of variations where you didn't lose the whole indexer, but something happened to the original bucket. I believe it is even possible where there are too many copies of a bucket and the cluster decides to remove the original copy. It may even be possible just from rebalancing the cluster. ... View more

I can verify I have seen other customers push certs via APPS and then just update the path in a .conf file in that app. You can even reference the certs with ./ I know a place that has been doing this since the 5.x days ... View more

The init script doesn't bother me that much; the bigger pain is Splunk_TA_nix needing updated; specifically the service list that is practically useless. Red Hat 7 has only been out 3.5 years, you would think they would want to be able to easily list the service on a server considering its importance for security and operations. We do have support and I've opened support cases, I've spoken to people to try and get this on the road map, but they don't seem to really care. ... View more

@MuS, You are completely correct that in this simple case that would work. I added the expected to show if I thought the filter should match the event or not; in the real data set I wouldn't have that. I'll try to clarify the question; I really appreciate you trying. ... View more

In this case, @peter7431's answer is probably the best answer. There are times when you aren't using stats to get the multi-value field so I wanted to follow-up with why it didn't work and two ways to make it work. Why didn't it work? mvcombine takes fields from different events and combines them. For example: | gentimes start=-1 | eval foo="cat;bear;monkey;horse;dog" | fields foo | eval foo=split(foo,";") | mvexpand foo Then if we try mvcombine and use nomv, you can see the effect of the delim argument: [...] | mvcombine delim="DelimsROCK" foo | nomv foo Other than the stats command, how can we make it work? Option 1: Utilize mvexpand and nomv The most obvious solution can be seen in my above example where I use mvexpand foo and nomv foo: source=/var/log/secure "pam_unix(sshd:session): session opened" | stats values(user) as "user" by host | mvexpand user | mvcombine delim="," user | nomv user Note: There are two additions mvexpand and nomv Option 2 (preferred): Use the mvjoin source=/var/log/secure "pam_unix(sshd:session): session opened" | stats values(user) as "user" by host | eval user=mvjoin(user,",") ... View more

If you have bash installed, its easy to check certificate expiration dates: ports="8089 8443 8191 8065 5100 5555 9997"; SPLUNK_HOME="/opt/splunk"; openssl="${SPLUNK_HOME}/bin/openssl"; timeout=15 ; for port in ${ports} ; do expiration=$(echo | timeout ${timeout} ${openssl} s_client -connect 127.0.0.1:${port} 2>/dev/null | ${openssl} x509 -noout -dates 2>/dev/null | grep notAfter); echo ${port} ${expiration:-non-SSL, timeout, or error} ; done There may be a couple of things you need to modify ports="8089 8443 8191 8065 5100 5555 9997"; ports needs to be a space separated list of ports to check; I pulled this from a combination of our search heads and indexers. SPLUNK_HOME="/opt/splunk"; This one should be completely foreign to any Splunk administrator 🙂 but update the path if need be timeout=15 I use the timeout command so openssl won't hang if a port isn't open or doesn't use SSL. This gives the openssl's s_client 15 seconds to return the cert. That was plenty of time on our systems, but you may need to increase it. It shouldn't be hard to write a very similar thing for Windows since Splunk bundles openssl; I just don't have a windows box handy to use to attempt to write one. If there's a request and no one else volunteers I'll spin up a VM tonight. ... View more

If you get an error about command not found, Splunk bundles openssl IMHO that best way to use the bundled version is $SPLUNK_HOME/bin/splunk cmd openssl ... View more

Is $SPLUNK_HOME/etc/auth/openssl really correct? Just FYI that doesn't exist on our systems; I would use /opt/splunk/bin/splunk cmd openssl The executable is typically /opt/splunk/bin/openssl ; on some systems that works, but others I have to use /opt/splunk/bin/splunk cmd openssl due to needing to have the correct path to load libraries. ... View more

Is using a self-signed certificate enough? Under recommendations, option 2 states: "Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate." Why the need to upgrade Splunk to 6.3 or above for option 2? Why isn't the second part sufficient? Especially since under impact it explicitly states "[...] does NOT affect Splunk deployments where: 1) Your configuration uses certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA)" My assumption is using self-signed certificates would be enough; my concern is there's a place where we aren't replacing the certificate (e.g. mongo or on 8065). ... View more

Of particular note is that this affected all searches. As far as I know no changes where made to our DMC setup; we noticed that all searches quit working on our cluster master with the above mentioned error message. ... View more

Yes, it should be possible to deploy Windows DNS Analytical and Diagnostic Logs on universal forwarders. Without more information, its hard to know why it isn't working. Enable Inputs According to the README, you need to enable the inputs. Looking at default/inputs.conf , there is one scripted input. Did you create a local/inputs.conf ? If not, can you please create with the content below and restart Splunk? [script://.\bin\get_dns_analytics.path] disabled = 0 Troubleshooting If you manually run the script $SPLUNK_HOME/etc/apps/TA-windnsanalytical/bin/get_dns_analytics.path does it work? Looking at that script, I have two concerns: It is manually invoking powershell.exe as C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe It seems likely to me that you will need to update that path; you might try just Powershell (that's what some apps we've downloaded use) It references the powershell scripts as $SPLUNK_HOME\etc\apps\TA-WindowsDNSAnalytical\bin\get_dns_analytics.ps1 ; is $SPLUNK_HOME defined and does that match your app name? ... View more

Could you please expand on why this is/was necessary? I realize this answer is quite old by now, but its always helpful to understand WHY something is recommended. ... View more

The posted answers attempt to workaround correct this issue by making the libraries appear where Splunk is looking for them; IMHO the "correct" solution would be to have Splunk look in the "correct" location. Note: I am not knocking the other solutions; they've been posted for months while I ignored the problem and something good enough today is better then something perfect tomorrow. I'm just trying to explain why I am posting a new answer. Edit: Please see the comments below about rpath; that would be the "right" way to fix it. I'll try to update my answer utilizing rpath. Solution Run the following command (copy and paste into a terminal -- Applications -> Utilities -> Terminal) splunk_home="/Applications/Splunk" ; for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do f=$(echo $file | rev | cut -d / -f 1 | rev) ; sudo install_name_tool -change $file $splunk_home/lib/$f $lib ; done ; done Caveats Assumes Splunk is installed in /Applications/Splunk (if not update splunk_home="..." ) Assumes you use an sh style shell (you probably do; if in doubt copy the script below into a file) Only tested on two versions of Splunk Enterprise; they were different enough to require slight modifications so I think I've made it generic enough. If not post the version (and product if not Splunk Enterprise) and I'll try to update the answer What does it do? If you run otool -L <object> , it will print the shared libraries (e.g. otool -L /Applications/Splunk/bin/splunkd ) For Splunk 6.3.1 this gives us: /Applications/Splunk/bin/splunkd: /Users/eserv/wrangler-2.0/build-home/ember/lib/libmongoc-1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libbson-1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.5.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libxml2.2.dylib (compatibility version 12.0.0, current version 12.2.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libxslt.1.dylib (compatibility version 3.0.0, current version 3.28.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libxmlsec1.1.dylib (compatibility version 4.0.0, current version 4.20.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libxmlsec1-openssl.1.dylib (compatibility version 4.0.0, current version 4.20.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libarchive.13.dylib (compatibility version 15.0.0, current version 15.2.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libbz2.1.dylib (compatibility version 2.0.0, current version 2.3.0) /Users/eserv/wrangler-2.0/build-home/ember/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8) /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) The last two libraries look good; the previous entries are looking for things in /Users/eserv/wrangler-2.0/build-home/ember/lib/ which probably doesn't exist Note: the exact path is version specific; 6.2.3 was trying to use /Users/eserv/wrangler/build-home/6.2.3/lib/ For each library, you then need to run install_name_tool to correct the path: install_name_tool -change /Users/eserv/wrangler/build-home/6.2.3/lib/libmongoc-1.0.0.dylib /Applications/Splunk/lib/libmongoc-1.0.0.dylib /Applications/Splunk/bin/splunkd After running similar commands 12 times, you'd try to start Splunk only to discover /Applications/Splunk/lib/libmongoc-1.0.0.dylib references /Users/eserv/wrangler/build-home/6.2.3/lib/libbson-1.0.0.dylib . To put an end to the rabbit hole, I just looped over all libraries in /Applications/Splunk/lib At this point, my laziness kicked in and I wrote a script: #!/bin/sh splunk_home="/Applications/Splunk" for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do f=$(echo $file | rev | cut -d / -f 1 | rev) sudo install_name_tool -change $file $splunk_home/lib/$f $lib done done Save it to a file, make it executable, and execute it. Note: The command at the top of this post is just a minimized version of this script ... View more

To install RPM's you will need to be root (short version; see below). This isn't specific to Splunk, but a general rule for every RPM based distribution I've used. If you really need to install Splunk without having root access, you can use the tared distribution; you won't be able to install the init script to start it on boot without root, however. Long Version: I don't have a RedHat 6.4 system handy, but I have a 6.6 system and if I do ls -al /var/lib/rpm/ I notice that: The /var/lib/rpm directory and all files and sub-directories are owned by root:root (technically uid: 0 gid: 0) The directory has 0744 permissions (rwxr--r--), so only the owner can write to it Thus by default the permissions would prevent a user other than root from installing RPM's. You could work around this by changing permissions, making another uid 0 account etc, but I would advice against it. If you think about it more generally, ideally all software on RedHat would be installed / managed via RPM's. The only user then who we are confident has write access to all the files that might need updating is the root user (okay the UID 0 user if you want to rename root). Aside: Running Splunk as a user other than root is highly recommended. The RPM distributed by Splunk creates a Splunk user/group that owns the files in $SPLUNK_HOME so its very easy to do. Since its not obvious you would need to be root to install an RPM, I'm assuming you aren't an experience Unix admin (although I could be wrong; we all have our days), so just be aware that if you want Splunk to read system logs (/var/log/*) and you're not running it as root, you may need to adjust the file permissions (e.g. /var/log/secure is owned by root:root with 0600 permissions by default). ... View more

Interestingly this worked (at least for the simplified test case; I'll try to test it in production tonight); I'm confused why. According to the docs(1) it defaults to false. Looking at $SPLUNK_HOME/etc/default/fields it defaults to False. I thought maybe it was because the field wasn't listed anywhere, so the "global" setting wasn't getting applied to the field and it was either undefined or in code it defaults to true. To test that I tried just having [third] in a fields.conf That breaks the search again, even though the btool output indicates that same output (see below where I outline three scenarios. Hopefully it just works and I don't need the why, but if some one could help my understand I would greatly appreciate it. http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/fieldsconf Test 1 fields.conf [third] INDEXED_VALUE = false btool output [third] INDEXED = False INDEXED_VALUE = false TOKENIZER = Works: Yes Test 2 fields.conf [third] INDEXED_VALUE = False btool output [third] INDEXED = False INDEXED_VALUE = False TOKENIZER = Works: Yes Test 3 fields.conf [third] btool output [third] INDEXED = False INDEXED_VALUE = False TOKENIZER = Works: No ... View more

While this solution works, it may not be ideal. Every Splunk environment is different; I help run a centralized Splunk service for over 100 local I.T. shops within our organization. Outside of the servers that Splunk runs on, I have no access to those servers so I have to send instructions to one of 100+ administrators. Thus we try to do EVERYTHING with the deployment server and have no local configuration files (we go so far as pushing an app with a deploymentclient.conf so once they've checked in they can just have the client name string in $SPLUNK_HOME/etc/system/local/deploymentclient.conf This let's us move our deployment server if we ever need. So in our environment we would create a new app and then manually edit the serverclass.conf and use whitelist to control which hosts it went to. The key is our deployment client names are fwd__ so its easy for us to limit who gets what (even when we have host name collisions between different groups). ... View more

I believe the query which shows the different counts rules out _time being the issue, but just to clarify... For the tests on my laptop, I ran the search for all time. When I originally discovered this, I hard coded earliest and latest in my search (and the sub-searches for append) ... View more

@Jeremiah Did you get it working? If not, I used Puppet with Vagrant to create virtual environments over a year ago and earlier today I used Puppet to setup new Indexers so I have done it and would be glad to try and help. ... View more

I realize this question is a bit old, but I came across it so I thought I'd try to answer it in hopes of helping others. To setup search peers by only modifying files on the file system (oaky a restart of Splunk is probably also required), there are two basic steps: Configure distsearch.conf Copy the certificate from the search head to the indexers. Step 1 is how the search heads know where to send search jobs while step 2 is how the indexers know the search is authenticated -- this is critical since the search heads are the ones that enforce permissions. Step 1. Configuring distsearch.conf In distsearch.conf there are two relevant settings: <dl> <dt>servers</dt> <dd>a comma separated list of search peers</dd> <dd>servers = index1.example.com:8089,index2.example.com:8089</dd> <dt>disabled_servers</dt> <dd>a comma separated list of search peers, just like servers, except these search peers are in a disabled state</dd> <dd>disabled_servers = index4.examples.com:8089,index5.example.com</dd> </dl> In typical Splunk fashion, if you edit the search peers from the web interface they settings are stored in $SPLUNK_HOME/etc/system/local but you can put them in an app $SPLUNK_HOME/etc/apps/search_peers/local so that its easy to push them from a deployment server. Step 2 Copy the certificate from the search heads to the search peers On the search head, grab a copy of $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem and put it in $SPLUNK_HOME/etc/auth/distServerKeys/${search_head_host_name}/trusted.pem (e.g. searchhead1 go to index1 and save the file as $SPLUNK_HOME/etc/auth/distServerKeys/searchhead1/trusted.pem ) Step 3: Restart After making the change to distsearch.conf, you'll need to restart. After adding the search head public keys, I would restart although I'm not positive its actually required. Since the trusted.pem is a public key, it should be safe to distribute it via a configuration management system (e.g. Puppet, Ansibule, etc). I'm not aware of an easy way to distribute the keys via the deployment server. Since they only need to be on the search peers (typically indexers, although for your Splunk instance that runs deployment console, you'll want your search heads to be peers), the set of servers that need the keys is likely within your control; this means I can use Puppet to distribute them even though we highly depend on the Deployment Server for general configuration management when it comes to Splunk. ... View more