Couldn't you theoretically deploy an app with a scripted input that would make changes to files in etc/system/local? Not saying this is the best method because if it fails you could brick the forwarder, but I would think it is theoretically possible if there are no other means possible. ... View more

Hello, Our test environment uses production LM and we never had any compatibility issue upgrading first test nodes : 6.2.3 > 6.5.2 6.5.2 > 7.1.4 7.1.4 > 7.3.4 We plan to upgrade 7.3.4 to 8.2.2, any possible issue? Thanks     ... View more

It is possible that there are other things going on that is causing this error than what is stated above.  Since I identified a unique root cause I wanted to share with all.  The last bullet below was what worked for me but the below bullets represents a summary of recommended steps to get to root cause for this. First verify that the size of the bundle being sent from SH is not greater than the bundle size limit setting on the SH (maxBundleSize distSearch.conf) or the Indexer (max_content_lengh server.conf) Then check for permissions/ownership errors on all the instanced by running “ls -lahR /opt/spunk | grep root” Then run ./splunk btool check Then check the CM bundle details and compare if the latest active bundle in the peers is same as the CM. Then run the top command to see if there are any resources using a significant percentage of CPU utilization over Splunk.  A new application could have been introduced that is preventing writes from taking place over a long period of time due to files being locked by other application.  This can be further verified by: Run the following on each indexer “sudo tcpdump <ipaddressofsourceSH>” then attempt to run your search from the SH and see if you see the commands coming over. If fails that there is an application that on in your environment that is preventing Splunk from doing what it need to do and you need to apply for an Splunk exceptions for the recently introduced application. ... View more

Multi-site search affinity should work automatically as long as you have a searchable copy of the data at the same site. So that much should be good without going to extra steps as long as the indexers are configured correctly. Adding site_search_factor = origin:1.total:2 as you have should take care of at least having one searchable copy of data at each site, so in case of network problem or indexer failure you'll still be able to search. (In those emergencies you may search across sites). So, on to your multi-site indexer cluster settings. You've set site_replication_factor = origin:2,total:3 with two sites that appear to be correctly set up with two indexers each. So, wherever the data originates (either site1 or site2) will get two copies - matching your two indexers - and the "other" site will get one more copy to bring it up to 3 total copies. In my eyes the multi-site stuff looks correct to accomplish your goals. The rest of the config I'm less an expert on but it doesn't look wrong. Does that help? ... View more

I am aware of this. I have created a separate server class to hold all of my site specific apps/conf files and one of those files (the outputs.conf file) contains the relevant indexers that pertain to the remote site. ... View more

What else could be causing this error message? Failed to contact license master: reason='WARN: path=/masterlm/usage: invalid signature on request from ip=xxx.xxx.xxx.xxx'..... Like I have stated before above, all of the pass4SymmKeys are identical to the master license server. Everytime I try to convert one of my indexers to a slave indexer it does not work ever. It makes me think something is really wrong with how Splunk does it's licensing. ... View more

How would you set this up in an index cluster? What would the outputs.conf file look like? ... View more

Thank you very much for your input. ... View more

Sorry to hear the docs haven't been helpful. Can you identify where in the configuration process you ran into problems? What were you trying to do when you saw that error message, and what were you doing right before? ... View more

For addons that involve perl scripts to filter and parse data, I think its mandatory to use heavy forwarder. But somehow, I feel its an overkill for such a small task. Basically, need to parse cassandra nodetools data and post it to splunk. If I configure splunk as forwarder (heavy forwarder) how much extra resources it will consume ? ... View more

Hey somesoni2, that worked like a charm. I thought it was something simple. Thanks again for the help. Tom ... View more

What tag are you referring to? ... View more