About transtrophe

transtrophe · ‎04-13-2015

I found the setting. Had to adjust the value for time window in the app from -1 to 0 (could Have chosen a different non-zero value but taking it wide open with 0 works fine for purposes at this point).

transtrophe · ‎04-13-2015

I have recently set up Splunk App for Stream. When I log in as the admin account, I can get all results from the included dashboards, but when I log in with an account assigned to the user role, all dashboard generated searches produce no results. What have I neglected to configure that would result in this outcome?

transtrophe · ‎04-12-2015

Problem resolved - problem being why I didn't get replication for the main index. I had set-up Splunk App for Stream to use the main index, but had the streams configured only for stats versus enabled for events. Essentially, no data was populating into the main index. So unlike our beloved Lucas "Luke" Jackson in Cool Hand Luke for whom "sometimes nothing is a real cool hand", for Splunk index replication, nothing could be further from the truth - lol!

transtrophe · ‎04-12-2015

Thanks esix_splunk - doing your recommended config now.

transtrophe · ‎04-12-2015

Have started to try using FB as you suggested. Installed it on a stand-alone sh (FB v 2.0.3 and TA v 2.0.1). Not clear from the documentation or from within the FB UI or from the sh where it is installed (probing about in different directories and hunting for it) where and how to configure monitored_indexes.csv. If you have some guidance on that that would be awesome.

transtrophe · ‎04-12-2015

That's what I would also expect, but main does not show up in the Indexes tab of Index Clustering when viewed on the cluster master. As per the recommendation of esix_splunk, I have (or rather should say, am trying - lol) to set up Fire Brigade to look at the state of the indexes. I haven't gotten Fire Brigade completely (aka correctly) set up - yet - (I don't know where or how to configure the monitored_indexes.csv file that FB seems to require), but in looking at what is picked up in FB from REST, specifically looking at the Troubleshooting view>Index Configuration Summary, there are 10 indexes listed including main and sos. As previously mentioned, sos is correctly replicating but main is not. In this troubleshooting panel main is listed as having 2 repFactors: auto and 0. sos is indicated as having only auto. In carefully reading the indexes.conf spec, it doesn't seem like there are all that many attributes to configure replication so really not sure how to get this resolved at this point. It looks to me like its basically a "set repFactor = auto" in an index stanza and its good to go. I have done this for main but I'm not yet "good to go", alas and alack.

transtrophe · ‎04-12-2015

I am setting up Fire Brigade v 2.0.3 to monitor my splunk deployment (using index clustering with RF = 5 and SF = 3). The documentation for Fire Brigade provided a brief discussion for a few options in terms of deployment, but I am a little unclear still as to the recommended deployment when monitoring an indexer cluster. It seems like my options are as follows: Deploy Fire Brigade and the TA on the cluster master including making the master a search-head. Same as 1 including distributing the TA to all the index cluster peers doing a cluster-bundle apply. Deploy Fire Brigade and the TA on and across the search-head cluster. Deploy Fire Brigade and the TA on a stand-alone search-head Same as 4 including distributing the TA to all the index cluster peers doing a cluster-bundle apply. I am also not really clear on configuring the monitored_indexes.csv. Firstly, I don't find anything so far in the Fire Brigade UI for configuring this csv. Secondly, looking on the stand-alone sh where I currently deployed FB and its TA doing a 'find /opt/splunk -name monitored_indexes*' as the root account returned no file. Same situation when looking for this file on the index cluster master (I uploaded the TA to the master in case it is recommended to apply the TA across the cluster).

transtrophe · ‎04-11-2015

Thanks. I'm actually really just interested in making sure that main is replicating. Any suggestions on that?

transtrophe · ‎04-11-2015

I am also confused over repFactor and how it controls replication. I have set repFactor = auto in ALL of the indexes I previously identified, yet only _audit, _internal, and sos are getting replicated. Another aspect of replication that is confusing is what is stated in the documentation for the Admin Manual > indexes.conf spec about the repFactor attribute: repFactor = |auto * Only relevant if this instance is a clustering slave (but see note about "auto" below). * See server.conf spec for details on clustering configuration. * Value of 0 turns off replication for this index. * If set to "auto", slave will use whatever value the master is configured with * Highest legal value is 4294967295 The setting of repFactor as declared here states that the value is picked up from the master. In my current situation, looking at the indexes on the master using btool shows that the repFactor = 0 for ALL indexes, including _audit, _internal, and sos. Yet as mentioned when running a show cluster-status, I see that there is replication for those 3 indexes. Like Mick Jagger's famous declaration about Satan, what's confusing me is just the nature of this game - oh yeah! - lol

transtrophe · ‎04-11-2015

Thanks. What is not adding up for me is that I have the replication_factor set to 5 and the search_factor set to 3. This is how it is set in the /opt/splunk/etc/master-apps/_cluster/local/server.conf in the [clustering] stanza on the cluster master as well as all of the 8 cluster peers. Also, if I execute show cluster-status --verbose on the cluster master, I get the following result: _audit Number of non-site aware buckets=0 Number of buckets=867 Size=23722392 Searchable YES Replicated copies tracker 867/867 867/867 867/867 867/867 867/867 Searchable copies tracker 867/867 867/867 867/867 _internal Number of non-site aware buckets=0 Number of buckets=1050 Size=1064524035 Searchable YES Replicated copies tracker 1050/1050 1050/1050 1050/1050 1050/1050 1050/1050 Searchable copies tracker 1050/1050 1050/1050 1050/1050 sos Number of non-site aware buckets=0 Number of buckets=753 Size=43275408 Searchable YES Replicated copies tracker 753/753 753/753 753/753 753/753 753/753 Searchable copies tracker 753/753 753/753 753/753 Yet, as previously noted, from SOS>Indexing>Index Replication>Cluster Master View, this app search returns that rep factor = 3 and search factor = 1. Just don't get what SOS is looking at to get these values.

transtrophe · ‎04-11-2015

/opt/splunk/etc/slave-apps/_cluster/local/indexes.conf [main] /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf maxDataSize = auto_high_volume /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf maxHotBuckets = 10 /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf repFactor = auto All the other attributes are using the /opt/splunk/etc/system/default/indexes.conf default values. I have also set repFactor = auto in the stanzas of _introspection, _audit, sos, _blocksignature, _internal, _thefishbucket, history, sos_summary_daily, splunklogger, and summary.

transtrophe · ‎04-11-2015

I am trying to verify that certain indexes are replicating across my index cluster. My splunk installation is a distributed deployment with 8 peers composing the index cluster with their master node, 3 search-head cluster members with their deployer, 1 stand-alone s-h running Splunk App for Stream, and 4 forwarders with their deployer server. I have SOS deployed on the shc members and the indexers. I have Splunk App for Stream deployed with App for Stream running on the stand-alone s-h and TA_Stream running on the 4 forwarders with the setuid.sh executed to handle the permissions for the streamfwd binary (I run splunk under the splunk account). All of the stream configurations that I set using the App for Stream UI on the stand-alone search-head are pointing to the MAIN index. The [main] stanza on all my indexers are configured with repFactor = auto (I am going to provide the [main] stanza in a comment after this first posting). After a moderate amount of troubleshooting this implementation which some of you have likely been following in earlier questions, I finally have a relatively stable platform though I think there are still some issues, the basis of this question about verifying index replication for certain indexes being one of those items I still don't fully understand or see as achieving completely normal operations. When I run SOS on my shc members and look at the Indexing > Index Replication > Cluster Master View the search query returns indicating that my replication factor = 3, the search factor = 1, Cluster initialization state: Index replication not enabled, and Cluster indexing state: Index replication not enabled. So how can I confirm that I am actually getting complete replication across my index cluster?

transtrophe · ‎04-11-2015

OK. All problems now resolved with this issue. Thanks esix_splunk and vshcherbakov_splunk.

transtrophe · ‎04-11-2015

Think I just found the answer to my last question but want to confirm this. Looks like the index is set in the Stream Configuration panel of the App for Stream Dashboard running on the designated search-head. Making the configuration change in this panel - I would suspect that gets pushed down to the forwarders over that "heart-beat config" channel? Going now to confirm on this to see in terms of forwarder conf files where those settings get populated.

transtrophe · ‎04-11-2015

I did set the streamfwd binary privileges using the setuid.sh script. IN the spirit of splunk newbieness, where do I set the configuration for forwarding to a specific index serviced in the index cluster? I am looking through the documentation for inputs.conf and outputs.conf at least up till now my lil' ol' brain has not found any documentation in the specs that call this configuration out - which config file, which stanza, which attribute.

transtrophe · ‎04-10-2015

2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found DataDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/data 2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found UIDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/ui 2015-04-10 00:55:51 FATAL 139845104486208 stream.main - No found in

transtrophe · ‎04-10-2015

I do get the following FATAL error on the forwarders, so that can't be too good: 2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found DataDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/data 2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found UIDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/ui 2015-04-10 00:55:51 FATAL 139845104486208 stream.main - No found in

transtrophe · ‎04-10-2015

Splunk is 6.2.2. App for Stream is 6.2.1. Executing a search on the "dedicated" app4stream search-head (versus the search-heads in my shc) for this search: index=_internal host="ip-172-31-21-117" sourcetype="splunk_app_stream-2" Produces this search result: 4/11/15 2:43:23.853 AM 2015-04-11 02:43:23,853 DEBUG stream:252 - DefaultDir /opt/splunk/etc/apps/framework/default/streams, LocalDir /opt/splunk/etc/apps/framework/local/streams host = ip-172-31-21-117 source = /opt/splunk/var/log/splunk/splunk_app_stream.log sourcetype = splunk_app_stream-2 Configuration of streamfwdlog on the 4 forwarders running TA_stream: Stream forwarder log file configuration log4cplus.appender.streamfwdlog=log4cplus::RollingFileAppender log4cplus.appender.streamfwdlog.layout=log4cplus::PatternLayout The name and location for the log file log4cplus.appender.streamfwdlog.File=/opt/splunk/var/log/splunk/streamfwd.log Right now I have only configured TCP and UDP streams for stats only. I don't get any results returned when running the Stream Stats dashboard, e.g. Don't get any results when using any of the dashboards, for that matter.

transtrophe · ‎04-09-2015

In the process of trying to get Splunk App for Stream up and running in a distributed deployment using an index cluster with 8 indexers set with repFactor = 5 and a single Stream App search-head. I have TA-stream installed on 4 forwarders. I have enabled Data Inputs > Wire Data on all 4 of these forwarders including setting the Splunk App for Stream location to the single Stream App search head (not using SSL so this is set to port 8000 using http://). The inputs.conf file is configured on all 4 forwarders with the following settings in the [streamfwd] and [streamfwd://streamfwd] stanzas: /opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf [streamfwd] /opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864 /opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf disabled = true /opt/splunk/etc/system/local/inputs.conf host = ip-172-31-21-115 /opt/splunk/etc/system/default/inputs.conf index = default /opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf source = stream /opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf [streamfwd://streamfwd] /opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864 /opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf disabled = 0 /opt/splunk/etc/system/local/inputs.conf host = ip-172-31-21-115 /opt/splunk/etc/system/default/inputs.conf index = default /opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf source = stream /opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf splunk_stream_app_location = http://ip-172-31-30-208:8000/en-us/custom/splunk_app_stream/ When I try to do a search of source=stream* from the search-head I get no results. What am I missing in getting this set-up? I do see the index is pointing to default - not sure if I should be pointing to a different index. When I look at indexes on the index cluster master DMC I don't see any events in the main index.

transtrophe · ‎04-09-2015

In my situation, the default for maxresultrows is used in the limits merge (set to 50000) and the warning indicates 100000 are being requested. Don't mind resetting the values across my search-head cluster if that is advised. Just trying to evaluate the impact of this warning on searching. Also, if I should change it, should I go with 100000 or some larger value like 150000.

transtrophe · ‎04-08-2015

I would like to recommend that Splunk consider building in this functionality to the Enterprise core capabilities - a kind of PKI mechanism, really. There are several situations I can envision that would call for this kind of large-scale deployment of new key material across complex indexer and search-head deployments. Say a large deployment experienced a hack. In that situation, best-practices from a GRC perspective would call for re-issuing key material. LIkewise, several industries such as the nuclear power generating infrastructures (for NEI 08-09 / 10 CFR 73.54) and the bulk electric system infrastructures (for NERC CIP) have password and secure information establishment requirements from a regulatory perspective that would call for this kind of key management capabilities. This could be done as you suggest with some set of home-grown approaches, but Splunk provided mechanisms would ensure a far greater degree of deployment and maintenance consistency.

transtrophe · ‎04-08-2015

Thanks esix_splunk - this fixed the issue. I was reviewing the following documentation from the Best Practices: Forward Search Head internal data to Indexer Layer: Configure the search head as a forwarder. Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers). You must also turn off indexing on the search head, so that the search head does not both retain the data locally as well as forward it to the search peers. Here is an example outputs.conf file: Turn off indexing on the search head [indexAndForward] index = false [tcpout] defaultGroup = my_search_peers forwardedindex.filter.disable = true indexAndForward = false [tcpout:my_search_peers] server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997 autoLB = true This example assumes that each indexer's receiving port is set to 9997. For details on configuring outputs.conf, read "Configure forwarders with outputs.conf" in the Forwarding Data manual. I'd respectfully suggest that this documentation entry specifically highlight making sure to use a different port for this implementation of forwarding the internal data from the index layer's replication port. It wasn't really very clear to me, perhaps also to others as I saw there were a few posts around this same topic, so adding that additional configuration information should put this issue to bed for a long, long time.

transtrophe · ‎04-08-2015

I tried doing the following in an effort to troubleshoot and resolve an issue I am having with an initial distributed deployment of Splunk: On the cluster master I created a server.conf in the cluster bundle (/opt/splunk/etc/master-apps/_cluster/local/) with only the clustering stanza and a single entry for the updated pass4SymmKey: [clustering] pass4SymmKey = myNewKeyPass I did the same thing on the search-head cluster deployer's bundle. I then applied both bundles. On the indexers the "pushed" server.conf was successfully deployed to /opt/splunk/etc/slave-apps/_cluster/local and on the search-heads to opt/splunk/etc/apps/StandaloneConfigs/default/. [Note: I am using the sub-dir titled StandAloneConfigs to push configuration files from the shc deployer to the shc members.] I then did a rolling restart of both clusters. The index cluster failed to establish so I checked the values of the server.conf on the indexers using btool and saw that the value for pass4SymmKey in each indexers merged server.conf never got hashed (it was the plain-text passphrase). Also, the values were sourced from /opt/splunk/etc/slave-apps/_cluster/local/server.conf files. On the search-head cluster members, the "pushed" server.conf that deployed to opt/splunk/etc/apps/StandaloneConfigs/default/ did NOT get used in the merge but the initial /opt/splunk/etc/system/local/server.conf got used. So if anyone has a correct/best-practices method for doing this kind of large-scale pass4SymmKey update to accomplish the combined objectives of 1.) ensuring coordination within an index cluster and 2.) ensuring coordination between a search=head cluster and an indexer cluster for forwarding the search-head cluster _internal events to the designated indexer cluster layer, that would be awesome.

transtrophe · ‎04-08-2015

I have been trying to troubleshoot my deployment which is not currently working properly (receiving on-going Search peer ip-172-31-18-186 has the following message: Too many streaming errors to target=172.31.25.77:9998. Not rolling hot buckets on further errors to this target messages) by tailing the splunkd.logs on both an indexer and a search-head cluster. On a search-head cluster member all I am getting in the splunkd.log are these records: 04-09-2015 00:37:58.100 +0000 INFO TcpOutputProc - Connected to idx=172.31.25.77:9998 04-09-2015 00:38:28.143 +0000 INFO TcpOutputProc - Connected to idx=172.31.20.120:9998 04-09-2015 00:38:58.195 +0000 INFO TcpOutputProc - Connected to idx=172.31.26.200:9998 04-09-2015 00:39:28.213 +0000 INFO TcpOutputProc - Connected to idx=172.31.22.253:9998 04-09-2015 00:39:58.232 +0000 INFO TcpOutputProc - Connected to idx=172.31.25.228:9998 04-09-2015 00:40:28.303 +0000 INFO TcpOutputProc - Connected to idx=172.31.20.173:9998 04-09-2015 00:40:58.322 +0000 INFO TcpOutputProc - Connected to idx=172.31.25.228:9998 04-09-2015 00:41:28.387 +0000 INFO TcpOutputProc - Connected to idx=172.31.18.186:9998 04-09-2015 00:41:58.461 +0000 INFO TcpOutputProc - Connected to idx=172.31.29.149:9998 [Note: these are the IPs of my indexers] I get varied records on an indexer in its splunkd.log - here is a snippet: 04-09-2015 00:45:58.407 +0000 INFO CMRepJob - job=CMReplicationErrorJob bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF failingGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 srcGuid=37D7692E-5D49-432E-9A6F-89C0C68FACEF tgtGuid=78E65DE9-B82B-4F0A-A383-D0BC1189F9A3 succeeded 04-09-2015 00:47:08.223 +0000 INFO CMReplicationRegistry - Starting replication: bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF src=37D7692E-5D49-432E-9A6F-89C0C68FACEF target=5BB335C9-340F-42CD-A5C6-C8269429D10A 04-09-2015 00:47:08.223 +0000 INFO BucketReplicator - event=asyncReplicateBucket bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF to guid=5BB335C9-340F-42CD-A5C6-C8269429D10A host=172.31.22.253 s2sport=9998 04-09-2015 00:47:08.223 +0000 INFO BucketReplicator - bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF earliest=1428291587 latest=1428291724 type=2 04-09-2015 00:47:08.223 +0000 INFO BucketReplicator - Created asyncReplication task to replicate bucket _audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF to guid=5BB335C9-340F-42CD-A5C6-C8269429D10A host=172.31.22.253 s2sport=9998 bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF 04-09-2015 00:47:08.223 +0000 INFO BucketReplicator - event=startBucketReplication bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF 04-09-2015 00:47:08.223 +0000 INFO BucketReplicator - Starting replication of bucket=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF to 172.31.22.253:9998; 04-09-2015 00:47:08.223 +0000 INFO BucketReplicator - Replicating warm bucket=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF node=guid=5BB335C9-340F-42CD-A5C6-C8269429D10A host=172.31.22.253 s2sport=9998 bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF 04-09-2015 00:47:08.224 +0000 INFO BucketReplicator - event=finishBucketReplication bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF [et=1428291587 lt=1428291724 type=2] 04-09-2015 00:47:08.224 +0000 INFO BucketReplicator - event=localReplicationFinished type=warm bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF 04-09-2015 00:47:58.902 +0000 INFO CMReplicationRegistry - Starting replication: bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF src=37D7692E-5D49-432E-9A6F-89C0C68FACEF target=5BB335C9-340F-42CD-A5C6-C8269429D10A 04-09-2015 00:47:58.902 +0000 INFO BucketReplicator - event=asyncReplicateBucket bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF to guid=5BB335C9-340F-42CD-A5C6-C8269429D10A host=172.31.22.253 s2sport=9998 04-09-2015 00:47:58.902 +0000 INFO BucketReplicator - bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF earliest=1428289848 latest=1428291580 type=2 04-09-2015 00:47:58.902 +0000 INFO BucketReplicator - Created asyncReplication task to replicate bucket _audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF to guid=5BB335C9-340F-42CD-A5C6-C8269429D10A host=172.31.22.253 s2sport=9998 bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF 04-09-2015 00:47:58.902 +0000 INFO BucketReplicator - event=startBucketReplication bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF 04-09-2015 00:47:58.902 +0000 INFO BucketReplicator - Starting replication of bucket=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF to 172.31.22.253:9998; 04-09-2015 00:47:58.903 +0000 INFO BucketReplicator - Replicating warm bucket=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF node=guid=5BB335C9-340F-42CD-A5C6-C8269429D10A host=172.31.22.253 s2sport=9998 bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF 04-09-2015 00:47:58.903 +0000 INFO BucketReplicator - event=finishBucketReplication bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF [et=1428289848 lt=1428291580 type=2] 04-09-2015 00:47:58.903 +0000 INFO BucketReplicator - event=localReplicationFinished type=warm bid=_audit~86~37D7692E-5D49-432E-9A6F-89C0C68FACEF 04-09-2015 00:48:08.228 +0000 WARN BucketReplicator - Replication connection to ip=172.31.22.253:9998 timed out 04-09-2015 00:48:08.228 +0000 WARN BucketReplicator - Connection failed 04-09-2015 00:48:08.228 +0000 INFO BucketReplicator - Discarding replication data as QueueRef=guid=5BB335C9-340F-42CD-A5C6-C8269429D10A host=172.31.22.253 s2sport=9998 bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF is deleted 04-09-2015 00:48:08.228 +0000 WARN BucketReplicator - Failed to replicate warm bucket bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF to guid=5BB335C9-340F-42CD-A5C6-C8269429D10A host=172.31.22.253 s2sport=9998. Connection failed 04-09-2015 00:48:08.228 +0000 INFO CMReplicationRegistry - Finished replication: bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF src=37D7692E-5D49-432E-9A6F-89C0C68FACEF target=5BB335C9-340F-42CD-A5C6-C8269429D10A 04-09-2015 00:48:08.228 +0000 INFO CMSlave - bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF src=37D7692E-5D49-432E-9A6F-89C0C68FACEF tgt=5BB335C9-340F-42CD-A5C6-C8269429D10A failing=5BB335C9-340F-42CD-A5C6-C8269429D10A queued replication error job 04-09-2015 00:48:08.230 +0000 INFO CMRepJob - job=CMReplicationErrorJob bid=_audit~87~37D7692E-5D49-432E-9A6F-89C0C68FACEF failingGuid=5BB335C9-340F-42CD-A5C6-C8269429D10A srcGuid=37D7692E-5D49-432E-9A6F-89C0C68FACEF tgtGuid=5BB335C9-340F-42CD-A5C6-C8269429D10A succeeded

transtrophe · ‎04-08-2015

Changed the pass4SymmKey in each server.conf on all 8 indexers and the search-head members. I even did this by leaving the index cluster master with the previous value for pass4SymmKey to make sure that the indexers all got changed correctly. Leaving the master with the previous pass4SymmKey value resulted in the cluster not forming. After verifying that the cluster didn't form, I then made the change of the cluster master's pass4SymmKey to the values on each of the 8 indexers and the cluster then formed. This was the only way I could think of to verify that the pass4SymmKey was fully synchronized across the index cluster. Really not sure where to take this now. I can certainly provide any splunkd.log records (or any other log) that anyone might suggest (either on on indexer and/or on a search-head member.

Posts	79
Solutions	4
Karma Given	0
Karma Received	24
Member Since	‎03-06-2015

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

What needs to be configured in Splunk App for Stre...

What is the correct installation and configuration...

How to verify an index is replicating?

What is correct way to set-up Stream Forwarders wi...

What is the impact on searching when getting a war...

Is there a method to change the pass4SymmKey value...

In a properly configured and operating splunk set-...

Why does number of Indexers on Machines count in D...

Should the main index be replicated in an indexer ...

What are recommended/effective techniques for trou...

Re: What needs to be configured in Splunk App for ...

What needs to be configured in Splunk App for Stre...

Re: How to verify an index is replicating?

Re: What is the correct installation and configura...

Re: How to verify an index is replicating?

Re: How to verify an index is replicating?

What is the correct installation and configuration...

Re: How to verify an index is replicating?

Re: How to verify an index is replicating?

Re: How to verify an index is replicating?

Re: How to verify an index is replicating?

How to verify an index is replicating?

Re: What is correct way to set-up Stream Forwarder...

Re: What is correct way to set-up Stream Forwarder...

Re: What is correct way to set-up Stream Forwarder...

Re: What is correct way to set-up Stream Forwarder...

Re: What is correct way to set-up Stream Forwarder...

Re: What is correct way to set-up Stream Forwarder...

What is correct way to set-up Stream Forwarders wi...

What is the impact on searching when getting a war...

Re: Is there a method to change the pass4SymmKey v...

Re: In a properly configured and operating splunk ...

Is there a method to change the pass4SymmKey value...

In a properly configured and operating splunk set-...

Re: What are recommended/effective techniques for ...