Thanks... BTW, Splunk for Facebook is gone, there´s no Apple Store or Google Play either. These are common good stuff sources of data and not so silly to be forgotten ... that´s a lack of sharing of the community, marketing blindness or really no one are doing this ? ... View more

Does any1 knows where is this APP ? ... View more

RT and Ad-Hoc remains not being replicated on SHC 6.2.3 ... only proxying is supported but adds a delay on the answer to the users: http://docs.splunk.com/Documentation/Splunk/6.2.3/DistSearch/SHCarchitecture#Artifact_proxying ... View more

Still on dirty solutions, if you take one old indexer at a time, the cluster will need to create new replicas and the new indexer should be selected. After all stats turn green for the replication, put that indexer back on track and go to remove excess buckets. Splunk will keep the buckets that are searchable in the new cluster instead of turn back the buckets of the indexer that were off. I´ll also go with that suggestion of changing RF ... If you trust in your backup, reduce SF and RF to 1 on a maintenance window and run remove excess buckets. When it finishes, raise the RF and SF to your defaults. Splunk will try to distribute buckets almost the same between all indexers. That´s something that Hadoop claims to do easily but still no clean way to do with Splunk. ... View more

I have the mounted bundle scenario working fine with : 7 SHs 1 Deployer 1 Master 8 Indexers My knowledge bundle is also big in the number of apps, lookups and so on. Sometimes error also 255 appeared. Used mounted bundles since SHP doing a rsync from the master, where the NFS exports was being shared, to all indexers. With SHC it changed to a script that runs in all SHs : 1st the script checks who is the captain (splunk show shcluster-status) if the captain is the SH running the script, it´ll rsync all non splunk default app to all indexers Indexers distributedsearch.conf are the same from SHP . Take a look at my question about deployer shcluser apps sync : http://answers.splunk.com/answers/241549/how-to-prevent-deployer-from-pushing-old-content-w.html ... View more

If the Search Heads are not forwarding sos logs to the indexing cluster you should create a static distributed peer for each SH on the master. It´ll then see Search Heads logs althought they will be listed as Indexers on the topology. Another way is to forward sos logs from the Search Head just to the Master if this could not be added to the indexing cluster disk usage. ... View more

Leo, I use almost the same way you did to build your APP. Does it compatible with SHC ? I´m very much concerned about how disk consumption is growing with the artifacts replication in a SHC environment. Below is a list of dashs I´m trying to get done usin my spare time : - individual disk usage (searchable UI) and top disk usage users - alert on artifacs with an expiration time of X hours and more than X MBs - SHC artifacs replication and configuration sync status among Search Heads - per search head total disk usage historical usage curve and a "prediction" of how much disk will be needed in X months/days - alert for a threshold on free disk space X growth percentage Take these as a suggestion to be added to your app or point me out any APPs that maybe already accomplishing doing at least part of it. Cya. ... View more

scarteratwork , Can you share with us the heartbeat cluster resource ? I´m trying to update a project with current corosync/pacemaker/drbd but got stucked at the pcs heartbeat daemon. Tried to get an apache daemon and modify it to splunk start|stop|status but it´s still failing ;( ... View more

I've done almost the same with Amazon. How about Google Play ? And Apple Store ? Same way ? Their API sends logs to my server, get Splunk to ingest it, parse my own way and then Dash it? ... View more

Need to confirm ... how about those apps... they are system related (6.2.3) : learned/ sanitycheck/ splunk_datapreview/ What about if IU find these apps on the shcluster/apps at the deployer ??? If I remove them, they get removed on the Search Heads, right ? To fix this, on all search heads : a manual backup Then remove the folders from shbundle, deploy from deployer; And then return backup on each search head ? ... View more

I have other odd error , but in the other way, from the Search Head to the captain : splunkd.log:05-25-2015 21:13:18.267 -0300 WARN ConfReplicationThread - Error pushing configurations to captain=https://:8089, consecutiveErrors=1: Error in acceptPush: Non-200 status_code=400: Error parsing JSON response: String value too long and metrics.log:05-25-2015 21:18:56.132 -0300 INFO Metrics - group=captainstability, stable_follower_pct=100, stable_captain_pct=0, num_polled_captain=0, num_polled_follower=155, num_polled_candidate=0, upgrades_to_captain=0, downgrades_from_captain=0, captain_changes=0 metrics.log:05-25-2015 21:18:56.133 -0300 INFO Metrics - group=subtask_counts, name=shccaptain_artifact, to_fix_rep_factor=0, to_fix_added=0, to_fix_removed=0, to_fix_total=0, count=0 metrics.log:05-25-2015 21:18:56.133 -0300 INFO Metrics - group=subtask_seconds, name=shccaptain_service, seconds=0.000 ... View more

Raj, This is related to local search head process, right ? How about when I get the following error showing up on the search head in SHC and searching accross clustered indexers : Do I really need to change ulimits on the indexers ? Note: Latelly i had increased queue sizes on all indexers (CM Bundle) per the addition of a bunch of new forwarders and some messages showing up on heavy forwarders´ metrics.log . Indexers are running with 128Gb of memory and using less then 32Gb. ... View more

I´m having other error : Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://peer:replication_port: Network-layer error: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol Any clue on that ? ... View more

What happens with the already indexed logs after changing mode to mixed ? ... View more

If there´s nothing wrong with the data or the integrity of the bucket, can the master request an indexer to discard a bucket just by interpreeting that does not belong to that cluster ? I´m trying to follow a probable scenario where peers were migrated from one master to the other and the new master seems to make the error to ask the peers to discard buckets and that lead to a loss of data. That happened with 3 peers that were replicating information normally and the master move were needed due to hardware failure on the master and on 2 other peers. ... View more

Is there any PowerPC HW sizing guide for running Splunk Enterprise (all roles in distributed search) ?? Or even with Red Hat on PowerPC ? ... View more

So... the solution was not to inherity user role, right ? As the tsdix are created inside each index folder, when a user runs a query against data accelerated by tscollect, it will return only logs from those indexes right ? Not both. Need just to clarify if your second post was the sollution. ... View more

Stefano, I´m planning to build a Master HA scenario using Linux HA daemon and rsync of NFS between two nodes. If your two sites network topology accepts vlan extensions that will work but no need of DNS pointing to both since all indexers must be on the same master node at the same time. You can have the IP, which all peers and search heads connects, managed by the HA daemon as an alias that needs to be Up only before running splunkd on the active Master node. That should require no change at peers or search heads. ... View more

Something came up here ... can I enable splunk users to change their AD/LDAP password thru splunkweb ? ... View more

Update: using a new installed Splunk instance made the difference instead of setting it as a MAster thru clustering gui : new 6.1.3 x64 install; setting initial password, changing licensing mode, set https and port; stop new instance; making changes on server.conf; untar .tgz file with $SPLUNK_HOME/etc/master-apps/ of the old master; start new master; stop all Indexers; change master_uri on [clustering] stanza of server.conf of all Indexers; start each Indexer at a time; I guess how to create a full automated solution using linux´s Heartbeat and a bunch of scripts, for example ;( ... View more

Rob, thanks for your both questions. Splunk should release a way to keep local _* Indexs out of the indexandforward=false config. That´s good for local auditing and troubleshooting, but that´s bad on a scenario that the space on a cluster is expensive compared to a heavy Forwarder with lots of gigs free on it´s local disk. ... View more

,Rob, You´ve mentioned right what I´m confused. 1st) a Heavy Forwarder with IndexAndForward=True on outputs.conf forwards logs to an Indexer, both using the same licensing server. Would it make license count twice for each inputed log ? 2nd) a Search Head on a distributed deployment running summarizations need to forward results to the cluster. Does the forwarded summarized logs counts on the license of the indexers ? ... View more