If youre not wanting to use the MC, you can easily take the searches out of the MC, and customize them to what you are looking for. The dash boards in the MC are meant to help understand, and to an extent, manage your distributed Splunk environment. There is plenty in there about SH, but your biggest points to monitor would be CPU, RAM, and search concurrency. Adapting these prebuilt searches out of the MC would be easiest. Aside from this, you could look at the deprecated SoS App (Splunk on Splunk.) However, most of the searches used in that app were all adapted and put into the MC. ... View more

Hi xsstest, I reckon this is still the best place to read about Where do I configure my Splunk settings? http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings but if you prefer the docs page here it is http://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurationparametersandthedatapipeline cheers, MuS ... View more

Thanks for the response. This will solve our issue 🙂 ... View more

hi xsstest, warm regards, alert 1 ... View more

Hi Esix, I am not getting the results for index which i applied FrozenTimePeriodInSecs = 31540000 with dbinspect command. its resulting with blank. And also observed that, the COLDDB became empty after rolling this. How do i make sure after rolling from HOT, the data must be in COLD for 3 months? -Thanks ... View more

This changes the ownership for both user and group, to indexer, for all folders below /opt/splunk. Since Splunk was trying to start as the indexer user, this user needed permissions for read, write, and execute on all bits under the /opt/splunk folder. ... View more

If the problem is that events are expiring out of _internal or _telemetry while you still need them and you cannot extend the retention, you can create a summary index (which will be TINY) and schedule a saved search to run nightly that dumps a daily summary and you can search from that. ... View more

Thanks for the reponse. I had an app with index= abc indexing data. But I had to transfer the app to a PCI complaint index =abc_sec. I made changes to the the inputs.conf substituting index=abc with index =abc_sec. Reloaded the serverclass and then the logs are showing up in _internal. ... View more

We are testing SoftIron.com Ceph at a major Splunk shop in Texas now. We'll relay the results here. Matt.Gair@softiron.com ... View more

There is no mechanism to do this automatically. Changes to the CM's master-apps should be infrequent. You could apply the changes in the folder, via copy or sync, and then hit a rest endpoint to apply the bundle as part of your script... curl -k -u admin:changeme https://localhost:8089/services/cluster/master/control/control/apply -d skip-validation=true -X POST This seems to work, even though undocumented... See here : https://answers.splunk.com/answers/146998/is-there-a-way-to-distribute-cluster-bundle-through-rest-api.html ... View more

Hi, we already have a HEC up and running and this is the only service we want to expose. cheers, Andy ... View more

Perfect! Thanks All.....I appreciate you guys weighing in on this. Indeed, we are pushing all the _internal logs down to the indexing cluster (which all of the elements in our distributed environment can search). ... View more

I have actually. So from the CM i am resolve the ip of the Indexer and vice versa: Note i did notice the "null" problem but its not really a problem it still resolves and i looked it up its for alot of reasons.. CM: ` nslookup w.x.y.z nslookup: can't resolve '(null)' Name: w.x.y.z Address 1: w.x.y.z ip-w-x-y-z.ca-central-1.compute.internal Indexer: nslookup a.b.c.d a.b.c.d.in-addr.arpa name = ip-a-b-c-d.ca-central-1.compute.internal ` I have resorted to pointing directly via IP now because at this point i am testing out register_replication_address. I have tested things like tcpdump traffic from the CM to the indexer and I am not even seeing the CM reach out to the indexer on that IP..So confused I have a question here if you want more details: https://answers.splunk.com/answers/818038/cluster-master-register-replication-address-invali.html ... View more

Thank you for your response. Unless I am missing something, yes all four members of the shc are participating. At least based on the results of shcluster-status. Actually, I did execute the "clean" command, without options, earlier, on ALL shc members. Very scary command, I think. I had to reinitialize the cluster members afterward to get the members back. (use with caution is right). Since I have not added the shcluster members to the load balancer yet there was no impact. I still get the bundle failure unfortunately. There are some things that don't add up too like I don't see the Monitoring Console or the shclustering dashboards that should be there. I believe I'll take down the instances and step through the install and configure one more time. Thanks again. ... View more

@sahils You can monitor if the Backups have failed or succeeded by monitoring the Windows Event Viewer Log for the same. The below configuration might help you, or you can use the Splunk Windows TA available : [WinEventLog:Microsoft-Windows-Backup/Operational] disabled = 0 index = wineventlog renderXml=false start_from = oldest checkpointInterval = 5 ... View more

Are your logs in Splunk? What does the "down state" look like? A general search like index=myjvm host=node* | stats dc(host) as hostCount by host | where hostCount <24 You can do a simple time search over last 5/10/20/30m or whatever time range you want. This will give you a count of hosts during that time < your total number of hosts... Theres alot more options, but your question is very vague. ... View more

One of the reasons we want to do this is that we don't want to index all the data that we've brought in using our UFs. As a result, I don't think the CEF app can help us. We want to send some of a particular sourcetype to a syslog destination. we want to First - Index all of it AND forward all of it. Second - Index a small amount of it AND forward all of it I think this means we need some content based filtering and I currently have this setup: outputs: {code} [syslog:Send_to_syslog_dest] type = tcp server = syslog_server:10518 timestampformat = %Y-%m-%dT%H:%M:%S.%3N%z [indexAndForward] index = true selectiveIndexing = true {code} props {code} [mysourcetype] TRANSFORMS-filtering = 1-forward-all-data,6-Index-bits-of-it {code} transforms {code} [1-forward-all-data] REGEX = . DEST_KEY = _SYSLOG_ROUTING FORMAT = Send_to_syslog_dest [6-Index-bits-of-it] DEST_KEY = _INDEX_AND_FORWARD_ROUTING FORMAT = local REGEX = (?msi)\"?(some|regex|matches|for|data)\"? {code} I've heard two things which make me question whether to continue on this tack. 1. That I shouldn't be trying to do this on an indexer, I should only be trying to do this on a 'heavy forwarder' 2. That I can only do this with splunktcp type 'cooked' data - as opposed to syslog format. In some cases (we have a HF layer in place) the data is arriving with us already parsed. Having said that, this sort of seems to be working (I haven't done exhaustive testing). Can someone answer the 2 questions at the end ? Thanks ... View more

Hi, Yes, the error is: Timed out waiting for peer indexerXX. If this occurs frequently, receiveTimeout in distsearch.conf may need to be increased. Search results might be incomplete! I'm getting the timeout error for around 3 search peers every time. ... View more

Thanks, I'm not aware to install the CIM app! ... View more

It looks like you are looking for system health metrics which is the exact use case for ITSI: https://www.splunk.com/en_us/campaigns/itsi-online-sandbox.html?ac=ga_usa_brand_itsi_exact_Mar17&_kk=splunk%20itsi&gclid=CjwKEAjw8OLGBRCklJalqKHzjQ0SJACP4BHrB8jui3j8G4D0dyqGzTrwVOdXmeyXaw5zICZV2YJa7BoC-S3w_wcB As a start, you should try these apps: Splunk app for Windows Infrastructure (it has pre-requisites): https://splunkbase.splunk.com/app/1680/ Splunk app for Unix and Linux (it also has pre-requisites): https://splunkbase.splunk.com/app/273/ Much of the rest will probably come in through syslog: http://www.georgestarcher.com/splunk-success-with-syslog/ ... View more

Well there you go; that shows me! ... View more

I think so, for now at least. ... View more

Hi esix you can ignore this i have checked your answer i have posted this question with some other thread someone asked me clarification with and they asked me to post this in separate question . ... View more