Hi, I hope you're at 6.6 already because Splunk introduced the join command: http://docs.splunk.com/Documentation/Splunk/6.6.1/SearchReference/Join Which does quite the same thing as a join does in SQL where you can join on a specified field. Skalli ... View more

Depends where you copied those configurations. Usually, you would copy the configurations to the deployer and let the deployer push the complete bundle to the search heads. Please read the documentation carefully. Seems like you are still at the beginning of Splunk, so you might want to read about the roles of master, deployer, deployment server and how SHC and indexer clusters work. See here for migrating from a standalone SH to a SHC. Skalli ... View more

They should disappear after some time. ... View more

What do you mean by "index master dashbaord"? Are you talking about the Distributed Management Console (DMC)? Does your master also host the DMC? (whose URL looks like this: https://URI:port/en-US/app/splunk_monitoring_console/...) There are two things you have to do (if you're also hosting the DMC there): 1. Settings > Distributed Search > Search peers > Select "Delet" next to the appropriate SH. This will delete them from the deployer (note that the "master" is the appropriate term for the indexer cluster, whereas "deployer" is the term used for a SH cluster). 2. If your DMC is also hosted there, you can simply rebuild your hosts like this: Open the DMC overview > Settings (in the DMC menu bar) > General Setup > Switch from "Distributed" to "Standalone" and clik apply. After that, click "Distributed" again and apply changes again. The old servers should be gone from the DMC then. Skalli ... View more

Yep, just wait 2-3 minutes after removing the member before you stop it. Job done. ... View more

Have you checked for duplicate RecordNumbers? Because sometimes you get a ridiculous high amount of the same message. Like this: index=*active_directory* sourcetype=*whatever* | stats count by RecordNumber, _time, host | where count > 1 Skalli ... View more

Almost correct. Your last statement actually differs if you're sending this command from the new member OR an existing member. So this: splunk add shcluster-member -current_member_uri https://<anyexistingmembername>:8089 is wrong. You can only use -current_member_uri from the new Search Head with its own URI. However, I prefer using the more logically one: splunk add shcluster-member -new_member_uri <URI>:<management_port> Sending this command from an existing SH cluster member where URI is the new host. Skalli Edit: typo ... View more

Hi, all the knowledge object settings which are done via the GUI are replicated amongst the Search Heads (see here what gets replicated by default). Except the user specific search history, you don't have to migrate anything (and: do you really need the search history on the new ones?). I would start by adding the 2 new members to the cluster, verify they replicated all settings correctly and then remove the two old SHs from the captain's CLI. Beware of the captain though. You have to check which server is the captain and if one of those old SHs is the captain, you have to specify another members as the captain (control captaincy). Skalli ... View more

Good to hear! 🙂 Glad to be of help. ... View more

I would try using BREAK_ONLY_BEFORE = RegEx Use the RegEx for your timestamp here (write it yourself), so your events should only get split up when a new timestamp occurs. Skalli ... View more

Could you filter your internal events for any errors, please? Right now, it's hard to tell why the connection got interrupted. Skalli ... View more

Did you check whether the UF also stops sending _internal logs? Please show us your splunkd.log and the metrics.log from the time the forwarder stopped sending logs. ... View more

We had the same issues, too when we were starting to integrate many different applications. The problem in enterprise environments is that you have many different applications, where a few may only be able to send syslog data, while other are only accessible via DBConnect or other vendor-specific apps from Splunkbase. A couple of big vendors already have documented some of these information in their documentation (how to get data into third-party tools). We started like this: Make a list with the available options to get data into your Splunk environment. If possible, concentrate on a few of them (syslog, directory monitoring, UF, DBConnect, scripted input, ...). You don't want to support 20 different ways in your company of how to get data into your Splunk environment. So, whenever a new application wants to get its data analyzed by Splunk, its responsible person could fill a check list which options are supported by the application (database connection, syslog stream, HTTP Event Collection, OPSEC-lea, local directory monitoring, ...). We have a couple of standard inputs we offer applications/application owners: - Syslog (and specify a port, we don't use 514 because splunk doesn't run as root) - DBConnect - vendor-specific app (like OPSEC LEA) - Universal Forwarder (deployed on the host of the application, for example useful with Domain Controllers) However, sometimes you need to allow the option to get data on a different way into your system. For example, if you have special applications (like anything on z/OS.. pain in the ass sometimes). Tl;dr: Look at the common ways to get data into Splunk, choose a couple of them and build your infrastructure around it. We, for example, are using a lot of Heavy Forwarders (HF) in different (V)LANs where applications send their data to us. So we are kind of flexible here. If a product doesn't support syslog, we can check for an existing Splunkbase app, install it on the HF and use a different way then. I don't think there are specific guides out there, atleast I don't know any. If you have a big project coming up, you might want to get Splunk involved when planning a big infrastructure. Skalli Edit: typo ... View more

There is a tool called bonnie++ (https://splunkbase.splunk.com/app/3002/) to measure IOPS. It's a good practice to place the path of your hot/warm buckets on the fastest disk, while you can remain the cold/frozen ones on the slower disk/storage. So, depends whether your SAN is faster than your local storage or not. Skalli ... View more

Yes, a third-party loadbalancer. Look at this blog post for the latest architecture Splunk presented: http://dev.splunk.com/view/event-collector/SP-CAAAE73 So, what you could do, is, place a third-party loadbalancer in front of your indexers, which get the data via HTTP event collection (HEC). I haven't set up a HEC environment myself yet, so I can't really give any tips about it. Skalli ... View more

Hi, I think this is a good start: http://docs.splunk.com/Documentation/Splunk/6.6.1/Data/WhatSplunkcanmonitor and http://dev.splunk.com/view/dev-guide/SP-CAAAE3A So, out of the box, Splunk can index: Files and directories Network events (also streams) Windows sources anything that uses Splunk's REST api What kind of answer do you expect? Your question is really.. abstract to be answered. Even if there was no appropriate method to get your data into Splunk, someone could use the Splunk SDK to write a modular input for your use case: http://dev.splunk.com/view/python-sdk/SP-CAAAER3 Skalli ... View more

Heavy Forwarders do not have a high availability/failover feature. ... View more

Now I do have S.O.S. configured (and running) on each of my search head cluster members, so do I also need to have S.O.S. installed on the indexers if what I want to have pushed down to the indexer layer from the search head is the _audit and _internal data?# You don't necessarily need to install the S.O.S. app on the indexers as well. You could just configure the index definition on your indexers yourself. An easier way, of course, would be to deploy the S.O.S. app from the master to the indexers. For Splunk data itself, there are no additional actions required besides modifying your outputs.conf to forward data to your indexers. But this forwardedindex.2.whitelist = (_audit|_internal|_introspection) will actually only send data from those three indexes. As a side note, we prefer weighted loadbalancing (I write this because I saw autoLBFrequency = 30 in your outputs.conf, so I assume you're not using weighted LB. Still, this value takes effect with weighted LB). You got quite many settings in your tcpout stanza. Skalli Edit: Damn, that other guy tricked me. It's an old thread. ... View more

Read the article. A SH sends its searches to all of the indexers in a cluster. The one with the data sends the data back. Basically, the cluster continues to work like it did when the master was last available. ... View more

This question and kind of similar questions have beend asked quite a lot. The cluster will continue to work as long as no other peers are going down, to give you a short answer. http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Whathappenswhenamasternodegoesdown Skalli ... View more

Hi, read these articles for how to make your app Splunkbase-compliant: http://dev.splunk.com/view/webframework-developapps/SP-CAAAEMY and http://docs.splunk.com/Documentation/Splunkbase/splunkbase/Splunkbase/Aboutsubmittingcontent Have a nice weekend. Skalli ... View more

Can you please post your complete inputs.conf and outputs.conf (masked of course) from your app directory and the inputs.conf from your indexer. This would help. Skalli ... View more

The answer kind of depends on the deployment. First, it's correct that you would need one master node for each indexer cluster. Second, you may have only one deployer for multiple SH clusters. This is only possible if all of your Search Heads get the same apps/configurations. Otherwise you need one deployer for each Search Head cluster. Reference: http://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/PropagateSHCconfigurationchanges#Deploy_to_multiple_clusters Third, your explanation is correct. You could set the master and the deployer to a deployment client and push configurations from the deployment server to the master and deployer. And from there, to the SHs and indexers. Edit: If you are not using SH clusters at all, deploying the configurations with the deployment server to the standalone Search Heads is alright then. ... View more