To manage a Splunk installation (Search head, indexers, forwarders etc) there is a lot of direct file manipulation in my experience.
What is an effective/efficient method of managing these files besides a file share or direct access?
I was considering using a versioning system like Git. Any experience or tips in doing it this way?
Yes, as @lukejadamec said: Splunk deployment manager would work nicely. Many people use Puppet, if they already are using it to manage other config files.
But any decent configuration manager should work: it just needs to be able to deliver the right text files to the right directories on the right hosts... That kind of sounds like the definition of a configuration manager to me. 🙂
A file share is usually not a good idea. For forwarders, you usually want different configurations for different types of production systems. The data that you forward from a web server is not the same as the data you forward from a database server, for example. For search heads (or machines that are indexers+search heads), users can probably save and share knowledge objects like searches, etc. - this will cause config files to be updated. In this case, there could be conflicts between systems - how are you coordinating writes to the file share? Search head pooling?
Finally, while I think versioning is a good idea - be careful. Splunk itself does not understand versioning. It looks for configuration files based simply on their directory location and name. So make sure that the files actually delivered to the Splunk instances have the correct names. Take into consideration that some configuration files (savedsearches.conf
for example) may be updatable by users on the fly. Avoid duplicating configuration files unnecessarily. Make sure you understand configuration file precedence.
In general, I try to establish certain Apps (like search
) that users will be allowed to update - I don't manage those Apps on the search head(s). I make sure that all the other Apps are locked down (read-only for users) and manage them from the deployment server or other config mgmt tool.
I don't know what you mean by "direct access" unless that is just another way of saying "file share." See above.
None of this stuff is rocket science, but you do have to think it through and plan it out.
Yes, as @lukejadamec said: Splunk deployment manager would work nicely. Many people use Puppet, if they already are using it to manage other config files.
But any decent configuration manager should work: it just needs to be able to deliver the right text files to the right directories on the right hosts... That kind of sounds like the definition of a configuration manager to me. 🙂
A file share is usually not a good idea. For forwarders, you usually want different configurations for different types of production systems. The data that you forward from a web server is not the same as the data you forward from a database server, for example. For search heads (or machines that are indexers+search heads), users can probably save and share knowledge objects like searches, etc. - this will cause config files to be updated. In this case, there could be conflicts between systems - how are you coordinating writes to the file share? Search head pooling?
Finally, while I think versioning is a good idea - be careful. Splunk itself does not understand versioning. It looks for configuration files based simply on their directory location and name. So make sure that the files actually delivered to the Splunk instances have the correct names. Take into consideration that some configuration files (savedsearches.conf
for example) may be updatable by users on the fly. Avoid duplicating configuration files unnecessarily. Make sure you understand configuration file precedence.
In general, I try to establish certain Apps (like search
) that users will be allowed to update - I don't manage those Apps on the search head(s). I make sure that all the other Apps are locked down (read-only for users) and manage them from the deployment server or other config mgmt tool.
I don't know what you mean by "direct access" unless that is just another way of saying "file share." See above.
None of this stuff is rocket science, but you do have to think it through and plan it out.
I don't think they are equal - AFAIK, you can organize the packages in a lot of ways in Puppet. Splunk is only going to do things the Splunk way - but Splunk can manage a mixed environment (Linux + Windows, etc) easily. I think that is harder with Puppet.
So I think the trade-off will be that Puppet can do more, and is more complex. It will require that you install and learn Puppet. Splunk deployment management is already built into Splunk, but it can only deliver apps and updates to apps.
I will take a close look at Puppet. I see lots of modules and manifests out there, so it looks promising.
As to the deployment manager, is it common for the Deployment manager and indexer to be separate hosts? Because, while the indexers are distributed across subnets, I really want to be able to manage the the forwarders from one central location.
Functionally, would Puppet and the Splunk Deployment Manager be equal?
Thanks for such a detailed answer! I have been sharing out the splunk installation directory for development, and this is just for admin access. However this will not fly for the production domain.
As to versioning, something like git hides all the versioned data and only presents one instance of a given file. But I still think I agree with you that it is not the right tool for the job.
By direct access I mean remoteing to the server (RDP, SSH, PS Session etc) and editing the files in that session.
Why not use the deployment manager?