Getting Data In

Best way to use git for source version control with our Splunk infrastructure?

Champion

Hi all,

I'm looking for some advice on source version control. We have a couple deployers, a deployment server, indexer cluster, bunch of forwarders and various supporting splunk servers.

And we of course have a lot of different apps/TAs that need deployed to different parts of the infrastructure, typically with some overlap. From everyone we talked to before and the sessions at conf, it seems like we need to come up with a source control strategy and git seems like a good tool to use. I've been reading the basics of using git (never have before), but trying to understand how best to use it with our splunk infrastructure.

Should we just have some central location where all of the apps and their repositories are? And then when we need to make a change, we check it out, copy it to our test instance, modify as necessary, copy back and commit? And then just manually copy the updated directory out to all of the places it needs to go.

Or do we actually have repositories on the various deployment servers and sync those up to some master repository. For example, our main indexes app goes out to the both shc and in the cluster master. Would the apps in all 3 locations be actual git repositories? Sharing some master repository?

For those that do something similar, any advice or lessons learned you could share would be appreciated. Right now we're just using the old central share approach with a lot of copying around.

Thanks!

Champion

Hello @maciep,

I used git in a couple of deployments of Splunk. I would recommend setting up a git server such as GitLabs, Stash by Atlassian, or use an preexisting Git server in your environment. This git server would host all your Master repos for all Apps and TAs. User would simply create local clone of the master. You can just make edits to the master branch or use any number of branch techniques. I am a big fan of not let anyone commit directly to master.

User Workflow:
Users would clone the remote master from your git server to test server or desktop. Then create a local branch which they could edit and test. After they are happy with their edits commit to the local branch and push to the remote repo. From the Git Server (Atlassian) the user initiates a pull request for their branch. Once the pull request has been approved the branch could then be merge into master.

Deploy Repos to servers:
You will most likely have to make use of git hooks. At this point you could try to tie into puppet or chef to deploy the bits to each deployment server, deployer, and or master. Alternatively you could setup a simple git server on each Splunk server which could be used by the Git Server (Atlassian or other) with a post hook to listen for merges and then push the updated master to the remote simple git servers. On the simple git servers (splunk servers) configure a Post-recieve hook to deploy the repo to the appropriate directories

Dont forget to include a .gitignore in each repo.

# .gitignore for Splunk
*local.meta
*local/
*.spl
*.tar
*.r.gz

Example of how user would interact with repo.
1. clone remote repo to local system: git clone git@github.com:httpstergeek/d3-splunk-extentions.git
2. create working branch: git checkout -b workingbranch
3. create some files and edit some files.
4. add new files to repo: git add --all
5. commit changes with comment: git commit -am "I made some changes"
6. push local branch to remote repo: git push origin workingbranch (Note: this will create branch on your remote git server inside the repo)
7. User logs into Remote server using UI/cli and initiates a pull request to master. Once pull request is approved. The branch called workingbranch will be merge to master.

Take a look at this article https://www.digitalocean.com/community/tutorials/how-to-use-git-hooks-to-automate-development-and-de.... There is a lot of moving pieces and I know I am simplifying a lot especially if you are not familiar with git and setting up a git server. I still hope this helps.

Path Finder

In addition the the good advice from @bmacias84 , I recommend also including a .slimignore file. This will exclude your .git and other directories that shouldn't be published in the app.

https://dev.splunk.com/enterprise/docs/releaseapps/packagingtoolkit/pkgtoolkitref/packagingtoolkitcl...

# Default Splunk Packager exclusions
.DS_STORE
Thumbs.db
*.py[co]
default/indexes.conf
local/
metadata/local.meta

# Additional exclusions
.git
.gitignore
venv*/
.idea/
.vscode/

 

Here is an example from Splunk's Cisco Webex Meeting App:

https://github.com/splunk/cisco_webex_meetings_app_for_splunk/blob/master/.slimignore

Another option is to use a subdirectory for the app, as Palo Alto Networks did recently when they consolidated their apps into one GitHub repository.

https://github.com/PaloAltoNetworks/Splunk-Apps/

0 Karma

Champion

thanks, great info!

0 Karma

SplunkTrust
SplunkTrust

great answer

0 Karma

SplunkTrust
SplunkTrust

Hi,

There are probably plenty of ways to manage this, but I can share my own experience.

We do use git to manage all of our Splunk dev (and we dev a lot !), definitively this is a very good solution and git is easy and wonderful. So in my opinion your intention to use git is very good idea 😉

What do we do is easy:

  • 1 git repository per application
  • In the repository, we split dev in 3 or more parts: PAxxx to be deployed in indexers, splxxx to deployed in search heads, TA_xxx to be deployed in heavy forwarded or universal forwarders (we add a suffix for HF and another for UF)

Then, we use the master node as the central server for deployments.
When dev wants to publish in qua / prod,
We use some developed tool script to clone and/or pull from remote in a local repository on the master node.

Then we rsync to master-apps/ for indexers, shcluster/apps for sh cluster, and deployments apps for deployment servers.

And finally we publish using internal Splunk commands (bundle publication and reload deploy servers)

For dev that will not have distributed infrastructure such as qua / prod, we clone locally and simply creates required symbolic links in etc/apps

It is easy enough and simple, these scripts can easily be scripts to match scheduled or production rules with dedicated shell menus and so on.

Next step for use will probably be some integration with some advanced configuration management, ansible most probably.

Just one way to do the job, hope this helps

Champion

it does, thanks!

0 Karma

1 git repository per application

I'm developing a Splunk app that contains several dashboards. I'll be distributing the app via a Git repo.

Currently, the Splunk/etc/apps/myappname directory is (a clone of) the Git repo.

I keep a (GitHub-flavored markdown) README.md in:

Splunk/etc/apps/myappname/README.md

Unless there are good reasons not to continue doing this - hence this comment; I'd like feedback - I'd like to keep it that way.

One reason I'm making this comment: in addition to the README.md, I want to distribute some additional files that are not strictly part of the Splunk app itself, such as:

  • Scripts (specifically, IBM z/OS JCL) for use outside of Splunk, to forward data to Splunk.
  • Example dashboard screenshots (png images referred to in the readme).

and I'm wondering where to put those files under the myappname directory, and how/whether Splunk will even "care". For example, could I just put a screenshots subdirectory directly under the myappname directory: myappname/screenshots? Or should I be more "Splunk-y", and create a static subdirectory under myappname, and then create a screenshots subdirectory under there: myappname/static/screenshots/ (and perhaps also myapps/static/jcl/)? But then, those screenshots and JCL aren't intended as files to be served via HTTP for use in the dashboards. Perhaps a "doc" directory, if Splunk doesn't "mind" that?

Argh, I didn't mean to "reply" to the particular comment I did. I'm hoping people will still read this; I'm not sure it deserves its own question.

0 Karma