If I have a standard deployment architecture with deployment server and clients and deploy an app to a client, the app will not get updated on the client until I do any changes on the deployment server.
However, if on the client the app is changed by a local system administrator (e.g. disabling audit log input) Splunk will not undo these changes (until I change the app on the deployment server) and I have no chance to recognize this change.
- Is there a chance to enforce an app on the client? (Besides changing the app regularly on the deployment server)
- Is there a smart way to at least monitor the app for changes on the client?
- Did the behavior change from any previous version?
It might seem a little silly, but if you absolutely want to not have your app modified on the deployment client you can trick it.
Now whenever the deployment client phones home to the deployment server it will see an updated app and download it.
The local sysadmin will hate and resent you for your omnipotence, but you'll be a Splunk ninja and you won't care.
(Use a Windows variation in AT if you are on that platform,)
Oh, don't go wild and be too aggressive in scheduling it to run in cron, because you need to give the download, deploy and restart some time to occur. Use your own judgement for your situation.
After discussing this otherwise again, I'm going to correct my comment in one thing:
All that changes live on the client as long as the DS will NOT get reloaded or restarted (not the restartSplunkd option). In that case woodcock is right, it instantly will override the app as soon as the client has been phoned home. But without that reload, you can make changes which are not wiped immediately.
When the client phones home it usually calculates a checksum of its app contents and requests the same from the DS depending on the serverclass he is configured in. But it is using the same checksum like it used before when the DS has not been reloaded. That means the DS triggers the client to do that recalculation. If not reloaded - the checksums remain the same - even though there could have been a change on the client. It is unfortunately not clear explained in docs.
If you are using Forwarder Management, every click and save triggers a reload of the DS. Doing changes on the CLI you have to use the CLI command. But if the DS has not been notified about changes he will give the client the last checksum since its last reload, so the client does.
So that is why the behaviour looks sometimes different, depending on how the DS is used.
I did a few checks on a deployed App
1) Just touched an file within the App Directory >> App was not updated even if I do a splunk reload deploy-serer
2) Changed GUI visibility within app.conf & transforms.conf >> App was not updated even if I do a splunk reload deploy-server
3) Restarted the Client so that changes >> App was not updated even if I do a splunk reload deploy-server
So no, in my Environment I do not see the expected behavior.
Strictly speaking, you need a file integrity monitor to send logs to Splunk and alert based on non-deployment server pushed changes.
For a low-rent solution, you could set up a cronjob on the endpoints that generates a checksum of app directories and logs those to Splunk. Then use a script to building a lookup table of checksums from the deployment server's app directories. Run a verification check against the checksums coming in to Splunk to alert.
Of course, either of these can be subverted by the local sys admins. Have you considered sitting down with them and discussing the problem to try to find out why they are making changes to Splunk configurations without working through you? It could just be that they don't understand the negative impact this has on managing the endpoints. Offering them a solution and working as a team to resolve the issue might result in a more positive outcome.
Splunk has a setting on the DS called
restartSplunkd. The documentation says that this means that when this app is deployed, that the splunk instance on that forwarder will be restarted. HOWEVER, I had a very heated disagreement with a client just yesterday where I insisted that DS aggressively maintains the app contents on the client such that anything that changes on the client is changed back immediately. The client insisted the opposite and actually showed me that he was right (at least on his deployment). The only difference that I could see was that I was using
restartSplunkd on my apps and he was not (and also the splunk version). I would try setting
true and see if this will enable the behavior that you desire. If so, this aspect is completely undocumented but I can assure you that I did see the behavior that you desire at my last client. Please report back your findings.
From what I have experienced: it has nothing to do with using restartSplunkd in serverclass or not. The use of that option depends on your configuration you put in to your app (http://docs.splunk.com/Documentation/Splunk/6.3.0/Admin/Configurationfilechangesthatrequirerestart).
A deployment client can have many local changes to an app without being overwritten from DS. If that would not be the case, you could never test apps or extend knowledge to an app while managed by a DS. Only if you change the app on the DS, next time the client phones home it gets its list of new or updated apps and downloads the bundle, after that it will wipe the app before unpacking the archive. Then you lose local configuration in such app if it has not been updated to the app on DS. You have to make your user aware of that fact. If you want to force no local changes to an app - you better restrict write permissions in metadata for parts of the config or in general by default. That all works only in splunk context. There is no protection against changing files on the filesystem to gain back write permissions or change other configuration directly.
Correct me if I'm wrong, behaviour sometimes changes with new versions.
I no longer have access to the systems but at ALL of my previous engagements, the forwarders' app files were constantly maintained with the DS and changes on the forwarder would immediately be overwritten, even when nothing was changed on the DS.