Deployment Architecture

How to monitor the status of an app for changes on a deployment client?

doitslu
Explorer

Hi,

My Understanding:
If I have a standard deployment architecture with deployment server and clients and deploy an app to a client, the app will not get updated on the client until I do any changes on the deployment server.

The Problem:
However, if on the client the app is changed by a local system administrator (e.g. disabling audit log input) Splunk will not undo these changes (until I change the app on the deployment server) and I have no chance to recognize this change.

The Questions:
- Is there a chance to enforce an app on the client? (Besides changing the app regularly on the deployment server)
- Is there a smart way to at least monitor the app for changes on the client?
- Did the behavior change from any previous version?

0 Karma

lycollicott
Motivator

It might seem a little silly, but if you absolutely want to not have your app modified on the deployment client you can trick it.

  • Create a cron script that contains a line like this: date > your_app_path/default/enforce.conf
  • Schedule it to run nightly or hourly or whatever you wish along with "spunk reload deploy-server -auto admin:password"

Now whenever the deployment client phones home to the deployment server it will see an updated app and download it.

The local sysadmin will hate and resent you for your omnipotence, but you'll be a Splunk ninja and you won't care.

(Use a Windows variation in AT if you are on that platform,)

0 Karma

doitslu
Explorer

I guess this is to aggressive. Imagie whathappens if an App forces Splunk to restart after it is installed ...

0 Karma

lycollicott
Motivator

Oh, don't go wild and be too aggressive in scheduling it to run in cron, because you need to give the download, deploy and restart some time to occur. Use your own judgement for your situation.

0 Karma

apfender_splunk
Splunk Employee
Splunk Employee

It wont work if you don't reload the DS. The client talks to the DS and doesn't look around itself. Without a reload the DS doesn't know about the changes.

0 Karma

lycollicott
Motivator

I guess I felt since the post was about reloading that that part was understood, but I have edited my original to be more explicit. I should have stated it up front. Good catch

0 Karma

woodcock
Esteemed Legend

What @apfender is saying is sometimes true but definitely sometimes is not. The problem is that I do not know what controls the behavior.

0 Karma

apfender_splunk
Splunk Employee
Splunk Employee

After discussing this otherwise again, I'm going to correct my comment in one thing:

All that changes live on the client as long as the DS will NOT get reloaded or restarted (not the restartSplunkd option). In that case woodcock is right, it instantly will override the app as soon as the client has been phoned home. But without that reload, you can make changes which are not wiped immediately.

When the client phones home it usually calculates a checksum of its app contents and requests the same from the DS depending on the serverclass he is configured in. But it is using the same checksum like it used before when the DS has not been reloaded. That means the DS triggers the client to do that recalculation. If not reloaded - the checksums remain the same - even though there could have been a change on the client. It is unfortunately not clear explained in docs.

If you are using Forwarder Management, every click and save triggers a reload of the DS. Doing changes on the CLI you have to use the CLI command. But if the DS has not been notified about changes he will give the client the last checksum since its last reload, so the client does.
So that is why the behaviour looks sometimes different, depending on how the DS is used.

doitslu
Explorer

I did a few checks on a deployed App

1) Just touched an file within the App Directory >> App was not updated even if I do a splunk reload deploy-serer
2) Changed GUI visibility within app.conf & transforms.conf >> App was not updated even if I do a splunk reload deploy-server
3) Restarted the Client so that changes >> App was not updated even if I do a splunk reload deploy-server

So no, in my Environment I do not see the expected behavior.

0 Karma

nnmiller
Contributor

Strictly speaking, you need a file integrity monitor to send logs to Splunk and alert based on non-deployment server pushed changes.

For a low-rent solution, you could set up a cronjob on the endpoints that generates a checksum of app directories and logs those to Splunk. Then use a script to building a lookup table of checksums from the deployment server's app directories. Run a verification check against the checksums coming in to Splunk to alert.

Of course, either of these can be subverted by the local sys admins. Have you considered sitting down with them and discussing the problem to try to find out why they are making changes to Splunk configurations without working through you? It could just be that they don't understand the negative impact this has on managing the endpoints. Offering them a solution and working as a team to resolve the issue might result in a more positive outcome.

0 Karma

woodcock
Esteemed Legend

Splunk has a setting on the DS called restartSplunkd. The documentation says that this means that when this app is deployed, that the splunk instance on that forwarder will be restarted. HOWEVER, I had a very heated disagreement with a client just yesterday where I insisted that DS aggressively maintains the app contents on the client such that anything that changes on the client is changed back immediately. The client insisted the opposite and actually showed me that he was right (at least on his deployment). The only difference that I could see was that I was using restartSplunkd on my apps and he was not (and also the splunk version). I would try setting restartSplunkd to true and see if this will enable the behavior that you desire. If so, this aspect is completely undocumented but I can assure you that I did see the behavior that you desire at my last client. Please report back your findings.

0 Karma

apfender_splunk
Splunk Employee
Splunk Employee

From what I have experienced: it has nothing to do with using restartSplunkd in serverclass or not. The use of that option depends on your configuration you put in to your app (http://docs.splunk.com/Documentation/Splunk/6.3.0/Admin/Configurationfilechangesthatrequirerestart).

A deployment client can have many local changes to an app without being overwritten from DS. If that would not be the case, you could never test apps or extend knowledge to an app while managed by a DS. Only if you change the app on the DS, next time the client phones home it gets its list of new or updated apps and downloads the bundle, after that it will wipe the app before unpacking the archive. Then you lose local configuration in such app if it has not been updated to the app on DS. You have to make your user aware of that fact. If you want to force no local changes to an app - you better restrict write permissions in metadata for parts of the config or in general by default. That all works only in splunk context. There is no protection against changing files on the filesystem to gain back write permissions or change other configuration directly.

Correct me if I'm wrong, behaviour sometimes changes with new versions.

0 Karma

woodcock
Esteemed Legend

I no longer have access to the systems but at ALL of my previous engagements, the forwarders' app files were constantly maintained with the DS and changes on the forwarder would immediately be overwritten, even when nothing was changed on the DS.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!