We discovered that when we stop deploying a particular app to a group of servers, that app is eventually deleted (not just disabled) from the deployment clients. This we like :-). Based on this, we had this idea of creating an app that exists on the deployment client after installation, but as soon as the deployment client gets real deployments from the server (which does not know about this temporary app on the deployment client) it would be deleted. In theory. But in practice that doesn't work. That sort of makes sense because there are other apps in $SPLUNK_HOME/etc/apps that don't go away either (search, learned, etc). So I'm curious about what makes the deployment client decide that an app should actually be removed from $SPLUNK_HOME/etc/apps? Clearly some interaction with the deployment server makes it do that. Is it because the deleted app is one that's "known" on the deployment server and it is clearly not deployed to the target client so it removes it? The key being that it's a known app on the deployment server? Or is it something else? Essentially the goal here is to create a temporary app that has only a valid deploymentclient.conf in it. It's valid in that it points to a host that will always have a deployment server, however, it could point this deployment client somewhere else via another another deployed app. This temporary app would really just boot-strap the client to start talking with a deployment server. Yes, I'm aware this is a little weird in that there are circumstances where the client could get orphaned if the communications get messed up. The problem for us is that creating a deploymentclient.conf in $SPLUNK_HOME/etc/system/local means we can't really remove it easily. (We don't have access to tools that would let us easily modify this dir for hundreds of hosts and it's easier to remotely control via deployments). Thanks ... View more

Typically we make new apps by just cloning the simple app (I forget what it's called) but that includes some ui files and other things we don't need. Suppose we're making an app that is just a couple of config files. What are the minimal files needed in an app if I'm not doing anything but a couple of conf files in local/*? Do I need the default directory at all? Thanks ... View more

We built a dedicated deployment server a while back when we started noticing (thanks to Splunk support) that running it in the same Splunk indexer was interfering with indexing operations a little. So we now have a dedicated deployment server instance that also runs on one of the indexer hosts. It seems as if the deployment server, while more than a fileserver, is really doing considerably less than a full Splunk server, yet that's what the basic file tree for the deployment server is. I realize there isn't a separate deployment server only package like the universal forwarder (maybe someday?), but I'm wondering if there isn't some way that one can turn off more things in the Splunk deployment server to lighten its footprint. I realize that I can turn off splunkweb and can tune the phonehome intervals. I don't suppose that other things can be turned off in the same vein as say things that were turned off in the lightweight forwarder to make the deployment server smaller? Thanks! ... View more

I've just started playing around with the PDF server on Splunk 4.1.6, SLES 10.2. I added the components mentioned by the docs page, although I'm not entirely sure about the fonts. I assumed that if the fonts were missing or an issue that I'd see an error message that made that pretty clear. After adding the PDF app and configuration the System Settings to use them, I wanted to try this out by generating an hourly PDF from the Search -> Index Activity dashboard. When I got mail, it said: Scheduled view delivery. A PDF snapshot has been generated for the view: index_status. An error occurred while generating a PDF of this report: Failed to contact appserver at http://fsplunk1:8000/en-US/report/: HTTP Error 502: Bad Gateway When I checked in python.log, it had the same HTTP error in it. The PDF server is the Splunk server (i.e. it's all local). If I hit the URL indicated above, it tells me: 405 Method Not Allowed Specified method is invalid for this server. So I clicked on the Status Page link under E-Mail Alert settings and everything shows up as expected. (I get the document in IE and I get the download for Firefox). What am I missing? Did I miss a step somewhere? Thanks ... View more

We're building out new Linux Splunk servers on dedicated hardware. These servers have a rather large amount of disk space available (over 2Tb) running on RAID10. They're both going to be indexers with events auto-load balanced between them. We're debating about how we want to allocate that for index space. Is there any particular reason why we would not want to create a 1TB or 1.5 TB or even 2 TB filesystem? As opposed to say, creating multiple smaller filesystems? We currently have no requirements for particular indexes so right now everything's just going to go into the same big pot. Thanks ... View more

We're trying to setup some test monitoring of a VMWare ESX host (not ESXi). Because our Splunk instance does not run as root, I setup the UDP listener port to be something above 1024. However, I'm not able to find anything about syslog configuration on an ESX server that shows how one might configure it to send to a remote syslog host on any port other than 514. So I don't know if it just won't work or not (the VMWare is going to try something I suggested in that regard tomorrow), but I'm not all that hopeful. So if I have to enable a UDP listener in Splunk on port 514, I assume that means I would now have to find a way to run Splunk as root rather than the non-privileged user I'm doing this as now? This becomes an issue because my team, who administers Splunk, are not the corporate sysadmins and as such are not given root privileges. ... View more

I have a question that I'm looking for some guidance on. Our division has a team that's interested in data that sits on a another division's (let's call them "ABC") server. That data (info on dialup users) is intermingled with info from users from many other divisions. As such ABC is hesitatant to give us access to it as we'll see everyone else's dialup data. Additionally, ABC does not run Splunk at all (although this is turning out to be a great introduction for them) and isn't likely to in the near term. So the discussion here is to let us put a LWF on their Windows server to slurp up this one particular log file that has everyone's dialup data in it. ABC has asked if they let us do this, can we only index the data for our division. I haven't seen the log file in question so I don't know how one would discern between our data and theirs, but I'm assuming it's possible. I'm hoping it's not a question of "here's 4,000 users, only index events that are from one of them". Obviously this would be much easier if our data was in a separate log file, but I don't think that's an option. I realize that the LWF isn't built to differentiate events as that particular engine isn't turned on to keep it light. So presumably the indexer has to do the heavy lifting here. If we can come up with some handy way to differentiate the events our data may be some minority of the full stream of events. We already filter out some very static events from being indexed, and I'm assuming that we can do this on a larger scale, but I worry that this will be a performance killer if we end up having to do it based on a large list of exclusions/inclusions. In a perfect world, I would see ABC having their own Splunk indexers and pulling the data in themselves. Then potentially giving us access to do a distributed search for this data against their servers. A question I would have there is, assuming there's some field we can use to differentiate our users from theirs, can they restrict a distributed search based on a field? That is, we could search a specific index on their server, but only if it matched a certain field value? Thanks! ... View more