Howdy, I want to ingest files on a universal forwarder that are still being written, and to delete them once the file has finished being written to - i.e. no other open file handles. Is this a default behavior? Basically I want to write a large number of files to disk, and deliberately only keep them open for a short period of time, where short could be a minute, and after that time, a new file would be created, timestamped with a different minute, for example. So I'm ONLY writing them to disk for Splunk to grab hold of them, and once splunk has read the entire file, it'll serve no further purpose. But I want Splunk to delete it when it knows it can (with UseACK enabled etc.). So if Splunk is poorly then the files will just mount up until it can pick up the backlog. I'm just concerned that the file will be deleted as soon as it reaches the end of the file at a given point in time, and then the rest of the data would be written to a deleted handle? Would Splunk just hang on each file until it is closed? Is there no EOF until that point anyway, or would it always see an EOF if it runs out of data? ... View more

Hi, I need to forward from a few Universal Forwarders to some third party indexers, and I'm not that keen on actually using the endpoint IP's in the UF config. I'd rather connect to a single local vip on a load balancer and let it worry about it. Is there any risk in doing so? I'd be monitoring the endpoint health with frequent TCP checks, but past that the UF would only have one IP to connect to. If it gets a broken connection, will it mark that one endpoint as unavailable, even if it has valid connections passing through to other remote endpoints behind the vip? Can I configure it to have, say, 4 different connections concurrently, with the UF oblivious to the fact that there are multiple indexers receiving those data streams? Thanks ... View more

Hi, Is it possible to enable useACK without sending cooked data? If we have two independent splunk worlds, and I'm forwarding data from one to another, it seems illogical to send cooked data when the source types and indexes don't match up. But I need the assurance that useAck (allegedly) provides. ... View more

I've been advised that if we want to send a single log file to two different indexers (with ACK's enabled) then we may have HA issues if one of the two clusters is down, i.e. it won't send to the active cluster either after a certain buffer is filled in order to avoid data loss. The suggested solution was to install a second indexer, and read the same file twice that way. We've been working with this presumption, but now it looks to me like we could instead just define the monitor twice, and use a different crcsalt value in each on (this is actually a directory tree of static gzip files that are not rotated, just deleted after a certain age), to allow them to read the raw disk data twice. Does this seem logical? Potentially via a symlink to the same directory if you can't define the exact same file twice in a monitor? ... View more

Hi, I'm building a cluster which will be authenticated and AD via LDAP. We will also be permitting a 3rd party to query the indexers from their own search head. How would user authentication work in this model? On the indexers we restrict certain roles to certain indexes, but I'm unclear what level of authentication will be inspected for the queries from the remote SH, which will manage its own users independently. How would an admin user be indentified by the indexes compared to a normal user? Will we have to duplicate their users into our AD for account lookups and presume they are legitimately pre-authenticated? does the search call also send the roles that user has from their Search Head? ... View more

Hi, I'm designing a deployment where there will be a search head on the other side of a NAT boundary to an index cluster. In order to fully connect the search head to the cluster, the SH will connect to the cluster master and poll back details about all the indexers, exchanging keys etc. As I understand it at least, you can't trivially just point the SH at a cluster. As such, the addresses that the SH will then query are the same addresses that are used within the cluster. So if raw IP's are used, then that will be what the SH tries to hit. If there is a NAT boundary between the devices, those actual addresses are not reachable. So am I right in believing that if the cluster is assembled using local hostnames that are locally resolvable within the cluster (e.g. in /etc/hosts), then on the SH alternative host entries can be provided (again in /etc/hosts) which can relate to the addresses on the NAT boundary, allowing end to end connectivity to happen? Alternatively... am I thinking this is more complex than it is on the clustering side and you can just tell the SH to hit an alternative IP? Additionally, if this boundary device is HTTP aware, would splunk care if the unique hostnames resolved to the same IP address? Here we can then use the HTTP Host header to connect the SH to the right indexer in a scalable way. (This is all configured with SSL disabled BTW, so I can now see the raw API calls trivially) Thanks ... View more

Hi, We have a Search head on the other side of a WAN which needs to search against an Index cluster in a sensitive network segment. A security requirement is that we must use an intermediate device as a broker of some form between these network areas. The protocol is apparently not HTTP, so an HTTP proxy is no use, what else might be available to allow these systems to connect in line with our security policy? ... View more

Hi, is there anything pokeable from a load balancer over TCP to validate the availability of a TCP data input? I can potentially just rely on a TCP connection working or not to validate it's availability, but ideally I'd like to get something back to my BigIP load balancer to validate it's health to a deeper level. We have the option of doing some form of script to do a search against the splunk port I suppose, but in the first instance I'd like to stick with only using the TCP data input port itself, not going round the back. Thanks! ... View more

Hi, I'm looking to replicate the output from streamfwd with a separate script, and from what I understand of the Stream solution, it's entirely on the client to send fully formed messages to the server. But obviously streamfwd provides all the Proprietary metadata compared to just spitting a JSON structure into a TCP port which is what my script will need to do (I presume). A key thing to allow the data to be used as if it were actually from streamfwd is the setting of the sourcetype. How can I best allow these messages coming through a single TCP input to have a proper sourcetype as defined by the client? I believe it's possible for props.conf to mangle sourcetypes upon indexing, but I'd like to stay away from that if possible and let the client define it just like I believe streamfwd does. I noticed I could just put "sourcetype" as a field in the JSON payload, but this left the logs having two sourcetypes, the one set there AND "syslog" as defined on the TCP input itself. ... View more