That's also not entirely true. When you say "expects already-defined json events" one could interpret it as "you can only ingest events which are json structures using this endpoint" which is completely untrue. You can ingest any data you want, you just have to use the enpoint by means of sending appropriately formated json event containing the raw event as "event" field. See https://help.splunk.com/en/data-management/get-data-in/get-data-into-splunk-enterprise/10.4/get-data-with-http-event-collector/format-events-for-http-event-collector for details. ... View more

Depends on what you mean by "access point". I've seen setups with where you'd log in into a central identity provider and from there you could pivot into either envionment's web interface and be automatically logged in with SSO. But that's it. You can't "have ES dashboards" on a non-ES-search head. You can't have KOs defined on one cluster available on another cluster. Every search head environment is separate so unless you're using federated search to spawn searches against another SH environment (which has its own caveats and possible issues as I've already mentioned I think) there is no "knowledge sharing" or "object sharing" between different SH(C)s. Of course you can have multiple environment searching the same data from the same set of indexers but that's a completely different story. And you can share datamodel accelerated summaries but you still need to have the same definitions and settings on both SH environments. ... View more

Yup, but the rules regarding HEC are not Kafka-addon-specific 🙂 ... View more

This is not 100% true. Most is but not all. With /event endpoint: The source _can_ provide index/source/sourcetype/host values but if it doesn't the input assumes the default values. Also if a source requests that event be sent to an index which is not allowed with the provided HEC token, the request will fail Unless explicitly requested with a proper URI parameter, timestamp recognition does _not_ happen with /event endpoint. And even if you want Splunk to parse timestamp but provide one on your own, time parsing won't happen. For /raw endpoint: To be fully honest, I don't think I've ever tested it with a not-line-oriented sourcetype but I wouldn't be surprised if it did breaking on a per-line basis. But that's just my hunch. Might as well be as you're saying so just test it for yourself if you wanna rely on it. Most notably I'm not sure how the input will react to different parts of a multiline events sent with different HTTP requests. I would be very surprised if it had a way to "correlate" those two separate requests into a single line. So test before use! You _can_ provide destination metadata with your request by means of URI parameters or custom headers. But you have to do it for a whole HTTP request, not for each single event separately. ... View more

That's a quite old thread but since you responded, instead of fiddling with script contents you can try setting TLS parameters in  [pythonSslClientConfig] section of server.conf. ... View more

I'm not sure the CM's policy of assigning replication partners is anywhere officially documented so the best you could hope for would be trial and error. Anyway, if you deliberately broke replication from site1 to site2 in hope of getting buckets to go from site3 to site2... You'd get a lot of errors and alerts. I suppose your CM would have to do much more work because initial replication attempts would fail and it would have to initiate fixups (which could fail again. So I would strongly advise against going that route. At best you'd have an unsupported setup with a very sensitive configuration which could go sideways at any moment. At worst - you'd break your whole clustering setup. And no sane support engineer will help you here other than by saying "go to the default settings". ... View more

Instead of @w+4d you can use the @w4 form ... View more

OK. As often with Splunk, there are several possible ways of doing something. In this case we're talking about ingesting windows eventlog to Splunk. You have 4 different possible methods. The first - recommended - one is by installing an UF on the source host and read directly from the EventLog. This is the most sure way, easiest to configure, debug, and maintain. The second - a bit less recommended but still OK - configure WEF to forward logs onto another host (possibly a common receiver for several WEF sources) and install the UF there and pull the events from Forwarded Events log. You can do it without installing UF on the source machine(s) but you still need the UF on the WEF collector. And WEF is tricky to set up if you don't have AD domain - it is possible but somewhat complicated and very painful to debug. The third option which is much worse than those ones above is to use another windows host - again with the UF installed - to pull the events remotely from the source host using WMI. This is a very bad solution performance-wise, it requires AD and granting your Splunk service account specific privileges on the source host and a lot of other configuration (you cannot do that without domain unless you open your WMI to just everyone in your network which would be somewhat unwise to put it gently) and is prone to all sorts of errors. WMI is generally not something you want to use everyday. And definitely not for this. And the fourth option is that you use some third party tool to get data from the event log and forward it to your Splunk infrastructure (typically by means of syslog). I've seen it done but it practically never ends well. The main reason is that third party components butcher the format of the events so that while you might technically end up with the events being indexed in your Splunk, they will be useless because they won't match the format expected by the windows add-on which is one of the most common add-ons you can think of. So you might even have the data but it won't parse properly and you won't be able to make heads or tails from the data. So don't go down this road unless you can somehow make the data ingested this way identical to the one you'd get with a UF. But that's completely unsupported method and it all on you. It's also worth mentioning that in this case you'd be adding another separate component to your environment. Such component would have to be configured and maintained by you. ... View more

Generally, you should be using CM-provided indexes.conf to define indexes on your indexers. Full stop. The docs state that you technically can create local-node only indexes.conf to define an index which is hosted on just a single indexer but it's not a good practice and nothing but pain awaits this way 😉 Docs even say explicitly that you should not try to deploy replicated indexes by means of local indexes.conf. The reason provided there is lack of consistency between peers and that the source indexer might try to start replicating to an indexer which has no knowledge of an index and fail. It's a valid point. That's why - as I said before - you must use CM-distributed indexes.conf. In a clustered environment CM should be the source of all configuration apart from some very very node-specific things like individual certs, node names, sometimes bind addresses. All "functional" configuration should be distributed in CM bundle! Otherwise you might get into all sorts of trouble if your config gets inconsistent across nodes. Therefore you should have your repfactor set consistently to either auto or 0 on all nodes in your cluster. The replication factors themselves are configured in server.conf on the CM (because the CM controls replication).   ... View more

Normally (but again - that might mean that in your particular case you just hit an outlier ;-)), sensitive data is encrypted on first use. That means that you either put into the file the data already encrypted with "splunk show-encrypted" (remember that this is encrypted with your local Splunk instance's splunk.secret key so data encrypted this way is not portable between different instances) or you put the plain text password/key/whatever and Splunk when reading that data for the first time replaces plain-text contents of the config parameter with encrypted version. EDIT: The typical case when you see unencrypted secrets in config files though is when you make a typo (usually a wrong-cased letter) in the parameter name so Splunk neither uses your parameter nor encrypts it. Like "pass4SymKey" or "pass4symmKey" instead of "pass4SymmKey" ... View more

It's a bit complicated 😉 If you want the full picture (but you might get overwhelmed and a bit confused), see the Masa Diagrams - https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774 The one that will be of interest to you is the last one - the one with a separate forwarder. Simplifying a bit it works like this: 1. A forwarder has inputs 2. Inputs push data (in case of UF the data doesn't have to be broken into single events yet but that's not important here) into the main pipelines (by default one, but there can be more). 3. From the pipeline the events go to the output(s). And now there can be some other stuff which can cause issues if you tweaked the internal Splunk queues but generally, you have fairly small queues which are responsible for current data processing within the forwarder process. What is important is that: 1) If the data cannot be sent out (network outage, indexers disaster, whatever), there is a pushback on the overall pipeline so the outputs block and almost immediately the inputs get notified that theh have nowhere to send to. 2) Some inputs (if they aren't explicitly configured to not do that) will stop reading from their sources as soon as it happens - most notable examples are file monitor input or windows eventlog input. In that case the forwarders simply wait for the output to "unclog" and then resume reading and sending data. The caveat is that if you have sufficiently long outage period on the output side, the source can eventually "roll over" some files or events in eventlog channels which appeared during the outage period might get rotated out of existence during that outage period. 3) Other inputs (typically those that passively receive data like network ports, HEC inputs) can use persistent queues which save the queue contents on disk so if there is an "output outage" they can handle even more data than the UF actually has memory. It's a case for just this type of scenarios. So to sum up your questions: 1. Technically the UF can hold in its memory some limited buffer of event data and - if configured - can store on disk a significantly larger (but still finite) buffer of input data. 2. It all depends on your configuration and resources. If you have a small "forwarding only" UF which is meant to just receive and send as soon as it gets the data and don't have much local storage to even configure persistent queue onto, you're limited not even by UF but by physics 😉  3. The normal procedure is that if an UF cannot contact the destination server it makes some more attempts then - if still unsuccessful - puts a particular output into quarantine for a fixed amount of time (so that it doesn't waste time and resources retrying to connect to an apparently unavailable server). If there are more outputs in the output group, UF just skips the quarantined output when choosing which output to send data to or if there are no working outputs in that group, it just blocks the output group (and then the whole thing I described above happens). If the quarantine period for a given output passes, UF retries connecting that output. If it succeeds, the output works. If it fails, it gets requarantined and the process repeats after another quarantine period passes. So unless you encounter some error in the code, the output should resume on its own after a short time. 4. Well... Generally the best practice is to avoid shutting down the cluster as a whole. If you absolutely must, just put your cluster into maintenance mode so it doesn't start replicating data wildly as soon as you bring the first one down, and shut down your indexers. You can consider putting all your indexers into manual detention before shutting them down so that the sources do not fail over to the last remaining one(s) overloading them with data just before shutdown. ... View more

As a rule of thumb, if an app is decently written and uses Splunk's mechanics for handling secrets, that is the way to handle encrypting/decrypting secrets. But noone can tell you with 100% confidence that you will never ever find an app in which the developer decided that they want to do everything their own way and use some custom encryption scheme. For example, many (most?) modular inputs are written in python and if they need, they use certificates saved as PEM files. But some modular inputs spawn java components and those use java keystores. So refer to your app's docs - they should know best 😉 ... View more

It's not quite clear what you want to achieve, but as a rule of thumb I'd say let users work with what they need. So if they need to work with ES, give them access to that stack. If you have a custom non-ES app which requires separate environment, give them access to that stack. Yes, as @livehybrid pointed out, keeping KOs in sync can be painful in any case when you have more than one SH(C) but that can be alleviated if you have a sound configuration management process and can easily - for example - upgrade an app in all places (SHs, HFs...) where it's needed. You could try setting up a separate SH and using federated search but this is yet another layer of complication of your environment with its share of potential problems and access control. ... View more

Disclaimer: I'm not digging into the Splunk side of what you're doing here. rm -fr $(ls -1|egrep -v "^splunk.secret") Ugh. Maybe in this particular case it will work but as a general pattern it's a very bad habit to do it like this. It will happily - as we in Poland say - "go into the bushes" even with a space in a filename $ mkdir test $ cd test $ touch "1 2" $ ls $(ls -1 | egrep -v whatever) ls: cannot access '1': No such file or directory ls: cannot access '2': No such file or directory Much better way to do it (and not prone to doing strange things in similar cases) would be to use find find . -maxdepth 1 -type f -not -name splunk.secret -delete   ... View more

1. You're butchering the copy-pasted content. Use preformatted paragraph or a code block. 2. Upgrading SHs shouldn't normally affect connectivity to indexers. 3. There is a channel on Splunk's Slack dealing with version 10 upgrade issues. You might have just hit an issue with specific version. ... View more

OK, it does seem that the add-on is a bit outdated (among other things, the installation tab on Splunkbase points to a non-existing link for OCI Splunk Logging plugin. You might want to look into  https://github.com/splunk/Splunk-Addon-for-OCI/issues/15 - if you're using Splunk version which still supports Python 3.7, you might be able to force it to run for now. ... View more

Which add-on have you tried? As I understand, https://splunkbase.splunk.com/app/5222 is "the right one". True, it's archived but that doesn't automatically mean that it won't work. It's just that noone has made any updates to it so that it just got autoarchived. Also while I'm no expert on OCI, the description for that app highly suggests that it works on a "pull" basis so no exposing ports to an outside world should be needed. ... View more

In other words, you have to implement half (if not more) of the ES functionality on your own. That more or less aligns with what we came up with. ES on its own doesn't have multitenancy capability. You have to invest a lot of manual effort to rewrite quite a lot of stuff which ES comes with (and maintain it afterwards). So while it can be done, it's no easy feat. I can understand why in @gcusello 's case some organizations would go that way - if they provided SOC as a service, the cost/benefit ratio of doing and maintaining such solution on their own could be justifiable. In many other cases the effort to do that on your own is simply too big. ... View more

Golden shovel for you, as well as for the one whose answer you answered 😉 The original thread is back from 2010 😄 Anyway, yes, that's true - current_only makes the UF subscribe to the eventlog channel upon the start of the service and listen only for the events which come during the forwarder's activity period. Obviously when you stop the service, you can't receive new events. If you set current_only to false, Splunk will read the backlog from the last saved checkpoint (or from the beginning if this is the first run). There is no option to read eventlog data only from a certain point in time (similar to ignore_older_than with monitor input). There was a thread not too long ago dealing with ingesting the events from the start but dropping "in transit" event older than a defined threshold but that's definitely not a standard solution. ... View more

I don't think there are any ready-made docs about all this. The default installation and configuration instructions for the Exchange TAs assume doing it "the blessed way" - installing UFs on the servers, ingesting the data locally with those UFs. If you want to change the architecture, it's up to you. I already gave you some hints but the resulting setup will depend on what kinds of data you want to ingest, which method of reaching for the data you choose and so on. As a side note, if you want to ingest eventlog data, there are generally four possible ways of doing so. In order of preference/feasibility/sensibleness/practicality. 1. Native ingestion with UF installed directly on the source host - this is the default recommended method of injecting windows eventlog data. 2. Using WEF to forward windows events to a collector node and ingesting events from there with a UF - this adds complexity to your environment, can be tricky to set up in a domainless environment, you can hit some performance limits (the same limits you can hit with local ingestion but it's much more likely with a WEF collector which receives data from several other nodes). 3. Using UF with WMI input which calls external server(s) and queries event logs there - this is awful to set up, WMI is not very efficient, requires "loosening" your security stance so that the UF node can reach other servers with WMI... Generally - don't use it unless you can't use either of the previous methods. 4. The absolutely worst idea - pull your events from eventlogs with a third party tool, and forward it another way to Splunk - this not only mixes different solutions needlessly in one environment and adds a lot of complexity and raises maintenance burden significantly, it also usually produces data in a format which is not easily parseable by Splunk (or even not reasonably parseable at all because the format is ambiguous). The only tool I've seen so far which looked as if it could produce logs which could be parsed with Splunk's standard TA-windows, was NX Log. But the docs stated that sending logs formatted like the original XML event was available only in premium paid version which I didn't have access to - the free one couldn't do that. So I'm listing this only as a remote possibility. It wouldn't solve your problem anyway because that also needs a piece of software, just another one, on the source server. So long story short - for windows eventlogs, your best bet would probably be to set up WEF in your environment. For scripts - that will be tricky. For files - you can try sharing the files over the network and accessing them "directly" from a remote UF but as I wrote before you can hit some issues. You're trying to do something which - while I can understand the reasons - is still a non-standard way and requires careful planning and tailoring the solution to your particular environment. That's what you typically pay your local experienced Splunk Partner to do. ... View more

OK. I would just blame it on the fact that @AceX is probably not a native English speaker (neither am I ;-)) so probably the statement turned out to sound a bit different than intended (at least I choose to interpret it that way). ... View more

@livehybrid It's not unusual to meet such requirements at customer's site. It's often not due to technical reasons but caused either by politics (interdepartamential tensions for example when one team is responsible for Splunk and another team manages the mail infrastructure) or by the fact that some solutions are maintained and supported by external company and introducing another element (in this case - the UF) to said solution/infrastructure might be grounds for said supplier trying to undermine the contractual obligations due to a component not provided/maintained/installed/etc. by them. So while I do agree that the UF directly on Exchange server would most probably be the best solution from the Splunk infrastructure point of view, it's sometimes simply not politicly viable. ... View more

It depends on what kind of data you want to pull from the Exchange host. As far as I remember, the TA for Exchange contains some scripted inputs - generally those you cannot run without installing the UF on the Exchange server (I suppose _maybe_ for some of them if you bent over backwards you could try to launch them remotely via something like Invoke-Command but it will require customisation of the existing add-on and further maintenance will be on you. If you want events from EventLog, you can use WEF (I've had some relatively successful installations running WEF and UF on the WEF collector; they might have some performance problems at some point but up to that point they run quite well). If you want contents of the log files... you can share them over the network but it can backfire. Firstly, the UF on the remote computer (which would access those events over the network) would have to run under a domain account and have permission to access those exported logs. That might be tricky if you normally don't run UF from a domain account but is a relatively normal task for a typical domain admin. The second problem, which might bite you in your behind at some point is that since Exchange produces a lot of logs (by default one log per hour if I remember correctly) and doesn't delete the old ones, the number of logs which Splunk has to keep track of grows quickly. And  checking status of remote logs has much much higher latency than local operations. So restarting such UF which monitors Exchange logs over the network might lead - in border cases - to several hours of pause in logs ingestion. I had a customer who precisely because of this problem copied files from exchange server to another server by means of a script and then UF would pick them up from that destination server. So while you can try to ingest the logs from Exchange without using the UF, the method with installing UF locally on the machine is the most convenient and best performing one. ... View more

OK. For starters, I don't actually know either solutions (the ZL Archive mentioned before or the Global Relay) - just googled what they do. But. The main question is not what "needs to be collected", because that is something that should stem from the business needs. The question might be what can be collected. Either you build your business case from the business need and verify if it can be achieved with the service provider (in other words, your business says - we want to be able to track this and that and you verify if either of those providers give you means of retrieving such data) or you go the other way around - check what data you are able to ingest from there and verify if there is any business case for that. A quick browse through their website doesn't yield very optimistic results. The only docs about any kind of API I could find was about sending data into their platform. I'm not sure if there is any way of either getting the data out of it or getting any form of automated query, retrieving audit records and so on. At least there's no easily available info on that. That might be a thing you need to contact their support about. ... View more

Yeah, but can you do _full_ multitenant? Like separate notable indexes for each tenant and so on? It does seem fairly tricky.  ... View more