As far as I remember, those containers mount stuff like $SPLUNK_HOME/etc and $SPLUNK_HOME/var/lib from outside. But into that you'd have to dig more yourself. My experience with dockerized Splunk is relatively limited (to let's spin up quickly some empty instance and check if it can do X or Y). ... View more

There are several groups on Splunk Slack - #docker, #kubernetes and #splunk_operator_for_kubernetes (or something like this; easily findable). So you might get help there. Generally - containerized Splunk to be "really functional" has to: 1) Externalize config and state 2) Expose ports So if you can get those two elements done properly I don't see why shouldn't it work as SHC culster. Having said that - I"m not an expert on containerized Splunk and I generally dislike containerized tools. ... View more

Your problem is a bit vague. Remember that Splunk deals with data represented as events. So when you're talking about some external "systems" we have no idea what describes those systems or their state within Splunk. Do you have some input actively monitoring those systems? (kinda like an active nagios/zabbix/whatever probe) Or do those systems simply send their state as some kind of a keepalive messages? ... View more

OK. The issue with _any_ product which supports "extending" and SELinux is that you can either provide a policy covering the base product and make people add further extensions for more use cases (like access to specific files for log retrieval, spawning processes for modular inputs and so on) or make it run "relatively unconstrained". Unfortunately that's the deal with any similar permissions system. For example, RHEL/Rocky/Alma comes out of the box with SELinux policies for exim or dovecot but when I wanted to configure both of them to run with mysql as authentication source I needed to adjust the policies manually. You can't predict all possible use cases. ... View more

Well... to some extent it makes sense. RFC says that HTTP/1.1 client SHOULD include User-Agent header. So if it doesn't, it's either not a fully implemented HTTP/1.1 client or might have other features also not fully implemented (persistent connections are also a SHOULD requirement for HTTP/1.1 implementation). ... View more

CIFS client in Linux is... well, something else. It has had its share of problems "since always" - it can hang on you when the source server rebooted and stuff like that. Generally - network file systems have their fair share of possible issues but NFS4 seems more robust. And the recommended way of ingesting files is of course deployment of the UF to the source server. ... View more

Adding to @richgalloway 's remarks about your problem being incorrectly formulated (maybe you wanted something else but didn't word it properly), this is a very badly used join command. As a rule of thumb the join command is to be avoided whenever possible. Your search can be equally well rewritten without it. Oh, and if you limit yourself to just one index with tstats' where condition there's no point of adding index to the by clause. So effectively your initial search might be swapped around and rewritten as | tstats count where index="bbs-firewall" earliest=-24h by sourcetype | append    [ |  makeresults | eval sourcetype=split("BBCN-Kunshan,BSCN-Suzhou,BBSP-Malasiya,BTCN-Tianjin,BXCN-Xian,BCCN-Suzhouheadquarters,BCIT-Italy", ",") | mvexpand sourcetype | eval count=0 ] | stats sum(count) as count by sourcetype | ...   ... View more

OK. You've already gotten some hints here. Generally, a HF needs a license. You could use a forwarding-only license but then you'd use the DS functionality. So you could call out to your Splunk sales contact for a 0-bytes license. It's normally meant for Cloud deployments for exactly those scenarios - you need to have a DS and/or HF on premises whereas the "main" part of your Splunk infrastructure is the Splunk Cloud service. True, your case is a bit different but boils down to the same thing - the need to be able to use a limited subset of functionalities in a site without actually doing any indexing/searching work in that site. You might be able to get a "blessing" for using the same production license in both sites as long as there is absolutely no indexing going on in the B site. But for that you need to talk with your local Splunk sales team. Generally - HF/DS functionalities need a license and somehow you must provide that host with one. And generally it's best to talk with your local sales team how best to tackle the issue. I'm still perplexed however why would normal event stream connectivity be allowed but LM connectivity - not. ... View more

Splunk Search Head clusters use RAFT algorithm to keep the state of the cluster - https://en.wikipedia.org/wiki/Raft_(algorithm) If you want to dig deeper into that - there are quite a lot of materials about it on the internet. Anyway, as others already pointed out, to choose a captain, a quorum of floor(N/2)+1 members is needed. So in a split-brain scenario with even number of member split in the middle, there is no way of choosing the captain. That's the "most common" part of the answer. But there are other factors. Typically if we're talking about the number of member nodes and possible split brain scenario there is a silent assumption that those nodes are somehow (usually evenly) distributed into more than one site - that's when the split brain scenario is most probable. The less common part of the answer is that there are some ways of dealing with this limitation: 1) You can designate the captain manually. 2) You can have a "voting only" SHC member - a small machine not being an "active" part of a cluster (not getting any searches to run) making sure that you have a quorum. The general rule of thumb that you should have odd number of nodes in your SHC is just so that the cluster can handle whole site outages without any need for manual intervention (which still might not be 100% true because if you spread your users' traffic to the nodes only in local site, you still might end up with some issues depending on how your LBs are configured). ... View more

What do you know about Splunk Enterprise? What have you tried so far? What do you want to achieve with your integration? How is it supposed to work? ... View more

Ok but that's another thing. One is whether data is gonna be indexed locally or sent to another instance. Another thing is what index it's destined for regardless of whether locally or remote. ... View more

@isoutamo This is a completely different thing. Your article is dealing with syslog output, not tcp one. And it doesn't say that the events must be formatted in any way, just that syslog outputs are treated differently than tcpout ones. @vikasg I'd try sniffing the network traffic. When I tried sending events from Splunk uncooked to rsyslog receiver, said rsyslog started complaning heavily about data format. Maybe your syslog-ng just discards the events treating them as "broken"? ... View more

Is this your only output? Are you sure your input is working properly and the issue is with the output? ... View more

Well done debugging the issue! Interesting finding. ... View more

OK. This configuration snippet has nothing to do with forwarding events anywhere so there must be more to this. Anyway, rsyslog works like this - unless you have more rulesets defined, it just processes the ruleset bound to your input (by default it's the main ruleset which is implicitly defined from entries in your config not defined within other explicitly defined rulesets. So if you don't have any other rulesets it just goes through the effective config  from top to bottom and executes the configuration directives. So if you don't explicitly stop the processing for some events (as you do in your snippet for events with specific $fromhost-ip values), processing goes further down the config list. So even if you have another action defined somewhere earlier (like omhttp or omfwd), if you don't explicitly stop processing for the event matching that rule, it will be processed further until it reaches those rules you pasted. ... View more

Two things. 1. Ampersand must be on a new line. It means "match the selector from the previous rule" 2. While that might technically work, it's not recommended to mix different config styles. The ampersand notation is the legacy format while the explicit if statement is RainerScript. The legacy configuration format while still working (rsyslog team goes to great lengths to make rsyslog as backward compatible as possible) is deprecated. For simple rules you might use the traditional sysklogd selector-destination format, for more complicated RainerScript should be used. So in RainerScript it would look something like this: if $fromhost-ip == "1.2.3.4" then {   *.*  /var/log/remote/1.2.3.4.log   stop } While specifying just the destination without the selector might work, it's more explicit this way. And since we're using RainerScript, I'd go for explicit if $fromhost-ip == "1.2.3.4" then {    action(type="omfile" File="/var/log/remote/1.2.3.4.log")    stop } To go even further you could define a filename template and use the source IP in the file name dynamically but that's beyond the scope of this forum. It's something to either look up in the rsyslog docs (which are pretty exhaustive) or ask on rsyslog mailing list to which I pointed earlier. ... View more

Well, a sourcetype can also be defined within an app to which access might differ between roles. As a side remark - "data.workstationId=*" is a releatively performance-hungry condition. If you can narrow down your events by specifying the field name (simply adding "workstationId" on its own) as search term - do it. (of course if 95% of your events contain this field it won't help much but if it's just 10%, it will give you a significant savings on search time). ... View more

Your requiremets are mutually exclusive. "We want to be able to search Production logs from the DRC site" and "There is no direct network communication between the Production and DRC environments". Yes, I know that "we can automate the file movement through an intermediate staging process" but this will not be searching production. You'd be searching a copy. The way to go about it would be probably to set up a periodic process on the production site which would export closed (not hot) buckets to some "staging" area from which another process would pick them up and pull into the DRC site. It would be messy, it would need quite a lot of bubble gum and prayer to work correctly (you'd need to keep track of which buckets have already been copier, which have not; if you have clusters, it gets even more complicated on both sides) but could work. Splunk on its own doesn't have built-in functionality doing this. ... View more

We don't know what powers those charts but the differences in results when a search is run as users with different roles usually boils down to: 1) Difference in index access permissions (remember that roles can also have search filters) 2) Difference in access to apps in which KOs are defined or even specific KOs. Also some users have private KOs which can affect what is being extracted/calculated and so on. And sometimes the search behaves differently (has access to different KOs) depending on which app it's being run in. So there are several possible points where the behaviour could differ. I'd start with cutting the search to the very initial part (before first pipe) and comparing: 1) Number of results 2) Extracted fields.   ... View more

Any Splunk instance you're working on knows only the indexes defined in indexes.conf on that particular machine. Defining indexes across your whole environment (or at least SH and HF layers) even if nothing is being indexed directly on those hosts gives you the possibility to use auto-completion (on SHs) and choose indexes from drop-downs (on HFs). That's why you might want to keep your indexes.conf synchronized across your whole setup. ... View more

1. Hard to say without knowing rsyslog config 2. It's not a Splunk problem. It's a topic more suitable for rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog ... View more

Well... neither solution is really compatible with Splunk since Splunk has no way of expressing the null value. An empty string is just a string which happens to be of zero length. So it's not a null. A null value for Splunk means that the field is not present, which is not the same as json field with a null content. It's just one more quirk which happens when Splunk tries to handle jsons - since it doesn't work with structured data it does a lot of ducttaping and handwaving to make _something_ happen but as we know there are at least three separate and differently behaving json extraction modes (indexed extractions, kvmode and spath), not including json eval functions so it's just another one of those things... ... View more

Wait. 1. Indexed extractions have nothing to do with spath command. And as I understand the question, only parts of the original event form json structures, not the whole raw event. In this case indexed extractions won't work anyway. 2. As a rule of thumb, indexed extractions shouldn't be used unless there is no other way. ... View more

You're mixing requirements for different products and services (Splunk Enterprise/Cloud, o11y, RUM, ITSI, you name it...). This is not a topic for community. This is something you need to engage your local Splunk Partner for. ... View more

1. 1MByte events are... huge. Whether it is kv-json or plain regex-based extractions, it's gonna be heavy. 2. As a side note - if splunkd brings down whole OS it might be the time to tweak the VMM parameters. (swappiness, zram, oom killer priorities...) ... View more