If you mean something that would give you a possibility to do something like index=auditlogs username~="johndoe" there's no way something like that would work efficiently. Yes, you could probably try to use some external command or something like that to that effect but it would be nowhere near Splunk's native search performance. ... View more

OK. Makes sense (a bit). Unfortunately regex-based extraction takes place before KV_MODE so you can't extract your values from already extracted field. Which is kinda unfortunate since you're at the mercy of the sending side - you have to account for possible changes within the event formatting even perfectly within the json specifications (possible whitespaces here and there). And you have to manually unescape the string. Ugh. BTW, does your props.conf entry have this "in customField" part? It shouldn't. As I wrote before - KV_MODE extractions take place after your regex-based extractions so your customField isn't defined yet when you're trying to extract your fields. You have to search through whole event (anchoring to the json field name). ... View more

Wait. Why would you even try to use regex to extract fields from json structure? That's what KV_MODE is for. Set it to json and you're good to go. ... View more

1. Use ruleset instead of transform. Transform won't fire on already parsed data (coming from HF, SH, DS...) 2. If you're using INGEST_EVAL you can use splunk_server value - you don't have to explicitly set the value. ... View more

I know that technically it's the same but I always struggle not to wince when I see latest(_time). max(_time) is the way to go 🙂 ... View more

And how have you determined that there is an _indexing_ delay? 1. Are the events available immediately for ingestion? (I've seen solutions which shared the events in batches effectively causing significant lags on some of them) 2. Are the inputs properly ingesting the events? Do they have any settings responsible for choosing proper events for ingestion? 3. Are the timestamps of the events correctly recognized? If the delay is constant and "round", that would suggest a timezone misconfiguration but that's just one of the possibilities. Way too little information to say anything reasonable here. ... View more

Whenever something about "vulnerabilities" pops up it triggers me to remind you that a "vulnerability" is not something absolute. Yes, I know that it's the easiest way to "manage vulnerabilities" to run your scanner once in a while, check what lights up red and call your admins to patch everything immediately but that's plain wrong. Everything works and runs in a context. Has anyone bothered to even read into the description of the vulnerability before jumping to removing the binary? ... View more

As far as I remember, those containers mount stuff like $SPLUNK_HOME/etc and $SPLUNK_HOME/var/lib from outside. But into that you'd have to dig more yourself. My experience with dockerized Splunk is relatively limited (to let's spin up quickly some empty instance and check if it can do X or Y). ... View more

There are several groups on Splunk Slack - #docker, #kubernetes and #splunk_operator_for_kubernetes (or something like this; easily findable). So you might get help there. Generally - containerized Splunk to be "really functional" has to: 1) Externalize config and state 2) Expose ports So if you can get those two elements done properly I don't see why shouldn't it work as SHC culster. Having said that - I"m not an expert on containerized Splunk and I generally dislike containerized tools. ... View more

Your problem is a bit vague. Remember that Splunk deals with data represented as events. So when you're talking about some external "systems" we have no idea what describes those systems or their state within Splunk. Do you have some input actively monitoring those systems? (kinda like an active nagios/zabbix/whatever probe) Or do those systems simply send their state as some kind of a keepalive messages? ... View more

OK. The issue with _any_ product which supports "extending" and SELinux is that you can either provide a policy covering the base product and make people add further extensions for more use cases (like access to specific files for log retrieval, spawning processes for modular inputs and so on) or make it run "relatively unconstrained". Unfortunately that's the deal with any similar permissions system. For example, RHEL/Rocky/Alma comes out of the box with SELinux policies for exim or dovecot but when I wanted to configure both of them to run with mysql as authentication source I needed to adjust the policies manually. You can't predict all possible use cases. ... View more

Well... to some extent it makes sense. RFC says that HTTP/1.1 client SHOULD include User-Agent header. So if it doesn't, it's either not a fully implemented HTTP/1.1 client or might have other features also not fully implemented (persistent connections are also a SHOULD requirement for HTTP/1.1 implementation). ... View more

CIFS client in Linux is... well, something else. It has had its share of problems "since always" - it can hang on you when the source server rebooted and stuff like that. Generally - network file systems have their fair share of possible issues but NFS4 seems more robust. And the recommended way of ingesting files is of course deployment of the UF to the source server. ... View more

Adding to @richgalloway 's remarks about your problem being incorrectly formulated (maybe you wanted something else but didn't word it properly), this is a very badly used join command. As a rule of thumb the join command is to be avoided whenever possible. Your search can be equally well rewritten without it. Oh, and if you limit yourself to just one index with tstats' where condition there's no point of adding index to the by clause. So effectively your initial search might be swapped around and rewritten as | tstats count where index="bbs-firewall" earliest=-24h by sourcetype | append    [ |  makeresults | eval sourcetype=split("BBCN-Kunshan,BSCN-Suzhou,BBSP-Malasiya,BTCN-Tianjin,BXCN-Xian,BCCN-Suzhouheadquarters,BCIT-Italy", ",") | mvexpand sourcetype | eval count=0 ] | stats sum(count) as count by sourcetype | ...   ... View more

OK. You've already gotten some hints here. Generally, a HF needs a license. You could use a forwarding-only license but then you'd use the DS functionality. So you could call out to your Splunk sales contact for a 0-bytes license. It's normally meant for Cloud deployments for exactly those scenarios - you need to have a DS and/or HF on premises whereas the "main" part of your Splunk infrastructure is the Splunk Cloud service. True, your case is a bit different but boils down to the same thing - the need to be able to use a limited subset of functionalities in a site without actually doing any indexing/searching work in that site. You might be able to get a "blessing" for using the same production license in both sites as long as there is absolutely no indexing going on in the B site. But for that you need to talk with your local Splunk sales team. Generally - HF/DS functionalities need a license and somehow you must provide that host with one. And generally it's best to talk with your local sales team how best to tackle the issue. I'm still perplexed however why would normal event stream connectivity be allowed but LM connectivity - not. ... View more

Splunk Search Head clusters use RAFT algorithm to keep the state of the cluster - https://en.wikipedia.org/wiki/Raft_(algorithm) If you want to dig deeper into that - there are quite a lot of materials about it on the internet. Anyway, as others already pointed out, to choose a captain, a quorum of floor(N/2)+1 members is needed. So in a split-brain scenario with even number of member split in the middle, there is no way of choosing the captain. That's the "most common" part of the answer. But there are other factors. Typically if we're talking about the number of member nodes and possible split brain scenario there is a silent assumption that those nodes are somehow (usually evenly) distributed into more than one site - that's when the split brain scenario is most probable. The less common part of the answer is that there are some ways of dealing with this limitation: 1) You can designate the captain manually. 2) You can have a "voting only" SHC member - a small machine not being an "active" part of a cluster (not getting any searches to run) making sure that you have a quorum. The general rule of thumb that you should have odd number of nodes in your SHC is just so that the cluster can handle whole site outages without any need for manual intervention (which still might not be 100% true because if you spread your users' traffic to the nodes only in local site, you still might end up with some issues depending on how your LBs are configured). ... View more

What do you know about Splunk Enterprise? What have you tried so far? What do you want to achieve with your integration? How is it supposed to work? ... View more

Ok but that's another thing. One is whether data is gonna be indexed locally or sent to another instance. Another thing is what index it's destined for regardless of whether locally or remote. ... View more

@isoutamo This is a completely different thing. Your article is dealing with syslog output, not tcp one. And it doesn't say that the events must be formatted in any way, just that syslog outputs are treated differently than tcpout ones. @vikasg I'd try sniffing the network traffic. When I tried sending events from Splunk uncooked to rsyslog receiver, said rsyslog started complaning heavily about data format. Maybe your syslog-ng just discards the events treating them as "broken"? ... View more

Is this your only output? Are you sure your input is working properly and the issue is with the output? ... View more

Well done debugging the issue! Interesting finding. ... View more

OK. This configuration snippet has nothing to do with forwarding events anywhere so there must be more to this. Anyway, rsyslog works like this - unless you have more rulesets defined, it just processes the ruleset bound to your input (by default it's the main ruleset which is implicitly defined from entries in your config not defined within other explicitly defined rulesets. So if you don't have any other rulesets it just goes through the effective config  from top to bottom and executes the configuration directives. So if you don't explicitly stop the processing for some events (as you do in your snippet for events with specific $fromhost-ip values), processing goes further down the config list. So even if you have another action defined somewhere earlier (like omhttp or omfwd), if you don't explicitly stop processing for the event matching that rule, it will be processed further until it reaches those rules you pasted. ... View more

Two things. 1. Ampersand must be on a new line. It means "match the selector from the previous rule" 2. While that might technically work, it's not recommended to mix different config styles. The ampersand notation is the legacy format while the explicit if statement is RainerScript. The legacy configuration format while still working (rsyslog team goes to great lengths to make rsyslog as backward compatible as possible) is deprecated. For simple rules you might use the traditional sysklogd selector-destination format, for more complicated RainerScript should be used. So in RainerScript it would look something like this: if $fromhost-ip == "1.2.3.4" then {   *.*  /var/log/remote/1.2.3.4.log   stop } While specifying just the destination without the selector might work, it's more explicit this way. And since we're using RainerScript, I'd go for explicit if $fromhost-ip == "1.2.3.4" then {    action(type="omfile" File="/var/log/remote/1.2.3.4.log")    stop } To go even further you could define a filename template and use the source IP in the file name dynamically but that's beyond the scope of this forum. It's something to either look up in the rsyslog docs (which are pretty exhaustive) or ask on rsyslog mailing list to which I pointed earlier. ... View more

Well, a sourcetype can also be defined within an app to which access might differ between roles. As a side remark - "data.workstationId=*" is a releatively performance-hungry condition. If you can narrow down your events by specifying the field name (simply adding "workstationId" on its own) as search term - do it. (of course if 95% of your events contain this field it won't help much but if it's just 10%, it will give you a significant savings on search time). ... View more

Your requiremets are mutually exclusive. "We want to be able to search Production logs from the DRC site" and "There is no direct network communication between the Production and DRC environments". Yes, I know that "we can automate the file movement through an intermediate staging process" but this will not be searching production. You'd be searching a copy. The way to go about it would be probably to set up a periodic process on the production site which would export closed (not hot) buckets to some "staging" area from which another process would pick them up and pull into the DRC site. It would be messy, it would need quite a lot of bubble gum and prayer to work correctly (you'd need to keep track of which buckets have already been copier, which have not; if you have clusters, it gets even more complicated on both sides) but could work. Splunk on its own doesn't have built-in functionality doing this. ... View more