Alternatively you could use Lookup Editor's API. It's not officially documented but there are examples of scripts using it on the web. ... View more

First things first - check status of your input (splunk show inputstatus, check logs). Maybe it's not outputs, maybe you're just not getting inputs (especially since we're talking about Security logs and access to those is sometimes limited). ... View more

You can find bits and pieces about kvstore in Splunk docs. It's "just" a mongodb instance so much of mongodb experience applies here as well. ... View more

You can use wildcards | stats dc(group) as groups values(*) as * by fname lname init   ... View more

rsyslog 8.2112 is over 4 years old. Unless you're just using some basic functionalities, you might want to upgrade to something more recent. ... View more

I'm not familiar with how the IIS logs are exported. If it is as you say - you can define manually which columns are exported and you're gonna keep it that way then yes, you could define the field order manually within props.conf and that would work OK. I suppose creators of the add-on didn't want to make assumptions and just did it so that it works whatever fields and in whatever order are exported. ... View more

For Windows data you have the TA_windows from Splunk. For some linux stuff you have the TA_nix. I think there was a separate addon for auditd. So there's no single add-on which will cover all your needs. You have to search on a per-case basis. ... View more

Wouldn't it be easier to do it with a lookup? ... View more

There's more to that than this. Sourcetype is the main way of telling Splunk what the particular "kind" of event is and what to do with it (including how to parse it into fields). Additionally ES usually expects most of relevant data to fit the CIM data models (have proper fields defined, sometimes have predefined values in those fields). So technically you could change the sourcetypes for everything you're ingesting. But in order to be able to effectively use this data with ES you'd need to define all those things that are typically done by add-on creators - parsing, normalizing, tagging. It's definitely _not_ worth the effort if you already have a working add-on which does that for you. ... View more

1. You're asking about Splunk Enterprise, not ES (which is Enterprise Security). 2. Well, without more info we can't know why your upgrade doesn't work. Maybe you skipped some versions before, maybe your kvstore database is corrupted. That's what the logs are for - see what's in them. ... View more

You're overthinking it. It seems that you need something like  | inputlookup users.csv | stats dc(group) as groups by fname lname init Whis will give you 1 in groups field if the person is only in GU and more than 1 if there are additional groups (assuming your assumption that each user _is_ a member of GU and might be a member of more groups). Additional remark - rethink your use case. In a sufficiently big organization (and in a small one as well if youre lucky) you'll hit collisions on first/last/middle name. ... View more

As far as I remember the IIS output logs, the problem with them is that they are CSV without a fixed column order. So you have to rely on the header row to extract the data in proper order. And that's possible only with indexed extractions. The other solution would be to create an external script reading the file, reformatting it and ingesting it as something else (json? k/v pairs?). That could work but it would mean that you weren't indexing the original event which might be important if you needed to use that as evidence - the event would already have been tampered with. ... View more

If we're talking about s2s (tcpout or httpout), Splunk uses output groups. In a simplest scenario (and I suppose with your environment sizing that't the case) your output group consists of just one output. [tcpout] defaultGroup=myIndexer [tcpout:myIndexer] server = 1.2.3.4:9997 That's the basic, most simple setup. (We're not digging into stuff like enabling/disabling TLS or useACK at this point). You might have more than one destination indexer within a single group [tcpout] defaultGroup=myIndexer [tcpout:myIndexer] server = 1.2.3.4:9997,2.3.4.5:9997 In this case your output traffic will be load-balanced between those two destination servers. But you can have more than one output group (in this case both groups have just one server each) [tcpout] defaultGroup=myIndexer1,myIndexer2 [tcpout:myIndexer1] server = 1.2.3.4:9997 [tcpuout:myIndexer2] server = 2.3.4.5:9997 With this setup each chunk of data will be enqueued for both output groups (each consisting of just one server inthis case but there could be more servers in each group for load-balancing). I explicitly say "enqueued" instead of "sent" because that's where we're getting into muddy waters of possible issues I mentioned earlier. With a single destination, you have a single queue so if something gets clogged your event processing just stops and that's it. Here it gets more complicated because each output has its own queue so you have to configure Splunk to either block the whole forwarding process if the data cannot be enqueued to the output or have to drop the events if an output cannot accept it. That results in some tricky scenarios where you either block the pipeline or lose data due to dropping. Actually, in some cases it could be easier to install two instances of an UF on one machine and send data independently from either of them. But while that can be relatively easy to achieve on a linux machine, with Windows it can be more complicated (I haven't done it myself on a windows box). ... View more

1. No, you can't do a "dynamic" search filter (like "use the current logged in user's username") 2. Do not use search filters with anything else than indexed fields using :: notation. Search filter with search-time fields is trivial to bypass. (left as excercise to the reader). ... View more

The answer is not that simple. Theoretically, you can have more than one output group and send to both environments. But. It gets problematic when you have an outage or connectivity problems with one of the receiver (groups). There are a lot of possible scenarios and issues regarding blocking the queues or dropping the events in such cases. BTW, if you send the same data to two separate environments you'd be consuming twice the license amount. ... View more

Ouch.  Your search is very heavy. 1. Exclusion is way slower than inclusion. Splunk has to read each event to decide whether it has the "test_cluster" term or not. 2. Those stats will most probably produce something different than you want. Especially that use of latest() after aggregation with values() will yield something very strange.  Not to mention that if you do values(*), you're not retaining the _time field so calling latest() makes no sense. 3. The mvexpand command is memory expensive so if you have a big result set it will eat up a lot of resources. 4. Running this (see my remark about mvexpand) over all time... that's gonna hurt. Badly. So while the overall idea of stats might be sound, with a big data set it might turn out to be more reasonable to do it in steps - summarize one part of your data, create a lookup with it. Apply the lookup to another part and so on. Yes, you'd have to do it asynchronously. 5. As a side note - I always advise to use fieldformat instead of creating a separate text field with time representation ... View more

This is not very probable that it's your case but I had such behaviour (and it was on HFs as in your case) when the KV-store was not fully upgraded. For some reason the HFs didn't fully migrate from the previous engine to wiredtiger. The files on the disk were still from the pre-wiredtiger mongo but the configuration suggested it was wiredtiger. As a result, the kvstore would start but after a very short time would die. And splunkd would start pumping up memory usage up to the point when OOM would kill it. Then the service would get automatically restarted and fun would start again. We had a "cluster" of 4 intermediate HF and they were restarting one at a time so we actually found out about it kinda by accident. But it was 9.0. I'm not sure if your 10.0.2 would even try to start not having fully migrated to wiredtiger. ... View more

OK. That suggests that your SEDCMD is not invoked at all (you could try something obviously right like SEDCMD-test = s/a/z/g to verify. If it indeed doesn't work, that means that either: 1) It's defined in a wrong stanza (a typical mistake here would be attaching it to a host value which is not provided at the beginning of the ingestion pipeline but rather comes from a rewrite in the middle of it). 2) it's set on a wrong component (either on a UF or a heavy component - indexer or HF - which is not the first one in event's path - like an indexer behind a HF). ... View more

AS @richgalloway already hinted - since you're creating a cluster anyway, there's really no significant difference between a single site cluster and a multisite cluster with just one site. So go for multisite cluster setup but define only one site for now. Then you'll be able to add another site later. It's a perfectly well supported setup (and even encouraged). ... View more

Unfortunately, you're trying to manipulate structured data with plain regexes. It might work if your jsons are always formatted the same way (which the jsons don't have to be since they rely on logical structure instead of strict formatting or fields order). If your SEDCMD doesn't work, you're probably not matching the part right with your regex. Maybe it's about other whitespaces than you expect. Or maybe there's another number of those whitespaces. Check your regex on your real data using https://regex101.com/ - it will show you if it matches or not and you'll be able to debug the regex and see where it fails if it does. ... View more

1. A golden shovel for you - it's a thread from 2018. 2. Did you verify if there are any entries in your $SPLUNK_HOME/passwd? 3. Did you restart your Splunk service? ... View more

This is a more than five years old thread so it's relatively highly probable that original participants are no longer active on Answers. But. The general approach to debug such thing would be to firstly verify the connectivity manually - do openssl s_client -connect tap-api-v2.proofpoint.com:443 and see the output. One of the most typical TLS isues is an TLS-intercept device which decrypts the traffic and re-encrypts it using a CA issued by your organization (which the app doesn't know). ... View more

Oh, and BTW, while I don't know the underlying problem you're trying to solve, often people say about two occurrences being within the same 5-minute bucket when they really mean that they want one thing happening not longer than 5 minutes after the other one. And those things are not the same. Binning events to 5 minute buckets means that you can have two events just a few seconds apart landing in different buckets. If you want to find the time difference you probably prefer streamstats with a time window instead of binning. ... View more

Well, it depends. Generally upgrade within the same minor version should not introduce any changes to index structure and even if Splunk introduces new index files format any possible changes are applicable only to newly created buckets so the index contents should be safe. Having said that, there is (a very remote, but still) a risk that something will suddenly misbehave in a new version and - for example - half of your buckets will erroneously get rollen to frozen. This generally "never" happens (I've personally never seen such case or heard about it) but as with any software more sophisticated than "hello world!" there is a possibility that something goes wrong. So if you want to be 100% (of course with the exception for the case where your backup was not done properly or your backup system failed) sure that you have a backup of a state you can roll back to - do a full backup. Stop the server and copy everything. This might get much more tricky in a clustered environment which you can't just stop and have a happy downtime. So you might instead back up  the configuration (etc directory and state except the index contents. And this is where it gets tricky because except for the obvious thing of backing up the kvstore contents (which was already mentioned in this thread) you might want to back up stuff which varies between components/apps - most typically checkpoints for inputs. The location of this stuff can vary and you might need to look for it. The safest bet here would be to back up $SPLUNK_HOME/var except for the index contents and possibly logs (who cares about the logs anyway ;-)). ... View more

Adding to what's already been said here, depending on your data and stats result, the "where" command logic might surprise you if you get multivalued fields. I'm not sure what your initial intent was but remember than when dealing with multivalued fields Splunk generally applies conditions to each field value separately when deciding whether an event as a whole matches. So you have to use different searches for each of those cases: - you want to exclude event for which _neither_ of values matches given value - you want to exlude event which consists solely of the given value - you want to remove a given value from multivalued field. - possibly something else. Also, I disagree here a bit with @gcusello - it's even better to use fieldformat than eval for formatting timestamps. ... View more