Well, we can't say retroactively what happened for sure. Syslog, especially UDP-transmitted one is sensitive to both network disruptions as well as receiver's performance. If the receiving Splunk infrastructure listens for syslog directly with splunkd process, without external syslog daemons, that might have caused the receiver to be "overwhelmed" with a burst of data from other hosts and might have caused it to not process the incoming syslog data properly. Performance is one of the reasons why in production environment you generally shouldn't listen for syslog data directly with Splunk process. You should use an external syslog daemon. See https://docs.splunk.com/Documentation/SVA/current/Architectures/Syslog for possible syslog ingestion architectures. ... View more

What puzzles me here is why are you trying to split the data on the receiving end when you have control over the sending solution and it would be way easier and more maintainable to simply split the array at the source and send each array member as separate event? Also remember then when sending to /event endpoint you're bypassing timestamp recognition unless you append that parameter which I always forget to the URI so you should send an explicit epoch-based timestamp along with your event. The upside to this is that you don't have to worry about date parsing in Splunk then. ... View more

Actually, assuming that you're managing your configs with a deployment server, it might prove easier and less error-prone to uninstall old UF version and simply deploy a fresh install of a new UF version and attach it to the DS. But this might cause duplication of ingested data if the state files (mostly fishbucket but also state of eventlog inputs or wmi inputs) are not handled properly. ... View more

There is too little information to even blindly guess. Firstly, how are those events getting into your Splunk infrastructure? Do you have a UF installed on remote hosts and monitor file inputs deifned on them? Or maybe those are eventlog inputs? Or are you receiving syslog data over the network? Directly or using a third party syslog daemon? Secondly, how did you verify that the data for those "outages" isn't ingested at all? Maybe the sources (or receivers) are getting clogged so your ingestion process stops for a while but then resumes and catches up but your data onboarding is incomplete so you don't have reliable timestamps? There are many things that can go wrong. ... View more

This is more of a question to ClamAV authors/database maintaners. I'd hazard a guess that the file contained within SE has some characteristic pieces of the Gitpaste method as part of its searches. And ClamAV detects their presence and flags the file. But I'd double-check it with ClamAV folks. ... View more

As it was already stated - the OS differences shouldn't be that problematic from the technical point of view. It can add some maintenance overhead because you have to maintain different package types, maybe different service launch methods, but generally, the software should work. It will be unsupported though. But the main issue with such setup is that you'll have a relatively spread out cluster. Here your main issue will probably be latency across your envionment. https://docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware#:~:text=A%20Splunk%20environment%20with%20search,should%20not%20exceed%20100%20milliseconds. ... View more

Remember that /raw endpoint accepts just raw data whereas /event endpoint requires specific format which it then "unpacks". So just posting the same data to /raw endpoint will result in differently represented events. ... View more

I'm not asking whether the right events are selected. I'm asking whether the fields are extracted. If you do index=finder_db AND (host="host1" OR host="host2") AND (("Wonder Exist here") OR ("Message=Limit the occurrence" AND "FinderField=ZEOUS")) | table uniqueId FinderField Message Is your table populated with field values or are they empty? ... View more

OK. Let's start at the start 🙂 index=finder_db AND (host="host1" OR host="host2") AND (("Wonder Exist here") OR ("Message=Limit the occurrence" AND "FinderField=ZEOUS")) This will select the events for further processing. But the question is whether you're extracting any fields from those events. Before we're going anywhere further, we need to know whether: 1) The uniqueId field (to which you're referring in subsequent posts in a case-inconsistent manner) is extracted. 2) The "data" field(s) which you want to "merge" are extracted. Generally, the field extraction should be (actually, should already have been) handled at data onboarding stage. When you have this one covered, you can get to the second part - handling the logic behind "joining" your events. ... View more

I assumed your fields are already extracted. After some thought, actually the stats doesn't add anything here. It should be enough to just do the xyseries. As long as you have fields properly extracted. ... View more

I'm not sure what "columns" you want from this data but assuming that you want to have a table with various messages per id and status you might want something like <your initial search> | stats values(Message) as Message by UniqueId Status | xyseries UniqueId Status Message ... View more

No. They are not being broken. The SEDCMDs are being applied and apparently it removes part of the event data so the remaining data sometimes happens to be valid JSON and sometimes isn't. But it has nothing to do with event breaking. ... View more

If I remember correctly, the alert created "normally" with a "create notable" action should work as well (although it will have less configuration options). But in order to be able to create notables the user who the alert is run with must have proper privileges within the ES app. So if an alert was created with a normal alert creation means instead of ES Content Management, the user might not be able to create notable due to insufficient permissions. (yes, I'm aware that it sounds a bit convoluted). ... View more

The CLONE_SOURCETYPE option in a transform causes Splunk to create a copy (at this moment of the ingestion pipeline, so all the state of the event at this point is retained) of the processed event, changes its sourcetype to the one specified in the CLONE_SOURCETYPE option and reingests the event back at the (almost) beginning of the pipeline (skipping the initial phases of line breaking and time recognition). See the usual https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platform-the-Masa/m-p/590774 CLONE_SOURCETYPE is called from a transform in the typing phase. The original event is processed without changes as if the transform containing the CLONE_SOURCETYPE option wasn't there. But the copy is moved back to the typing queue and starts the whole typing phase with a new sourcetype and triggers completely new set of transforms according to the new sourcetype (and possibly new source and host if they were overwritten during the initial transforms run).   ... View more

Apparently you have problems with proper extraction of the message field. So you should verify your data onboarding and start with fixing the extractions. ... View more

It's hard to say what's "wrong" not knowing your data but while transaction can be sometimes useful (in some strange use cases) it's often easier, and faster to simply use stats. Mostly because transaction has loads of limitations that stats don't have. Quick glance at your search suggests that for some reason the message field is not extracted properly from your event so you're not getting two separate values in your multivalued message output field. As I said I'd go with index=... ("SENDER[" OR ("RECEIVER[" AND "POST /my-end-point*")) | rex "\[(?<id>\d+)\]" | eval request=if(searchmatch("SENDER[",message,null()) | eval response=if(searchmatch("\"RECEIVER[\" AND \"POST /my-end-point*\"",message,null()) | stats range(_time) as duration, count, values(request) as request, values(response) as response, values(_raw) as _raw by id   ... View more

Ok, a word of advise - it's usually better to specify indexes explicitly than to have them as searched by default. Especially with an admkn role! It spares you unnecessary load on your environment for searches in which you haven't specified the indexes and it saves you a lot of debugging when you have different roles with different default indexes and people report mismatch in searches functionality. You have been warned. One additional hint - it's way better to do a quick check with | tstats count where index=rapid7 by sourcetype than index=rapid7 | stats count by sourcetype The first one only checks the summarized indexed fields while yours needs to plow through all events from the index. And there is something that doesn't add up. On Cloud you cannot have the admin user role. You can only have sc_admin (which is a limited admin role). So if you're trying to edit the admin role you shouldn't be able to do so.   ... View more

Firstly, let me start by stating the obvious - vulnerability scanners are notorious for being way overly trigger-happy with their findings. It takes an experienced person to filter their results and get the actual reasonable results. Having said that - those processes are spawned by the splunkd process (not directly -  via compsup daemon). So that finding is at least questionable if not simply a false positive. ... View more

What exactly do you want to do? The command you provided will "empty" the index without touching its definition. Also, I haven't tried this in a cluster (I assume that's what you mean by 3 indexers and "a management node") but I'd expect the cluster to start fixups as soon as you do the operation on the first node unless you enable maintenance mode. Anyway, if you want to leave the index definition but only remove the indexed events, that's one of the possibilities. Another one is to set very short retention period and let Splunk roll the buckets normally. If you want to remove the index along with its definition, you have to remove it from indexes.conf on the CM, push the config bundle (this will trigger rolling restart of indexers) and then manually remove index directories from each indexer. ... View more

Wait. As far as I remember (it's been some time since I did it last time) you don't manually copy anything. When you run the installer in deployer mode it takes care of preparing the shcluster bundle. That's why you run it exactly as described - upload the app to the deployer, run the installer on the deployer, apply shcluster-bundle. No manual copying stuff anywhere. ... View more

That's more or less what I was talking about 🙂 ... View more

I don't recall ever seeing dbconnect configured so that it sends to a HEC input outside of the HF it's running on. Theoretically it's possible - see https://docs.splunk.com/Documentation/DBX/3.18.2/DeployDBX/settingsconfspec but I must say I've never seen it configured this way. Anyway, first check your config., then debug apropriate HEC inputs. ... View more

Be also aware of the subsearch limitations. You can't run them for long and they have a limit for returned results. So if you run them over a big data set you might get incomplete results and since the subsearches will get silently finalized you won't even know about it. So i  your case it would probably be better to search for all matching events initially and tag them according to matching specific set of conditions using conditional field assignment ( | eval something=if( [...] )) ... View more

I don't understand your objections vs. methods 1 and 2. Complexity of the json structure shouldn't matter as long as the event is a valid json and - in case 2 - doesn't exceed maximum number of fields handled by auto-kv. ... View more

If you do tstats by time without binning and then do bin, you'll have to stats again to summarise your data. Bin on its own doesn't aggregate data, just aligns the field into discrete points. ... View more