1. Is it a fresh installation or an upgrade? 2. You have the immediate debugging steps on screen. ... View more

Could be. I didn't copy-paste it but written here by hand so there might have been a typo. ... View more

OK. It seems that even with "where" Splunk optimizes this search and it turns into index=whatever source=CASE("stderr") "stderr" Which obviously again searches for the source as indexed field only. (same goes  You can make it work if you disable optimizations index=whatever stderr | noop search_optimization=false | where source="stderr"   ... View more

That is intriguing because I was pretty sure it would work. I tried to recreate your case locally with makeresults | collect and it indeed doesn't find it with where. I'll keep digging. ... View more

If it works, it works. 🙂 It's worth noting though that it's a rather "heavy" way of doing it. Spath is a fairly intensive command and you're doing it over all your events. I suppose for a one-off ad-hoc search it might be OK but if you do it often, you might want to optimize it a bit. ... View more

I think the easiest way to verify whether that field is indexed (there might be some additional index-time extraction, apart from simple indexed-extraction configuration for the whole event; yes, I know it's confusing ;-)) is to try to search for index=your_windows_index EventID::4624 The important thing is that you're not looking for  EventID=4624 but for  EventID::4624 If you get any results that will mean that this field is indeed indexed and you have to search where it's extracted in index time. ... View more

Are you sure you don't have indexed extractions enabled by any chance? Since automatic KV extractions happen after manual extractions the EventID field should not be populated when you're hitting the transforms so the first transform (EventID_as_EventCode) should _not_ set the field to any value. ... View more

As far as I remember, the license consumption for Cloud in the ingest-based option is the same as on-prem one which means the event is measured by its _raw part just prior to indexing. This means that: 1) However you modify your event prior to indexing it in terms of the raw event contents (like cutting out some headers or unnecessary trailing parts) will affect your license usage 2) Indexed fields which are saved in the tsidx files but are not "exploding" your _raw event contents do not affect your license usage. Having said that - indexed extractions are very rarely the way to go but not for license-related reasons. ... View more

@livehybrid Let me interject here 😉 If it was just because it resulted in a multivalued field, you could happily search for just one of those values. But in case of this particular field (as well as other indexed fields which are not (supposed to be) present in the raw data) it's a bit different. When you do index=something source=aaa and check the job log you'll get 07-01-2025 10:47:22.082 INFO UnifiedSearch [3225984 searchOrchestrator] - Expanded index search = (index=something source=aaa) 07-01-2025 10:47:22.082 INFO UnifiedSearch [3225984 searchOrchestrator] - base lispy: [ AND index::something source::aaa ] This means that Splunk will only look for those events which have metadata fields of index and source with given values. In case of index it's not really a field but in case of source, it's gonna be a search only for indexed terms in form of source::something. Splunk will not try to bother with parsing anything out of the event itself. It's in the later part of the processing pipeline that the field might get parsed out and then be used for further manipulation. The problem with the obvious approach index=something | search source=something_else is that Splunk's optimizer will turn this seemingly superfluous search command back into index=something source=something_else which will end up with what I've already shown. That's why I used the where command - it works differently and won't get optimized out. Of course narrowing the search only to the events containing the value of "stderr" will speed the search (but won't be very effecitve if the "stderr" term appears in other terms of the event; tough luck). I'm not quite sure though if TERM() makes any difference here. I'm pretty sure just searching for "stderr" itself would suffice and it doesn't make the resulting SPL look too cryptic 😉 ... View more

Source is one of the default metadata fields which are supposed to be indexed along the event, not included in the event. Therefore the initial search does not look for the fields parsed out from the event itself when looking for fields like source or sourcetype. As a walkaround I'd try to instead of <rest of your search> source=stderr do <rest of your search> stderr | where source="stderr" ... View more

As @livehybrid mentioned, the general idea behind UFs and their management is the other way around - the deployment server distributes config items which are pulled and applied by the forwarders. This way you should not backup your confogs centrally from your forwarders but the completely opposite - have the config centrally distributed. This way if your forwarder fails or when you need to reinstall it you can simply push the same piece of config again. I can understand that you might want to back up state of the forwarder (not the configs) so that if your forwarder breaks and you reinstall it you don't have to ingest all sources from scratch. But this is a fairly border case and there is no built-in mechanism for this. And there is definitely no built-in way to send the config or state of the forwarder to either the indexer(s) or the Deployment Server. Since you can run almost anything as a scripted input you could try to do your own duct-tape bound solution which would gather state files from the forwarder (not sure about reading open files though and consistency of their contents) and - for example - compress such archive and base64 it so that it can be indexed as a text event but that's a horrible idea.   ... View more

The recommended way of receiving Netscaler events is by Splunk's own addon https://docs.splunk.com/Documentation/AddOns/released/CitrixNetScaler/About You might want to try to use the same sourcetype but I have no idea what the format will be if you do it according to Netscaler's docs. ... View more

You are thinking about this "backwards". These fields aren't "added" to your events in the sense that they are not stored anywhere additionally and taking up space and license. They are dynamically extracted and/or calculated when you are searching your data. If you are searching in verbose or smart mode, all possible fields will be extracted (unless you explicitly limit them with the "fields" or "table" command. But if you search in fast mode (which is the prefered approach because then searches run faster) only the fields you explicitly specify will be extracted. So you don't need to "remove" the fields. If you don't want to see them when searching in smart or verbose mode you can add | fields - whatever_fields you want_removed And you won't see them. @malix_la_harpe 's answer while techinically not wrong is not the way to go. ... View more

Has it ever worked and now suddenly stopped? (If so - what changes were made to your environment?) Or is it a new installation? (How exactly did you install it?) Do the logs show any related errors? ... View more

The github project seems kinda old. Very old. As far as I remember the modern UF runs... fairly well with SELinux but needs tweaking in order to grant access to specific items. So the audit2allow approach is a fairly proper one. ... View more

That's one of the limitations of ingesting windows events in the "traditional" form. Open Event Viewer on your windows computer. Open the Security log and find a 4624 event. What you're ingesting at this point is what you can see in the bottom panel in the "General" tab - the event rendered to a human-readable text. It does contain fields named the same way (like Account Name) just differently "scoped" (indented a bit in sections regarding either Subject or New Logon).  So Splunk parses those fields as key/value pairs and simply gathers two different values of the same named field because the source data contains it. You could probably bend over backwards and try to write custom regexes to extract those specific values but it will be very ugly and fairly bad performance-wise. If you switch to ingesting XML versions of events, apart from saving on space occupied by events (and license usage!), you get more unambiguous structure. You'd be ingesting the event as it's presented in the bottom Event Log panel in the Details tab in XML view. The structure might not be as readable here but Splunk can parse this XML much better and present it to you in a useful form. And here you have much more straightforward and unique field names - in your case it would be SubjectUserName and TargetUserName - two completely distinct fields. ... View more

Yes. The range of interoperability between UFs and receiving components (intermediate forwarders/indexers) is quite big. Even if the official documentation doesn't list something as supported, things might just work. I've had UFs as old as 6.6 sending to version 9 indexers and it ran OK. There might be a minor issue with v9 UFs sending to older indexers because new UFs generate config change events which are supposed to go to indexes not present on older Splunk instances. The temporary walkaround for this is to disable the config tracker inputs on the UFs until the indexers are upgraded to v9. But even if you don't do that, they will generally work, it's just that those events will either land in your last chance index or will generate a warning about non-existent index and get dropped completely. ... View more

4. That was the most obvious example. There might be some other dependencies - for example, if you're using dbconnect, you require JRE. 5. Yes, chowning should take care of it. But as I understood from your earlier comments, you have your index volume(s) outside /opt/splunk. You need to take care of its ownership as well.   ... View more

This is... bad, Firstly, it seems that it's data already received by something else, embedded in another format and sent to Splunk. Then secondly, these are completely different sourcetypes. So if you absolutely cannot separate them earlier, you should overwrite sourcetype on ingestion so that each of those types is parsed differently. ... View more

And, to add to already provided answers, there is no such thing as syslog meaning a strictly defined protocol. Syslog can mean many different things depending on context and it's definitely not limited to 514 port. It's a perfectly normal situation when "syslog" data is sent to another port. ... View more

Probably the easiest thing would be to switch from "traditional" format to XML logging. Now you're gathering a plain-text representation of windows events. This format has some limitations (one you've just hit) and on average consumes more license than the XML-formatted events. So you might simply want to switch to renderXml=true sourcetype=XmlWinEventLog in your eventlog inputs. Provided that you're using TA-windows, the rest should happen automagically. This will of course only affect events ingested after the change. ... View more

Just wtiting "solved" is not very helpful to others. Please provide us with more details - what was the cause of the problem and how you fixed it so that others can benefit from your experience in the future. Remember that Answers is about colaboration - sometimes others help you, sometimes you help them. Everybody wins. ... View more

The general idea is OK but there are details which can pop up unexpectedly here and there. 1. I assume (never used it myself) that Amazon Linux is also an RPM-based distro and you'll be installing Splunk the same way it was installed before. 2. Remember to shut down Splunk service before moving the data. And of course don't start the new instance before you copy the data. 3. I'm not sure why you want to snapshot the volumes. For backup in case you need to roll back? 4. You might have other dependencies lying around, not included in $SPLUNK_HOME - for example certificates. 5. If you move whole filesystems between server instances the UIDs and GIDs might not match and you might need to fix your accesses. Oh, and most importantly - I didn't notice that at first - DON'T UPGRADE AND MOVE AT THE SAME TIME! Either upgrade and then do the move to the same version on a new server or move to the same 8.x you have now and then upgrade on the new server. ... View more

Firstly, a golden shovel. This is a very very old thread. Secondly, you are mistaken. While the event will not get "immediately deleted" but for a completely different reason. There are several factors here: - events are not handled on their own but by buckets - hot buckets do not roll to frozen directly - "unusual" events (too far in the past or "from the future") are indexed in quarantine buckets which might get rolled completely differently than your normal buckets. ... View more

Unparsed or incorrectly broken? If they are incorrectly broken you might want to tweak that line breaker. Use https://regex101.com to test your ideas against your data. If they are not/incorrectly parsed, either the events are malformed or you might be hitting extraction limits (there are limits to the size of the data and number of fields which are automaticaly extracted if I remember correctly). ... View more