Does the script send to HEC or write to a file? If HEC - which endpoint? ... View more

Wait a second. File or table? What kind of source does this data come from? Monitor input? Dbconnect? Have you checked the actual data with someone responsible for the source? I mean whether the ID or whatever it is in your data corresponds to the right timestamp? ... View more

Splunk (monitor input to be precise) doesn't care about the checksum of the whole file. It is obvious that the hash of the whole file will change as soon as _anything_ changes within the file. Whether it is a complete rewrite of the whole file contents or just adding a single byte at the end - the hash will change. The monitor input stores some values regarding the state of the file. It stores the initCrc value which will obviously change if the file is overwritten (and length of which can be manipulated in settings). But it also stores the seekCrc which is a checksum of the last read 256 bytes (and a position of those 256 bytes within the file). I suppose in your case the file ends by closing the json array, but after subsequent "append", the actual array is appended so its closing bracket is removed, another json structure is added and after that the array is closed in a new place. Unfortunately, you can't do much about it. As I said before - you'd be best off by scripting some external solution to read that array and dump its contents in a sane manner to another file for reading. ... View more

Salting crc is very very rarely the way to go. Usually it's about the length of the initCrcLength. If your files contain a long "header" which is constant between files, you need to raise its value. But. Problems with crc duplication manifest themselves with the opposite to what you're getting - data _not_ being indexed at all due to Splunk considering two files the same, not indexing data multiple times. ... View more

@yuanliuYou meant RHS, not LHS 🙂 @ganesanvcI hope you're running this snippet on an already relatively filtered event stream. If you want to use it as an initial search because you're getting the text_search parameter from elsewhere (like a token in a dashboard) you might be way better off using a subsearch to create a verbatim search term. ... View more

This seems to be some fancy modern top-like program. And I supose it shows separate threads of single splunkd process. Notice that the memory usage is identical for all those entries. ... View more

1. Ekhm, your "dev team" cannot handle epoch timestamp? That is... surprising to say the least. 2. Who produces those logs? Another app written by another "dev team"? ... View more

Ok, there have been many ideas here but  oone asked the main question. Why do you want to do it? ... View more

https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Xyseries ... View more

Trying to fiddle with structured data by means of simple regexes is doomed to cause problems sooner or later. You have a single json array. If you want to split it into separate items you should use external tool (or force your source to log separate events). ... View more

You can use the splunk_server_group argument for the rest command to dispatch it to defined group of servers. See https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Distributedsearchgroups But the user running the search must have the dispatch_to_indexers (or however it is called) capability. ... View more

To add a bit of additional context to what's already been said - actually while most of the "other" Splunk components should be able to communicate with each other (or at least should be able to be able), forwarders are often (usually) in remote sites and environments which are completely separate from the "main" Splunk infrastructure so in many cases querying them directly doesn't make much sense. So yes, for _some_ HFs a separate role could be beneficial but there can be many HFs (and most UFs) which you should simply have no access to. And that's also why app management with DS works in pull mode - you serve your apps from the DS but it's the deployment clients (usually forwarders) which pull their apps from DS and you have no way of forcing them to do so. They have their interval with which they "phone home" and that's it. ... View more

I wouldn't call it broken. Short of rebalancing buckets between restarting each single indexer (which is completely ridiculous) you can't make sure that every bucket is searchable throughout the rolling restart process. If you have RF=SF>=2, then it's all just a matter of reassigning primaries and maybe restarting only one indexer at a time (which can be a long process if you have a big setup). But what if you have RF=SF=1? (yes, I've seen such setups). So yes, it's a bit frustrating but I wouldn't call it broken. ... View more

MC doesn't normally directly monitor forwarders. It can do indirect monitoring by checking their logs in _internal index. Sometimes people add HFs to MC with indexer role but AFAIR it causes false alerts since HFs don't actually do indexing. ... View more

Ok. Let's get it out into the open - rolling restart is one of the basic concepts of managing Splunk environments, multisite cluster is one of the more complicated setups. You're trying to manage the latter without understaing the former. I understand that you might simply have inherited an environment and "someone has to" take care of it but I'd urge you to take some proactive steps to learn how to do stuff properly. Not just as asking case by case for help, but getting some more organized training. Having said that - rolling restart is a mechanism with which you automatically restart clustered indexers or search heads but not all at once. That's why it's called rolling. The process is managed by either CM in case of indexers or captain in case of search heads and only part of your components is restarted at the same time until the whole layer is restarted. How big this part is is configurable (if I remember correctly it's 20% by default meaning that at any given time 20% of your indexers can be down due to the ongoing rolling restart). On its own it doesn't cause any configuration changes (although it's typically triggered by them) so no "token loss" whatever that could mean. I can imagine some badly engineered environments where rolling restart would cause interruptions in data ingestion but in a well designed setup it shouldn't. There are two most important effects of rolling restarts of indexers. 1. Your data availability during restart is reduced so the searches spawned during that time might return wrong/incomplete results. 2. Since Splunk limits some replication-related activities during rolling restart you have reduced resiliency in that time and your environment is more prone to data lose should anything like server crash happen during this process. Additionally, roling restart as any restart if done too often can create many small buckets. ... View more

Wait a second. Technicalities aside, it seems you're trying to do exactly opposite what you say you want to do. ... View more

I'm not 100% sure what you want to do and you're being quite vague about it. As @livehybrid already said, there are some ways to overwrite the default timestamp recognition but I'll add to it that it's needlessly complicated, might be difficult to maintain and adds extra load on the indexers since the timestamp has to be parsed twice out of the event. While dynamical routing to another index is a pretty common thing, recasting one general sourcetype to "subsourcetypes" which are slightly differently parsed into fields in search-time is also not unusual. But spllitting single sourcetype/source/host stream into completely differently treated events is typically an indication that someone didn't bother to properly classify and split the data upstream (like reading whole /var/log/messages or getting syslog from the whole environment as "syslog" sourcetype). ... View more

Well, sometimes you have to work what you have. Nothing shameful about it 😉 Just be aware that this format might require you to craft your searches much more thoughtfully if you want them to be relatively quick. For example, since you have this whole keyname=something,value=somethingelse setup, you can't do a simple something=somethingelse search because that field isn't extracted and isn't know until you plow through your data with the foreach command. But you can limit your initial search results in that case by simply searching for "somethingelse" as search term regardless of where in the event it is. This can hugely improve your search times. ... View more

I wouldn't expect any modern UF to be not only supported but also to work. That windows release is so out of support that I'd be more concerned with the OS itself than the UF version. ... View more

You should not need to enable the inputs separately on each indexer. That's what the CM-pushed apps are for. ... View more

As I said before in another thread - this is something that should be best discussed with your local Splunk Partner. I can tell you that you can just enable HEC input on the HF and send all the data there so that the HF distributes it to the indexers but there are other possible issues with this setup. Since you probably have just one HF, you're gonna be creating a SPOF. It might also not be able to process all the load you're gonna push on it. That's why at the start I said that this problem only looks easy. There are several approaches to solving it which might look right from the technical point of view, meaning that each of them "should work" but some of them might be better from the business point of view given specific circumstances unique to your situation and environment. ... View more

You don't define HEC input on CM. CM will not be ingesting your HEC events. You define HEC input (either just a token if you have the general HTTP input already enabled or enable whole HTTP input, define TLS parameters and define a token - for maintainability I'd split it into two separate apps - one with general HEC input settings, another with a token definition for your particular input needs) with CM in an app which is pushed to indexers. So the only thing you should do (as with pretty much everything regarding configuring your indexers) is creating an app in the manager-apps directory and pushing it with a new configuration bundle to your indexers. I'm not sure if enabling HTTP input in general in case you hadn't had one already will not require you to rolling-restart your indexers (and that's why I prefer a dedicated HF-layer in front of indexers so that changes to ingestion process and parsing do not cause indexer restarts but that's a topic for another time. ... View more

As usual, you have a relatively simply sounding problem which might not turn out to be so simple. As @marnall already pointed out - the HEC input is just an input on the component you define it on. So you can have multiple options here. 1) Create a HEC input on each indexer using the same HEC token and let the sources load-balance their requests between the receivers. But that requires the sources which can actively do load-balancing. 2) Deploy an external HTTP LB which will distribute HTTP requests to HEC inputs defined on indexers (again - with the same HEC token). 3) Create a HF with a HEC input which will receive data and distribute it to indexers using normal s2s load-balancing mechanisms 4) Create multiple HFs with HEC input and either LB between them on source or set up a HTTP LB in front of them. Each of those scenarios have their own pros and cons (simplicity, cost, maintainability, resilience). ... View more

The first thing to check is the splunkd.log on the problematic (sending) machine. It should tell you if the connection is established at all or if it's being actively rejected or anythin else. ... View more

Just for the sake of completness - stats by _time is fairly useful if you manipulate your timestamps (usually by means of bin/bucket). With raw untouched _time it can be useful if you have several events emmited at the same time (and you can be 100% sure about that) and you have no other unique identifier to mark them by. But this is rather unlikely since separate events, even regarding the same "physical event" usually come from separate sources and are slightly offset in _time. ... View more