It kind of looks like you might have created an indexed field extraction using EXTRACT-foo. If that is the case, try the following search: index=bar foo::"*hello*"   ... View more

There are ways to remove the data from the buckets, but if the buckets are flagged for bucket integrity checking enableDataIntegrityControl=true then it will effectively render the buckets useless in splunk. If you are planning no trying some method to delete the data, you should know that it can result in difficulties. If it is very sensitive data, delete the buckets with the data, but don't edit the buckets. I was asked to see if I could modify buckets and no one know. I was successful, but they didn't have the integrity checking set up, so that was possible. If you have something like passwords and you are needing to maintain security, you will have to delete the buckets. I've had to do that before as well. Losing data isn't fun, but if it means being truly secure, you may have to bite the bullet. ... View more

Use the HTTP Event Collector (HEC), which is simple enough to do. You just have to put the data in JSON format with the index, sourcetype, etc. in the JSON string along with the data package and send it to the HEC endpoint that you activate on your IDX layer via an HTTP request. If you are already getting that data into a python script, it can make the HTTP call to the HEC endpoint immediately and the data will be in your IDXs within seconds. https://docs.splunk.com/Documentation/SplunkLight/7.3.6/GettingStarted/UsingHTTPeventcollector ... View more

The disk usage had to be tweaked so that the max data used on a partition wasn't being used. So setting the number of hot/warm buckets had to be tweaked to the right number. This has some bad side effects if you aren't careful. If you cut it too close to the max storage, then it will cause you to sometimes hit the point that you have IDXs go into auto detention. Then if you do a restart to the IDXs in the cluster, you can then get too many buckets going into cold, and if you don't have enough storage in cold, you get data being dropped off. So be careful to set your limits so that you don't end up with an auto detention problem. ... View more

Are your events so long that they are being split before the end of the event? Look at the raw events. If you have a raw event that this is happening on, please provide a "cleaned" version of the event here, both in the raw version that is the original, but also the _raw value in splunk, in case there are differences. ... View more

What do you mean by "the column statement_text is splitting in different rows"? Do you mean that there are more rows because there are "unintended" newlines among your CSV file? Is that because there are newlines in the statement text? Or is there something else that I'm not understanding? ... View more

In the case that you end up with escaped double quotes embedded in your string (e.g. "this is \"just\" what the doctor ordered") you will need to have a slightly different regex. Something like: | rex field=_raw "statement_text\=\"(?<statement_text>(\\"|[^\"])+)\"" This should work for any number of singly-escaped double quotes in your string. I only add this because of the chance of a double quote ending up in there somewhere with such a long, unstructured string. There is also the strange possibility that there could be unescaped double quotes, which might require something like:  |rex field=_raw "statement_text\=\"(?<statement_text>[\s\S]+)\",\s*cnt=\"" ... View more

The following run-anywhere search string uses a rex command which will produce the results you want from the two examples provided:     | makeresults | eval source="a+b.zgeypynd.pcsdatei.600.gpg.1.20210127014546.gpg" | rex field=source "^[^\.]*\.(?<filename>.*\.600)" The first part  ^[^.]*\.  is used to get rid of anything before the first . The rest just captures the file name unto and including the 600. This is also assuming that the filename is in the source , since what you seems to indicate that, but you can substitute whatever field works. ... View more

See System Requirements for more information on your hardware needs If you intend to use the hardware you described in your post, then you may be disappointed, whatever tool you decide upon. 24GB/day is not much (we handle more than 1,000 times that on our clusters), but you may be disappointed with so few cores (you are talking about using a minimal requirement using your best hardware). The drive configuration is also suspect. It really depends on the number of IOPs you can get from them. Configuring them wrong would be degrading to the performance. If no one searches the data, you could probably keep up with the ingestion of the data, but searching AND ingesting will probably be slow and you won't be pleased. I used to manage a cluster that did 200GB/day with 3 indexers, but each one had 16 cores and less RAM. It did fine. Where I am currently we use 48 core machines with 256GB RAM with SSD drives for hot storage. The clusters have more than 100 IDXs in them (we have 5 clusters).  The following information is given with the assumption that you are using effective server hardware: Splunk is very capable at receiving the data, and data sent to it is 95% likely to be searchable with in 5 seconds. However, that is dependent on how the data is sent to Splunk, and how much data is going to the indexers (or at least the parsing layer you use). I would say that it meets the first criteria just fine. As to the second criteria of getting UI interactions within 1 second, that is a very hard one to determine. It is dependent on a number of things How good/bad is the search How long is the time range over which you are searching How much data is going to be returned by the indexers How many of the fields are indexed at index time vs search time etc. Some searches will return within a second if you have all the stars aligned and the moon is full. But there are so many variables that the search is dependent upon that it is hard to say. That is the case with ANY log tool. Never assume that all interactions will be fast. I've seen really bad searches that never complete in a day (I usually see this from users who have no clue what they are doing and to all-time searches in a cluster with over 200 IDXs in an index that is over 200TB in size. These get killed and we have a serious talk with the user that does this.) Can you get fast interactions with Splunk? Absolutely. Can you mess it all up and get slow ones? Absolutely. If you don't know what you are doing you can keep a Ferrari from going more than 10 miles an hour. The important thing is to learn all you can so that you don't keep it from performing optimally.  ... View more

I do see that you are using Windows (sorry about that), but the best option that I have ever seen for syslog data is the Splunk Connect for Syslog app available from Splunkbase. It would be installed on a Linux server, so if your environment doesn't support Linux, or you have no clue how to use a Linux machine, then you would not want to use it. It doesn't require an HF to send the data to splunk, only an HEC endpoint (whether that is an HF or an indexer (or cluster), so it is light weight and fast (I've had 4 times the throughput of a UF and 20 times the throughput of an HF on the same server using this method and still not maxed out the server). I would seriously look into this method. Syslog straight into a splunk machine is not a good idea, you will have packet loss if you do. Splunk is slow to restart, so any time you try to update or restart that type of HF you will get packet loss during that whole time. Something to think about. ... View more

This regex will reduce your steps from 1322 to 1228, but I have no idea if it will prevent your limits errors: [{(, ](?<Dimensions>[-.\w[\]+ ]+)[.&[]  The data that you are working with and the result set that you want is rather abstract and hard to make better, since it isn't very comprehendible. The actual goal of what a Dimensions is is not clear.  ... View more

I think that @cmerriman has a valid point about your data. Is that the exact set of matches that you want to get? If it is, then that makes it so that we know the actual constraints. If you are saying only that you want to get that data exactly as it is with that data, but you just want to reduce the steps so that you don't have to make changes to limits.conf, then that is one thing. Looking at the results you got in regex101, I'm not seeing any kind of correlation to useful data that you could use from the regex you are using. I think that is your biggest problem in getting a good answer. If you want to get different results, then that would make finding a better regex a whole lot easier. As it is, I'm finding this very difficult to help find you a solution because if the data changes slightly but still be valid SQL, then you won't get a similar resulting set of Dimensions from your regex. Hopefully you can see the point I'm trying to make. ... View more

I think that your understanding of NO_BINARY_CHECK is incorrect. If you set that to true, then Splunk will process a file with binary data. It doesn't change the data to readable text. From the docs: NO_BINARY_CHECK = <boolean> * When set to true, Splunk software processes binary files. * Can only be used on the basis of [<sourcetype>], or [source::<source>], not [host::<host>]. * Default: false (binary files are ignored). * This setting applies at input time, when data is first read by Splunk software, such as on a forwarder that has configured inputs acquiring the data.   ... View more

Please edit the example regex above and make it be code text formatted by using the 101010 formatting button to make it so that all your characters are visible. It's likely that the * characters can be guessed, but there could be other characters that are missing. ... View more

Slightly more generic (since it looks like a comma is the delimiter): ... | rex "activityId: (?<ACTIVITY_ID_VALUE>[^,]+)" ... View more

A UF doesn't parse the data, so this functionality must be done on the indexer, or an HF. Your title for the question specifically states doing it on a UF, so the answer is, you can't. But you can do it on the indexers or an HF. ... View more

Your reply still lacks sufficient information. Please use the text formatting button that looks like 101010 in two rows next to the double quotes " button to provide preformatted text to clarify answers and example data. So, please provide some (sanitized) example data that shows the original data AND the various parts of the data (as you state "paragraphs") so that it can be determined exactly what you are trying to explain. It is still unclear what constitutes a "paragraph" and you talk about a first and second paragraph. So what part is the first paragraph? What part is the second paragraph? Are there always two paragraphs, or can there be just one? Are there ever three or more paragraphs? Without an understanding of your data it is impossible to help you out. As you can see by the lack of responses, no one is able to help because there isn't enough information to help you. This is likely not a hard problem to overcome, it's just a lack of information to come up up with an answer. Sorry to be blunt. I would truly like to help. ... View more