It kind of looks like you might have created an indexed field extraction using EXTRACT-foo. If that is the case, try the following search: index=bar foo::"*hello*"   ... View more

Use the HTTP Event Collector (HEC), which is simple enough to do. You just have to put the data in JSON format with the index, sourcetype, etc. in the JSON string along with the data package and send it to the HEC endpoint that you activate on your IDX layer via an HTTP request. If you are already getting that data into a python script, it can make the HTTP call to the HEC endpoint immediately and the data will be in your IDXs within seconds. https://docs.splunk.com/Documentation/SplunkLight/7.3.6/GettingStarted/UsingHTTPeventcollector ... View more

Are your events so long that they are being split before the end of the event? Look at the raw events. If you have a raw event that this is happening on, please provide a "cleaned" version of the event here, both in the raw version that is the original, but also the _raw value in splunk, in case there are differences. ... View more

What do you mean by " the column statement_text is splitting in different rows"? Do you mean that there are more rows because there are "unintended" newlines among your CSV file? Is that because there are newlines in the statement text? Or is there something else that I'm not understanding? ... View more

In the case that you end up with escaped double quotes embedded in your string (e.g. "this is \"just\" what the doctor ordered") you will need to have a slightly different regex. Something like: | rex field=_raw "statement_text\=\"(?<statement_text>(\\"|[^\"])+)\"" This should work for any number of singly-escaped double quotes in your string. I only add this because of the chance of a double quote ending up in there somewhere with such a long, unstructured string. There is also the strange possibility that there could be unescaped double quotes, which might require something like:  |rex field=_raw "statement_text\=\"(?<statement_text>[\s\S]+)\",\s*cnt=\"" ... View more

See System Requirements for more information on your hardware needs If you intend to use the hardware you described in your post, then you may be disappointed, whatever tool you decide upon. 24GB/day is not much (we handle more than 1,000 times that on our clusters), but you may be disappointed with so few cores (you are talking about using a minimal requirement using your best hardware). The drive configuration is also suspect. It really depends on the number of IOPs you can get from them. Configuring them wrong would be degrading to the performance. If no one searches the data, you could probably keep up with the ingestion of the data, but searching AND ingesting will probably be slow and you won't be pleased. I used to manage a cluster that did 200GB/day with 3 indexers, but each one had 16 cores and less RAM. It did fine. Where I am currently we use 48 core machines with 256GB RAM with SSD drives for hot storage. The clusters have more than 100 IDXs in them (we have 5 clusters).  The following information is given with the assumption that you are using effective server hardware: Splunk is very capable at receiving the data, and data sent to it is 95% likely to be searchable with in 5 seconds. However, that is dependent on how the data is sent to Splunk, and how much data is going to the indexers (or at least the parsing layer you use). I would say that it meets the first criteria just fine. As to the second criteria of getting UI interactions within 1 second, that is a very hard one to determine. It is dependent on a number of things How good/bad is the search How long is the time range over which you are searching How much data is going to be returned by the indexers How many of the fields are indexed at index time vs search time etc. Some searches will return within a second if you have all the stars aligned and the moon is full. But there are so many variables that the search is dependent upon that it is hard to say. That is the case with ANY log tool. Never assume that all interactions will be fast. I've seen really bad searches that never complete in a day (I usually see this from users who have no clue what they are doing and to all-time searches in a cluster with over 200 IDXs in an index that is over 200TB in size. These get killed and we have a serious talk with the user that does this.) Can you get fast interactions with Splunk? Absolutely. Can you mess it all up and get slow ones? Absolutely. If you don't know what you are doing you can keep a Ferrari from going more than 10 miles an hour. The important thing is to learn all you can so that you don't keep it from performing optimally.  ... View more

I do see that you are using Windows (sorry about that), but the best option that I have ever seen for syslog data is the Splunk Connect for Syslog app available from Splunkbase. It would be installed on a Linux server, so if your environment doesn't support Linux, or you have no clue how to use a Linux machine, then you would not want to use it. It doesn't require an HF to send the data to splunk, only an HEC endpoint (whether that is an HF or an indexer (or cluster), so it is light weight and fast (I've had 4 times the throughput of a UF and 20 times the throughput of an HF on the same server using this method and still not maxed out the server). I would seriously look into this method. Syslog straight into a splunk machine is not a good idea, you will have packet loss if you do. Splunk is slow to restart, so any time you try to update or restart that type of HF you will get packet loss during that whole time. Something to think about. ... View more

I think that your understanding of  NO_BINARY_CHECK is incorrect. If you set that to true, then Splunk will process a file with binary data. It doesn't change the data to readable text. From the docs: NO_BINARY_CHECK = <boolean> * When set to true, Splunk software processes binary files. * Can only be used on the basis of [<sourcetype>], or [source::<source>], not [host::<host>]. * Default: false (binary files are ignored). * This setting applies at input time, when data is first read by Splunk software, such as on a forwarder that has configured inputs acquiring the data.   ... View more

Please edit the example regex above and make it be code text formatted by using the 101010 formatting button to make it so that all your characters are visible. It's likely that the * characters can be guessed, but there could be other characters that are missing. ... View more

Slightly more generic (since it looks like a comma is the delimiter): ... | rex "activityId: (?<ACTIVITY_ID_VALUE>[^,]+)" ... View more

A UF doesn't parse the data, so this functionality must be done on the indexer, or an HF. Your title for the question specifically states doing it on a UF, so the answer is, you can't. But you can do it on the indexers or an HF. ... View more

Your reply still lacks sufficient information. Please use the text formatting button that looks like 101010 in two rows next to the double quotes " button to provide preformatted text to clarify answers and example data. So, please provide some (sanitized) example data that shows the original data AND the various parts of the data (as you state "paragraphs") so that it can be determined exactly what you are trying to explain. It is still unclear what constitutes a "paragraph" and you talk about a first and second paragraph. So what part is the first paragraph? What part is the second paragraph? Are there always two paragraphs, or can there be just one? Are there ever three or more paragraphs? Without an understanding of your data it is impossible to help you out. As you can see by the lack of responses, no one is able to help because there isn't enough information to help you. This is likely not a hard problem to overcome, it's just a lack of information to come up up with an answer. Sorry to be blunt. I would truly like to help. ... View more