See System Requirements for more information on your hardware needs If you intend to use the hardware you described in your post, then you may be disappointed, whatever tool you decide upon. 24GB/day is not much (we handle more than 1,000 times that on our clusters), but you may be disappointed with so few cores (you are talking about using a minimal requirement using your best hardware). The drive configuration is also suspect. It really depends on the number of IOPs you can get from them. Configuring them wrong would be degrading to the performance. If no one searches the data, you could probably keep up with the ingestion of the data, but searching AND ingesting will probably be slow and you won't be pleased. I used to manage a cluster that did 200GB/day with 3 indexers, but each one had 16 cores and less RAM. It did fine. Where I am currently we use 48 core machines with 256GB RAM with SSD drives for hot storage. The clusters have more than 100 IDXs in them (we have 5 clusters).  The following information is given with the assumption that you are using effective server hardware: Splunk is very capable at receiving the data, and data sent to it is 95% likely to be searchable with in 5 seconds. However, that is dependent on how the data is sent to Splunk, and how much data is going to the indexers (or at least the parsing layer you use). I would say that it meets the first criteria just fine. As to the second criteria of getting UI interactions within 1 second, that is a very hard one to determine. It is dependent on a number of things How good/bad is the search How long is the time range over which you are searching How much data is going to be returned by the indexers How many of the fields are indexed at index time vs search time etc. Some searches will return within a second if you have all the stars aligned and the moon is full. But there are so many variables that the search is dependent upon that it is hard to say. That is the case with ANY log tool. Never assume that all interactions will be fast. I've seen really bad searches that never complete in a day (I usually see this from users who have no clue what they are doing and to all-time searches in a cluster with over 200 IDXs in an index that is over 200TB in size. These get killed and we have a serious talk with the user that does this.) Can you get fast interactions with Splunk? Absolutely. Can you mess it all up and get slow ones? Absolutely. If you don't know what you are doing you can keep a Ferrari from going more than 10 miles an hour. The important thing is to learn all you can so that you don't keep it from performing optimally.  ... View more

I do see that you are using Windows (sorry about that), but the best option that I have ever seen for syslog data is the Splunk Connect for Syslog app available from Splunkbase. It would be installed on a Linux server, so if your environment doesn't support Linux, or you have no clue how to use a Linux machine, then you would not want to use it. It doesn't require an HF to send the data to splunk, only an HEC endpoint (whether that is an HF or an indexer (or cluster), so it is light weight and fast (I've had 4 times the throughput of a UF and 20 times the throughput of an HF on the same server using this method and still not maxed out the server). I would seriously look into this method. Syslog straight into a splunk machine is not a good idea, you will have packet loss if you do. Splunk is slow to restart, so any time you try to update or restart that type of HF you will get packet loss during that whole time. Something to think about. ... View more

I think that your understanding of  NO_BINARY_CHECK is incorrect. If you set that to true, then Splunk will process a file with binary data. It doesn't change the data to readable text. From the docs: NO_BINARY_CHECK = <boolean> * When set to true, Splunk software processes binary files. * Can only be used on the basis of [<sourcetype>], or [source::<source>], not [host::<host>]. * Default: false (binary files are ignored). * This setting applies at input time, when data is first read by Splunk software, such as on a forwarder that has configured inputs acquiring the data.   ... View more

Please edit the example regex above and make it be code text formatted by using the 101010 formatting button to make it so that all your characters are visible. It's likely that the * characters can be guessed, but there could be other characters that are missing. ... View more

Slightly more generic (since it looks like a comma is the delimiter): ... | rex "activityId: (?<ACTIVITY_ID_VALUE>[^,]+)" ... View more

A UF doesn't parse the data, so this functionality must be done on the indexer, or an HF. Your title for the question specifically states doing it on a UF, so the answer is, you can't. But you can do it on the indexers or an HF. ... View more

Your reply still lacks sufficient information. Please use the text formatting button that looks like 101010 in two rows next to the double quotes " button to provide preformatted text to clarify answers and example data. So, please provide some (sanitized) example data that shows the original data AND the various parts of the data (as you state "paragraphs") so that it can be determined exactly what you are trying to explain. It is still unclear what constitutes a "paragraph" and you talk about a first and second paragraph. So what part is the first paragraph? What part is the second paragraph? Are there always two paragraphs, or can there be just one? Are there ever three or more paragraphs? Without an understanding of your data it is impossible to help you out. As you can see by the lack of responses, no one is able to help because there isn't enough information to help you. This is likely not a hard problem to overcome, it's just a lack of information to come up up with an answer. Sorry to be blunt. I would truly like to help. ... View more

Given your question and the data that you have provided, I think that this "run anywhere" search shows a rex that will work as you have requested: | makeresults | eval data="c:\test directory with spaces\test_directory_with_underscores\filename (with: horrible habits).txt|c:\test directory with spaces\test_directory_with_underscores\little-child-directory" | makemv delim="|" data | mvexpand data | rex field=data "(\.(?<ext>[^.]+)|\\\(?<dir>[^.\\\]+))$" This rex requires some additional backslashes to make it interpret the backslashes that might appear on the file path, but it clearly shows that you can get one or the other of the fields that you want to extract from the data. The first three lines are just setting up the data, and the last one (with the rex command) is the one with all the magic. ... View more

An event is any data item that comes back from a splunk search. In most cases that would be equivalent to a log entry, whether it is a single line or multiple lines that are part of a single log message. What does the "entry" (or event) look like when you search the data? Can you provide a "sanitized" event that can show what you full event looks like and if there are fields that are extracted from that event? ... View more

More information needed. Can the term appear anywhere in the last entry of the paragraph? What constitutes a paragraph (is it the entire event), and what constitutes an entry (I assume it is a part of the event, but what delineates one entry from another)? ... View more

Can you fix your information above by using the 101010 button to keep Answers from interpreting things like <fieldname> improperly? That will make I much easier to answer you question. ... View more

It would actually be: | makeresults count=1 | eval val=4 | eval ptn="(?<dig>\d)" | map search="| rex field=val $ptn$" Except that the search results don't go into the map command for val in that way, and you can't send the val value into the search like this: | makeresults count=1 | eval val=4 | eval ptn="(?<dig>\d)" | map search="| rex field=$val$ $ptn$" because the val value isn't a field name. So you are stuck between a rock and a hard place. The rex command requires a quoted string for the regex that it will use, not a field. I don't know of a way that you can do what you are wanting to do. ... View more

You can always use the rex command to create/modify a field that is always extracted. For example: | makeresults | eval IP="10.0.0.1:9997" | rex field=IP "(?<myIP>[\d.]+)" will result in myIP containing just the IP, and not the port from the IP field. For future reference, it is always best to give some example data with your question so that it is easier to help answer you particular problem. ... View more