Yes, I promoted my last comment to an answer so please accept it below ... View more

What is the problem with putting one or more monitor stanzas over those files in that directory? ... View more

Hey I just tested your exact EVAL and you do can change it in Search Time with that EVAL. The problem is not the time itself, I think it is the strptime: Change the expression to EVAL-_time=if(match(_raw,"SB"),_time+86400,_time) Also... Have you restarted Splunk after changing the props? (Pardon me if it is a stupid question, but I don't see anything else wrong with the eval expression) ... View more

I'm not sure you can change an indexed field such as _time with an EVAL in the props.conf. Could you test doing an EVAL to another field, such as time_test? ... View more

Assuming that when you stopped the indexer, the cluster was in a complete state: When you take the indexer down without the offline, you are not giving the cluster master time to redefine the searchable primary buckets, so it is possible that while Splunk is yet determining that, you can't get data from the once primary buckets of the dead indexer while splunk doesn't mark the ones in the up node as primary. ... View more

The EVAL you are setting in the props.conf is not done on Index Time. It is done every time you do a search on that sourcetype mydata What exactly are you seeing, the time is not modified as you expected? ... View more

You can just monitor events_last.xml Splunk user CRC so although it is still monitoring the same file, he registers in the fishbucket the last line he has read, and a Checksum of the beginning and end of file. So when you events_last.xml is cleaned and is starting to be written again, Splunk will read that again automatically. You won't lose any event. ... View more

What are your sites replication and search factors? ... View more

To check where splunk is using your default host, use btool $SPLUNK_HOME/bin/splunk btool inputs list --debug It will show you the running configuration in your inputs and show you for the stanza you'rel looking for, where is it getting the default host from (where meaning from which conf file specifically) ... View more

Notice that a normal append won't produce that strange behaviour: index=_internal source=*scheduler.log OR source=*metrics.log OR source=*splunkd.log OR source=*license_usage.log OR source=*splunkd_access.log | stats count AS raw_count, dc(processor) AS raw_dc_processor by host | append [ | tstats prestats=t summariesonly=t count, dc(server.processor) FROM datamodel=internal_server BY host | stats values(raw_dc_processor) AS raw_dc_processor, dc(server.processor) AS dm_processor, values(raw_count) AS raw_count, count AS dm_count by host] ... View more

It was a good discussion, but only now did I fully understand your index-time need! ... View more

(index="x") AND (host="y") | bucket _time | stats sum(eval(if(status > 199 AND status < 400,1,0))) as Success, sum(eval(if(status>400,1,0))) as Failures BY application, methodPath, _time ... View more

I believe you have a mistake here: SOURCE_KEY = _MetaData:Sourcetype ... View more

Indexed fields will always need more space because of the need to create btrees to make them searchable. That one is a fact. About performance, indexed fields are really good but you have to choose wisely, specially fields that don't have high cardinality (number of different values). If you don't need to control the subsequent places the data is travelling through. If a source of events is stamped with a sourcetype, then when you search for them in a Search Head, Splunk will send those EVAL I mentioned as a bundle so the indexer knows what to extract. So search time will solve it. Finally, yes, it is possible to do that in index time indeed. If you look here http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Configureindex-timefieldextraction You'll see that you can yes use a REGEX on the _raw data, but you can also put a specific value in it REGEX = <regular_expression> FORMAT = retention::5 This would put a retention=5 to a specific sourcetype. I honestly think it is an overkill for your use case, but of course it is your call ... View more

Let's use a Single Site Cluster case. The replication factor is total the number of copies of _raw data that exist in within the cluster members. If your cluster has 3 elements and you have a replication factor of 2, then you can be sure that 2 of your cluster members have one copy of a specific bucket (_raw) The search factor is the number of "indexed data" meaning the structure that makes data searchable. The same thing applies, if you have a search factor of 2 in a 3 members cluster, then 2 of them have the indexed structure. ... View more

There is no need for that join, you can do that in one shot: (index="x") AND (host="y") | stats sum(eval(if(status > 199 AND status < 400,1,0))) as Success, sum(eval(if(status>400,1,0))) as Failures BY application, methodPath | sort by Success DESC | sort by Failures DESC Let me know if it worked for you ... View more

The thing is, if you have two separate instances and are sending segmented data to Prod or Lab, you don't want to get mixed data in either of the environments. What I would suggest to avoid getting your tcpout queue full is explore this in outputs.conf: dropEventsOnQueueFull = <integer> * If set to a positive number, wait <integer> seconds before throwing out all new events until the output queue has space. * Setting this to -1 or 0 will cause the output queue to block when it gets full, causing further blocking up the processing chain. * If any target group's queue is blocked, no more data will reach any other target group. * Using auto load-balancing is the best way to minimize this condition, because, in that case, multiple receivers must be down (or jammed up) before queue blocking can occur. * Defaults to -1 (do not drop events). * DO NOT SET THIS VALUE TO A POSITIVE INTEGER IF YOU ARE MONITORING FILES! Using this this on the [tcpout:lab] server = lab.net:9997 ... View more

Agreed, maybe I'm overthinking, let's see what the user says ... View more

Ok so the reason is probably the Universal Forwarder has a Queue size that would fill up when your indexer fails, as it can't release any of the events in that queue ... View more

I don't think the user wants production data going to the dev environment in case the pro indexer fails... ... View more

If the answer was helpful, accept it for future reference to this ... View more

Honestly I don't see a reason for that. Maybe there is some misleading info there. It surely is not a best practice. ... View more