I'm running into this same issue and have not found a solution. Was Splunk Support able to help you resolve it? ... View more

Had similar ideas, with 2) being the most logical. We use Puppet so 3) could be an option, but I don't like two independent systems fighting over control. I was hoping there was an easier native Splunk DS method but it looks like option 2) is closest to that. ... View more

For the backfill case you can just use fill_summary_index.py and tell it over which timerange it should run the searches. See http://docs.splunk.com/Documentation/Splunk/7.1.3/Knowledge/Managesummaryindexgapsandoverlaps#Use_the_backfill_script_to_add_other_data_or_fill_summary_index_gaps ... View more

I thought as such since there are certain cases that would cause odd behaviour. When the forwarder is reading the file in (relatively) real-time and they are getting indexed in a similar amount of time, then the added log timestamp would be useful to understand the order of the events when looking at the logs afterwards (outside of Splunk), but since you may have network issues and delays in forwarding/reading the file - if it indexed the entire file at once they'd all have the same timestamp which isn't that useful. ... View more

You are correct. Once a file becomes ignored, it never comes back to being monitored even if its timestamp is updated. ... View more

I tried, it doesn't change anything - it still only gives me the first event (startswith) and last events (endswith) - the events in between are not included. ... View more

You may want to interrogate the splunk indexer's contributions to the _internal index as a timechart by SOURCE. The difference in log events by time should correspond to to your hourly CPU temper tantrum. Hopefully you can see a periodic difference in the number of events by source, which may help you identify events that only occur in this span. Do you have any batch operations indexing data every hour....maybe being directed to only one indexer instead of being load-balanced? ... View more

I have literally just specified index=myindex and set the custom timeframe. I have a feeling the "earliest date" column in the index configuration is lying to me, and it is actually rolling over faster than I thought. ... View more

@ineeman, we're getting XML parse errors from jobs.export, where jobs.oneshot completes the same query. Is it possible to export either the csv.gz, json or python OrderedDict representation? ... View more

It sounds very strange. If you search for * you will get all the raw data. Which you could then export. So leaving the data at the indexers but being allowed to search them provides no security whatsoever. ... View more

Thanks, I found the file and it made a new one on restart, and I can see the data now so looks like it was that. ... View more

Hi, Clustering seems to work exactly in your case but you don't want to have more than 2 VMs / hosts in your infrastructure. Either increase the number of nodes and opt for Splunk supported clustering or use external HA solution like VMotion etc. Regards, Amit Saxena ... View more

You could work yourself around that with bin span=1s count somethingorother, then you can use streamstats' event window just like you use a rolling seconds window. Edit: http://splunk-base.splunk.com/answers/72115/streamstats-with-time-window ... View more

Does this work for 2 servers only? i.e. both servers acting as indexers and search heads? ... View more