Production had a bug.  One of the results of that bug was massive "over logging" of production nodes and those logs were forwarded (universal forwarder) to our splunk server. Development reverted production, but Splunk was "log-jammed" as indicated by the queues for several hours: We know that we can clear the backlog on the clients (splunk universal forwarders) by turning off the forwarder, cleaning out the application logs as well as the following forwarder files: \<application logs> \var\log\splunk\metric* \var\lib\splunk\fishbucket* <restart forwarder>   It seems that several of the forwarders successfully forwarded data, so this jammed up our queues.  I realize that Splunk is designed NOT to lose data, but assuming we were willing to accept some "pending" data loss, is there any way to clear the server side queues or "dump data" from indexing to clear a backlog?   We considered "blacklisting" specific files from indexing as was done in this post, but as indicated by the post, removing un-doing the blacklist results in those files going back for index processing.         ... View more

On the indexer, I have the following property config: [host::newdatamine] TIME_PREFIX = ^[INFO|ERROR|WARN]\s+[ TIME_FORMAT = %y.%m.%d\s+%H:%M:%S BREAK_ONLY_BEFORE = [INFO|ERROR|WARN] example logfile: [INFO] [12.02.27 17:00:24] [mysp_Process_DataAffiliateAutoXML] Refresh data from lmAffiliateAttemptLog [INFO] [12.02.27 17:00:42] [mysp_Process_DataAffiliateAutoXML] Processing data for 2012-02-26 Splunk now breaks the lines correctly and picks up the correct date. It does not pick up the correct time: Splunk output: 1 2/29/12 12:02:29.000 PM [INFO] [12.02.29 19:00:43] [mysp_Process_DataAffiliateAutoXML] Refresh data from lmAffiliateAttemptLog * host=NEWDATAMINE Options| * sourcetype=newdatamine-too_small Options| * source=C:\DataWarehouse_Report\Splunk\newdatamine.log Options 2 2/29/12 12:02:29.000 PM [INFO] [12.02.29 15:21:28] [mysp_Process_DataAffiliateAutoXML] Processing data for 2012-02-28 * host=NEWDATAMINE Options| * sourcetype=newdatamine-too_small Options| * source=C:\DataWarehouse_Report\Splunk\newdatamine.log Options ... View more