You can route selected data to a 3rd party system, which might be a better way to do this. However, I think the *nix example in this link answers your question ... View more

The Delete command in search does not actually delete any data it marks it as deleted so that it no longer shows up in searches. That is one option if you do not care about recovering space. You need to explicitly grant the can_delete permission for your role. If you want to completely clean the index, then why not bring one indexer down at a time and use the clean command, i.e. splunk clean eventdata -index nameOfSummaryIndex ... View more

Great I'm glad to hear it is working now. Those two lines are actually doing nothing right now but I included them for completeness. They would be used if you wished to have different config per individual connection to each server. At the moment, using the config I posted above, you are using a Target Group to define your settings for the whole target group. And yes it will be load-balancing data across both servers in the Target Group. ... View more

You have checked that there are ESTABLISHED TCP connections from the SH to both indexers on port 9997. You have checked the splunkd.log on the SH for clues. You have changed outputs.conf to be more specific. Is the summary index defined on both indexers? Are there clues in the splunkd.log on indexer1 to suggest problems? If you configure outputs.conf on the SH to only send to indexer1, does this now work? Have you ever got any data to go to indexer1 on port 9997? Is there frequent summary data being written and have you left it long enough to be sure nothing is going to Indexer1? ... View more

Could you paste in your outputs.conf from the SH? Are there errors in "\SPLUNK_HOME/var/log/splunk/splunkd.log" on the SH that give a clue as to why there is no forwarding to Indx_01? Assuming you are forwarding to port 9997 on both indexers, does the following command, Linux: netstat -na | grep 9997 Windows: netstat -na | findstr "9997" , show ESTABLISHED connections to both indexers? ... View more

Your blacklist should probably be: blacklist = \.gz$ I believe your whitelist is missing a pipe symbol between the last two file specs, it should probably be: whitelist = (xyz_debug_ms[1-4]{1}.txt|app1_ms[1-4]{1}.txt|app2sos_ms[1-2]{1}.log|system_ms.log|remoteserver.log) When you say you then created two stanza, can you include them so we can see what they look like? You should not have overlapping monitor stanzas. ... View more

If you write the output to a file in a sinkhole like, .../var/spool/splunk, then Splunk will consume it. If you need the sourcetype to be the same as when it runs as a script then create a different sinkhole and specify the sourcetype ... View more

It might help to include your search, but I would guess that the search just needs to have the following added to the end of it: ... | fields - _raw ... View more

The inclusion of the _raw field is down to the search itself, so you can have it excluded by using the fields command, i.e. sourcetype=blah | fields - _raw or to exclude all hidden fields sourcetype=blah | fields - _* If you really only need specific fields, then use the table command to specify which fields you want displayed, i.e. sourcetype=blah | table source, sourcetype, host, index The dashes at the top underline the header row, do you not want any column headings? Without the _raw field you will only have a table of fields and the header row is surely useful? ... View more

Check that the sourcetype has not changed since applying the new setting, Splunk may have learned a new sourcetype. The TIME_FORMAT string should define the string that follows the TIME_PREFIX string. Remember that strftime format strings are not regular expressions so you cannot use things like \s+ to represent spaces. However the TIME_PREFIX string is a regular expression. Is the character between the "date" and "time" string definitely a space? You can check with the od command on linux: grep "date=" logFileName | head -1 | od -b and make sure the the space character is octal 40. It could be a tab (octal 11) or Hard-Space character (octal 240) Be careful copy/pasting from web pages as some characters (i.e. '-') can get changed to different, similar looking characters. Given the above checks out, the following should work: TIME_FORMAT = date=%Y-%m-%d time=%H:%M:%S TIME_PREFIX = ^\s*<\d{3}> ... View more

I would run the search more frequently (so less than 50K results) and save the results to a summary index if this is something you need to report often. ... View more