I think, you are hitting some basic issues. So my suggestion is you to learn a bit more into making settings correctly and read the configs to setup a larger cluster system. Please find suggestions in order of preference 1. Get a developer license (10GB per day) with full license options. Its free. 2. Setup a standalone System with SH, IDX all in same machine with license . Check everything is working 3. Now create 4 indexers separately and use a Cluster Master 4. Now split license server outside and so on.. 5. Finally make a SH cluster If you don't want to have all the pain, Use docker. Best docker images are 1. Full blown Splunk-n-box => https://github.com/mhassan2/splunk-n-box 2. Simpler Docker Splunk => https://github.com/getkub/docker_splunk You can then customise the docker setup to bring up all the cluster etc, rather than having individual installations in VM ... View more

@rakshithreddy, based on what you have asked. In case both filters $transactionid$ and $anonymousaccountid$ are provided and still do not return any results, the filter will remain as AND through the following logic. However,a detailed description of your use cases might be required so that you don't run into similar "pitfalls" (or NFRs in your case): <done> <!-- No Results found by performing AND, hence change the filter to AND if both inputs have been provided else set it to OR --> <condition match="$job.resultCount$==0"> <eval token="filterType">case(len($transactionid$)>0 AND len($anonymousaccountid$)>0,"AND",true(),"OR")</eval> </condition> <!-- Result/s found by performing AND, hence retain the filter to AND --> <condition> <set token="filterType">AND</set> </condition> </done> ... View more

Following is a run-anywhere search that you can try out to see if it fits the need. This search performs an AND to fetch at-least one record(using head 1 so that it does not consume a lot of resources). In case it find no results then Search Filter is set to OR, otherwise it is set to AND <search> <query>index=_internal sourcetype="$selSourceType$" log_level="$selLogLevel$" | head 1</query> <earliest>-1d@d</earliest> <latest>now</latest> <done> <!-- No Results found by performing AND, hence change the filter to OR --> <condition match="$job.resultCount$==0"> <set token="filterType">OR</set> </condition> <!-- Result/s found by performing AND, hence retain the filter to AND --> <condition> <set token="filterType">AND</set> </condition> </done> </search> <fieldset submitButton="false"> <input type="dropdown" token="selSourceType" searchWhenChanged="true"> <label>Select Sourcetype</label> <choice value="splunkd">splunkd</choice> <choice value="splunkd_ui_access">splunkd_ui_access</choice> <choice value="doesnotexist">doesNotExist</choice> </input> <input type="dropdown" token="selLogLevel" searchWhenChanged="true"> <label>Select Log Level</label> <choice value="INFO">Info</choice> <choice value="WARN">warn</choice> <choice value="UNKNOWN">Unknown</choice> </input> </fieldset> <row> <panel> <table> <title>Filter Type=$filterType$</title> <search> <query>index=_internal sourcetype="$selSourceType$" $filterType$ log_level="$selLogLevel$" | stats count by sourcetype log_level</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">20</option> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> </table> </panel> </row> PS: log_level= INFO and WARN exist only in splunkd and not on splunkd_ui_access. You can also try out doesnotexist and UNKNOWN for unmatched scenario testing. ... View more