The catch-all index is used when the input does not specify an index. Double-check indexes.conf on the syslog server and make sure every monitor stanza has a index= setting.
Or if they are supposed to go to a non-existant index (for example, specified when supplying event via HEC) if I remember correctly.
@richgalloway , I checked the available indexes.conf but i did not found monitor stanza section. Can you please specify the file location on linux OS ?
Not specifically. The files will be in directories under $SPLUNK_HOME/etc. Use btool or the Linux find command to locate them (yes, it's most likely there will be more than one).
splunk btool --debug inputs list
find $SPLUNK_HOME/etc -name inputs.conf
@richgalloway, On syslog server we have custom .conf file in syslog-ng directory where all palo alto logs coming on udp_port(10527) , tcp_port(10527) . In this file only i added the new pan source. Rest all pan sources from this conf are correctly landing to proper index on Splunk cloud except one new pan source.
Are you using any kind of an intermediate syslog layer? (This syslog-ng you're speaking of)
Does it change/manipulate the events in any way? (For example I have in one of my environments heavily complicated rsyslog-based solution that in the end supplies events to splunk via HEC).
If you're using a UF to get the data from the syslog server to Splunk then .conf file should be somewhere in /opt/splunkforwarder/etc/ rather than in a syslog-ng directory.