If I'm running multiple applications, say we have a mobile application, a web application, and some back end services applications and they all send their logs to the same Splunk server, what is the best way to distinguish/split/group logs by application? I was thinking:
1- Send the application name in the log entry
2- Create a UDP port data entry for each of the applications and filter on these ports.
We have our logs going to specific indexes based upon the application sending the data and the type of data it's sending. We have an index for web service calls, another index for access calls such as web requests, another index for our release environment etc.. We have 4 public facing applications including our mobile sites which all the web requests go to the same index and we than created a field which we could easily define which application we wanted to look at. Joining indexes or creating subsearches can kill search performance so we decided to go this route.
You may not want to have the same set up but you will need to ask yourself how often you will need to search across your applications, is everything isolated between applications, do you have dedicated servers for each application etc.