I have logs coming from different sources like juniper IDS, cisco firewall, bluecoat proxy, nessus etc. Currently I have sample logs from these sources and these are present on my local system. I want to forward/upload these logs to Splunk Enterprise Security and check different dashboards.
I have read in Splunk documentations that the logs must be having correct sourcetypes so that splunk can properly index it and perform efficient search. The problem which I am facing is how to forward these logs to Splunk with correct sourcetypes so that these are processed by Enterprise Security.
I have already tried following things
My question is
Edit to address your questions:
1.) Use a syslog server to collect the logs via TCP/UDP 514
One solution you could explore is configuring rules in your syslog.conf file to sort the logs into their own directories based on host name or IP. From here, you could create a stanza for each source in your inputs.conf file to monitor each directory and forward the logs to their respective sourcetypes.
For example, here are a few lines from my rsyslog.conf file:
## Juniper firewall
if $fromhost-ip startswith 'x.x.x.x' then /var/log/JuniperFirewall
& ~
##Network appliances
if $hostname contains 'myhostname' or $hostname contains 'anotherhostname' or $hostname contains 'anotherhostname' then /var/log/network
& ~
## Log Bluecoat stuff
if $fromhost-ip startswith 'x.x.x.x' then /var/log/bluecoat
& ~
And some of the corresponding entries in my inputs.conf file:
[monitor:///var/log/JuniperFirewall]
disabled = 0
start_from = oldest
no_priority_stripping = true
index = syslog
sourcetype = juniper
[monitor:///var/log/bluecoat]
disabled = 0
start_from = oldest
no_priority_stripping = true
index = syslog
sourcetype = bluecoatProxy
2.) Yes, Splunk will automatically identify sourcetypes but you should assign your own as sometimes the sourcetype Splunk assigns isn't always ideal
3.) There is some great information on best practices and considerations for Splunk ES, and data inputs in general, here:
http://docs.splunk.com/Documentation/ES/4.6.0/Install/InstallEnterpriseSecurity
http://docs.splunk.com/Documentation/ES/4.6.0/Install/Planyourdatainputs
While I hope someone else chimes in on this subject, from my own experience, some of the considerations and challenges I faced with surrounding Splunk enterprise security so far are:
Tuning the appliances and log sources appropriately. Filtering out unneeded or noisy events will greatly help utilize all the neat features Splunk ES has and reduce the amount of alerts/results generated.
Following a common information model (CIM) to standardize your fields
@adayton20 I am also facing the same challenges with Splunk ES. Facing issues while deciding the sourcetypes because Splunk ES dashboard does not show the results if incorrect source types are set.
Most sourcetype names are arbitrary and can be changed, so you could always go in and modify the searches and/or configuration files to meet the needs of your environment. To my knowledge, ES does not REQUIRE any particular naming convention for sourcetypes. You can go in an change or add more sourcetypes to searches. If you need help with this shoot me a message and I'll be happy to help.