Correlate multiple log sources

[ElliottP]

I am new to Splunk and would really appreciate some guidance or advice on how to do the following:

I have some log files that I wish to build a search/report that correlates them. I have loaded them into Splunk as separate log sources via the "Choose Data From File or Directory" option. I followed the documentation to do this and they now appear as separate log sources in the search app.

The logs are of the following systems:
IDP/S
Firewall
NAT
DHCP
Domain Controller
Antivirus

I wish to correlate them by linking together certain fields of each source type to create a report of an instance of network incident such as malware threat picked up by Antivirus. This should then show the full path, step by step through the log sources above.

I am conducting this for a university project on logs that I have generated. I would really be grateful for any help or guidance anybody can offer.

Many thanks,

Elliott

[tgow]

There many different methods for solving this in Splunk but probably the easiest method initially is to create field aliases. For instance if you want to track an ip address through your systems then you can come up with a common name for the source ip which will make it easier to use search commands such as "stats" or "transaction". Keep in mind that this is just one of many methods in Splunk.

Let's imagine that we have several sources and the ip address has a different field name in each sourcetype.

sourcetype=firewall src_ip=10.10.10.10
sourcetype=ids source=10.10.10.10
sourcetype=WinEventLog:Security Source_Network_Address=10.10.10.10
etc....

Now you just need to come up with a common name and alias the unique names to it. Field aliasing can be done in either the props.conf file or in the UI (Manager-->Fields-->Field aliases). Here is a props.conf example:

[firewall]
FIELDALIAS-src = src_ip AS src

[ids]
FIELDALIAS-src = source AS src

[WinEventLog:Security]
FIELDALIAS-src = Source_Network_Address AS src

This will create an additional field of "src" for all of the sourcetypes. Now I can run the following search to start watching source ips through my systems:

sourcetype=firewall OR sourcetype=ids OR sourcetype="WinEventLog:Security" | transaction src maxspan=1h keepevicted=true

Here are some links for more information:

http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Abouttagsandaliases

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction

[rashid47010]

HI
index=websense OR index=bcoat | transaction src
what i understand from output that he grouped the activity(browsed URL's) by src IP.
means that source IP browse which URL's... AM I Right...?

secondly My actual requirement is that I want to see all the source IP's whos are bypassing websesne means there is no browsing record in websense logs and available in bcoat proxy.
I made one query but there is no result displaying.:

index=websense sourcetype=websense src NOT [search index=bcoat sourcetype="bluecoat:proxysg:access:file" | fields src ]

Please advise

[krugger]

Correlation is quite difficult, because there is no way to do and efective automatic correlation. So all the correlation will have to be written as custom queries/alarms.

If you have and IDS or IPS on the network that will already have signatures for most things you could detect by correlating all other sources you listed. Usually the logs will help incident response team analyze the IDS/IPS alerts.

Example correlations:

• Users login/accessing many different resources (bruteforcing/scanning multiple machines)
• Users login/accessing from many different locations (credential sharing/compromised accounts)
• Users login/accessing systems from multiple locations at the same time (credential sharing/compromised accounts)

The above correlations will probably only use AD logs. Things like known suspicious source IP accessing systems sucessfully (incoming hostile connection - for example China) and users accessing known suspicious destination IP (outbound data exfiliation and botnet command and control) should already be handled by IDS/Firewall.

Another thing to consider is what are you going to do about detection and false positives. I have so far not seen any really useful correlation being done by log managment systems in an automated way, because it is not possible to tell the system what makes sense for your network.

Usually you analyze the logs to try and identify things that are not an expected pattern. But maybe someone else has some better correlations working.