I have two indexes. One is index=intrusion and the other is index=threat_list, index=ips consists of IPS event logs which tracks dropped traffic on our perimeter IPS, the second index is index=threat_list containing the IPs of known threat sources. We would like to correlate IP addresses hitting our permiter IPS in index=intrusion with IP adresses on the index=threat_list. Then we would then like to map those results every month. The threat list has more than 2 million IPs per day and the IPS can receive about 3 million "attacks" per month. This is what we have come up with so far but the problem is when reporting on these indexes on a monthly basis the event counts are so high that searches time out even though we have raised their limits in limits.conf Google Maps App for plotting search results: index=intrusion | join src [search index=threat_list] | stats count as _geo_count by src | geoip src | search _geo=* | stats sum(_geo_count) as _geo_count by _geo Can anyone suggest a way to make the search more efficient so we do not see these timeouts or suggest a different method perhaps using lookup, dedup, stats, etc. that we can then use to map the results similar to the image below and avoid timing out? ... View more

I used the IFX tool in Splunk for some of my logs. I am trying to extract the severity of attacks in my IPS logs which are categorized (high, medium, low, info/informational). The logs look something like this: Dec 25 08:58:14 172.16.48.100 Netdefender: 25-12-2013 08:55:21 WARNING 12572 Intrusions "SIP-Scanner-SIPVicious" UDP 188.138.34.198 5077 172.16.48.100 5060 1 Regular "Primary_Aggregate" occur 1 3 N/A 0 N/A high drop FFFFFFFF-FFFF-FFFF-0027-000052F2B407 Dec 25 08:55:21 172.16.48.150 Netdefender: 25-12-2013 08:55:21 WARNING 350 Anti-Scanning "TCP Scan" TCP 198.20.99.130 0 0.0.0.0 0 0 Regular "Primary_Aggregate" ongoing 0 0 N/A 0 N/A medium drop FFFFFFFF-FFFF-FFFF-1E24-000052F089EF Dec 25 08:55:26 172.16.48.200 Netdefender: 25-12-2013 08:55:20 WARNING 350 Anti-Scanning "TCP Scan" TCP 42.114.23.195 0 0.0.0.0 0 0 Regular "Primary_Aggregate" ongoing 3 1 N/A 0 N/A medium drop FFFFFFFF-FFFF-FFFF-52BF-000F521C7F60 Dec 25 08:55:16 172.16.48.150 Netdefender: 25-12-2013 08:55:16 WARNING 350 Anti-Scanning "TCP Scan" TCP 198.20.99.130 0 0.0.0.0 0 0 Regular "Primary_Aggregate" ongoing 2 0 N/A 0 N/A medium drop FFFFFFFF-FFFF-FFFF-1E24-000052F089EF Dec 25 08:55:16 172.16.48.150 Netdefender: 25-12-2013 08:55:16 WARNING 350 Anti-Scanning "TCP Scan" TCP 198.20.99.130 0 0.0.0.0 0 1 Regular "Primary_Aggregate" start 0 0 N/A 0 N/A medium drop FFFFFFFF-FFFF-FFFF-1E24-000052F089EF Dec 25 08:55:21 172.16.48.200 Netdefender: 25-12-2013 08:55:15 WARNING 350 Anti-Scanning "TCP Scan" TCP 198.143.173.176 0 0.0.0.0 0 0 Regular "Primary_Aggregate" term 0 0 N/A 0 N/A medium drop FFFFFFFF-FFFF-FFFF-6C70-000F521C7F60 This is the regex that IFX generated: (?i)/.*? (?P<FIELDNAME>[a-z]+)(?= ) So to test I did: sourcetype=netdefender | rex field=_raw "(?i)/.*? (?P<FIELDNAME>[a-z]+)(?= )" For some reason the field FIELDNAME is extracting some occurrences of the values (term, ongoing, sampled, TCP). Does anyone have any ideas on a regex they can help me with? ... View more

According to the documentation here I need to have ids_type in my events as follows ids_type="network" or ids_type="application" or ids_type="host". How can I add a field value pair to my events automatically? Would I have to create this in transforms.conf? [netdefender] REGEX = Netdefender FORMAT = ids_type=network WRITE_META = true This is a portion of a sample event: Jan 30 10:32:43 192.168.1.1 Netdefender: 30-01-2014 02:54:05 WARNING I am not sure how this works its been a long day. Help please! ... View more

My event looks like this: Jan 30 10:32:43 192.168.1.1 Netdefender: 30-01-2014 02:54:05 WARNING We would like to use the second timestamp for our events. We have configured props.conf in /local like this: [netdefender] TIME_PREFIX = \w+\s\d+\s\d{2}:\d{2}:\d{2}\s\d+\.\d+\.\d+\.\d+\s\w+\:\s MAX_TIMESTAMP_LOOKAHEAD = 44 We are still seeing index time as the timestamp. What are we missing? ... View more

Receiving the following error message in S.o.S. Splunk in the splunkd.log: 12-28-2013 16:32:28.175 -0500 ERROR ExecProcessor - message from "/opt/splunk/bin/scripts/fetch.sh" wget: /opt/splunk/lib/libssl.so.1.0.0: version `OPENSSL_1.0.0' not found (required by wget) I have seen these two posts but they do not give instructions on exactly how to change LD_LIBRARY_PATH for Splunk: http://answers.splunk.com/answers/105439/no-port_scan-data http://answers.splunk.com/answers/106816/commands-broken-after-upgrade-to-splunk-6 Any instruction on the exact procedure of changing LD_LIBRARY_PATH for Splunk in this situation would be highly appreciated. Thank you! ... View more

these are my logs and i need to grab complete .exe filenames: 1366986567.625 41 94.229.0.20 TCP_DENIED/403 1896 GET http://193.142.244.17/lxkj3824y896yursilh/5492.exe cindy@demo.com NONE/- - BLOCK_WBRS-DefaultGroup-Demo_Clients-NONE-NONE-NONE - - 1366984129.742 47 27.35.11.11 NONE/503 1890 GET http://topwinsystemscan.com/install/installpv.exe maximus@demo.com NONE/topwinsystemscan.com - OTHER-NONE-Demo_Clients-NONE-NONE-DefaultRouting - - 1366965031.191 8 203.172.197.2 TCP_DENIED/403 1866 GET http://81.174.66.128/.comete/10.exe tom@demo.com NONE/- - BLOCK_WBRS-DefaultGroup-Demo_Clients-NONE-NONE-NONE - - this is my progress so far: | rex field=_raw "http://[a-z0-9./]+(?< EXE>)[A-Za-z0-9].exe)" ... View more

I am a complete noob and I need help configuring two forwarders using a deployment server: Forwarder A Need to monitor index.log from 3 apache directories /opt/log/www* << Do I need a whitelist or blacklist here? If so, need help there too. Need them to go to index=www Need to label host as webA webB and webC Forwarder B Need to monitor denied.log from /opt/log/syslog << Do I need a whitelist or blacklist here? If so, need help there too. Need to monitor allowed.log from /opt/log/syslog Need them to go to index=firewall Need to label host as firewall1 Thank you ... View more