I have the following excerpt of exchange logs. There are more fields before and after this excerpt. ,awells@atcorp.com,awells@atcorp.com,Hi, my dear friend!, The regex I have developed is: ,(?P[^,]*|,),(?P[^,]*|,),(?P[^,]*|,),] Using comma (,) as a field delimiter The first field "sender_address" is parsed as "awells@atcorp.com" - the desired result The 2nd field "return_path" is parsed as "awells@atcorp.com" - the desired result The 3rd field "message_subject" is parsed as "Hi," but it should be "Hi, my dear friend!" I observed that a comma followed by character is field delimiter, but a comma followed by blank space is not. Question: what is a correct regex for the 3rd field? ... View more

I have installed the RTO app. (Scenario 1). To keep things simple, i have started with a simple scenario trying to enable only two field from my Netflow index search index=netflow | fields s_ip d_ip I was not able to get Output Assistant to work, i.e, nothing would appear in the "Splunk FIelds". What I did was to change the search to search index=netflow | fields s_ip d_ip | eval cef_static_map="dst:d_ip,src:s_ip" in the search field and enabled the search but NO output can be seen whether I am forwarding the data to an IP/host at port 514 or to a file (I check the log in $SPLUNK_HOME/var/log/rtouput) (Scenario 2). Subsequently, I changed the search to search index=netflow This time around, Output Assistant shows the all "Splunk Fields". This allows me to do the mapping of CEF fields to Splunk fields. I enabled the search but NO output can still be seen whether I am forwarding to an IP/host at 514 or to a file (I look at the log in $SPLUNK_HOME/var/log/rtouput) Question 1. I do not understand the behavior of Output Assistant in scenario 1. This is important as I am dealing with many logs and I only want to send specific fields to ArcSight, and not necessarily the complete log record where there are many fields that don't have a matching CEF field. Question 2. What did I do wrong in both scenarios because I did not see any output? THANK YOU for your support. ... View more

I am trying to feed Arcsight with the results of a Splunk search using the real time output app. I get the following error message. _time log severity raw 21 7/17/14 4:35:36.214 PM rtoutput.log ERROR [Errno -2] Name or service not known Traceback (most recent call last): File "/opt/splunk/etc/apps/SplunkRealTimeOutput/bin/scripted_inputs/rtoutput.py", line 462, in asyncore.loop(use_poll=True) File "/opt/splunk/lib/python2.7/asyncore.py", line 216, in loop poll_fun(timeout, map) File "/opt/splunk/lib/python2.7/asyncore.py", line 201, in poll2 readwrite(obj, flags) File "/opt/splunk/lib/python2.7/asyncore.py", line 117, in readwrite obj.handle_error() File "/opt/splunk/lib/python2.7/asyncore.py", line 503, in handle_error self.handle_close() File "/opt/splunk/etc/apps/SplunkRealTimeOutput/bin/real_time_output/clients/httpsclient.py", line 235, in handle_close self.handle_parse_chunked(self.linebuffer) File "/opt/splunk/etc/apps/SplunkRealTimeOutput/bin/real_time_output/clients/httpsclient.py", line 333, in handle_parse_chunked self.reset_and_callback() File "/opt/splunk/etc/apps/SplunkRealTimeOutput/bin/real_time_output/clients/httpsclient.py", line 472, in reset_and_callback self.callback(result) File "/opt/splunk/etc/apps/SplunkRealTimeOutput/bin/scripted_inputs/rtoutput.py", line 330, in _cb_syslog sent = syslog.send(output.formatted_events) File "/opt/splunk/etc/apps/SplunkRealTimeOutput/bin/scripted_inputs/rtoutput.py", line 240, in send self.socket.sendto(message, self.address) gaierror: [Errno -2] Name or service not known Question 1 - The search produces many fields. What happens when these fields not match with any of the CEF fields found ceftool.py? Should I restrict the numbers of fields from the search result to the ones that have a match in ceftool.py ? Question 2 - the RTO app seems to provide different names than the ones found in ceftool.py? Question 3 - What is the meaning of the listed error message? ... View more

Greetings, The sample logs are listed below 2014-06-18T02:25:16.879Z,TSEAET01\NEW - Internet receive connector TSEAET01,08D1456B7AFF9BDF,22,147.81.121.139:25,147.81.122.24:61707,,"CN=Entrust Certification Authority - L1C, OU=""(c) 2009 Entrust, Inc."", OU=www.entrust.net/rpa is incorporated by reference, O=""Entrust, Inc."", C=US",Certificate issuer name 2014-06-18T02:25:16.879Z,TSEAET01\NEW - Internet receive connector TSEAET01,08D1456B7AFF9BDF,23,147.81.121.139:25,147.81.122.24:61707,,4C1B9021,Certificate serial number 2014-06-18T02:25:16.879Z,TSEAET01\NEW - Internet receive connector TSEAET01,08D1456B7AFF9BDF,24,147.81.121.139:25,147.81.122.24:61707,,27A7B6AAACBE39610C3A148D60EF4F5F2BE60FB0,Certificate thumbprint 2014-06-18T02:25:16.879Z,TSEAET01\NEW - Internet receive connector TSEAET01,08D1456B7AFF9BDF,25,147.81.121.139:25,147.81.122.24:61707,,TSEAET01.tascnet.tasc.com;Mail1.tasc.com;Mail.tasc.com;Mail.tascnet.tasc.com,Certificate alternate names 2014-06-18T02:25:16.910Z,TSEAET01\NEW - Internet receive connector TSEAET01,08D1456B7AFF9BDF,26,147.81.121.139:25,147.81.122.24:61707,*,,Received certificate The field headers are Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context My (non-working) regexes are as follows (? [^,]+),(? [^,]+),(? [^,]+),(? [^,]+),(? [^,]+),(? [^,]+),(? [^,]+),(?(?=")(? .+),)|(?(?=,)(? ,),)|(?(?=.)(? [^,]),),(? .+)\r\n I have trouble parsing the field named "data", which can take any one the following forms 1) "CN=Entrust Certification Authority - L1C, OU=""(c) 2009 Entrust, Inc."", OU=www.entrust.net/rpa is incorporated by reference, O=""Entrust, Inc."", C=US" 2) 4C1B9021 3) , It appears that the time stamp is processed correctly, however. ... View more

The search below produces multiple values for c_ip index=proxy* | fields c_ip s_op d_ip r_host d_port cs_bytes cs_uri referer c_agent | lookup RedSkyIOCip-Proxy Indicator AS d_ip OUTPUT Source ReferenceAttribution | search Source=RedSkyIOC | table _time c_ip d_ip Source ReferenceAttribution cs_uri referer c_agent When it is modified to become a subsearch as below, the subsearch only returns 1 value for c_ip. What is not working? index=in_index [ search index=proxy | fields c_ip s_op d_ip r_host d_port cs_bytes cs_uri referer c_agent | lookup RedSkyIOCip-Proxy Indicator AS d_ip OUTPUT Source ReferenceAttribution | search Source=RedSkyIOC | dedup c_ip | rename c_ip AS s_ip | return s_ip ] | table _time s_ip d_ip d_port action | sort s_ip ... View more

I run a search on a field that has multiple values. For example the field quest_name has the following values quest_name 1 wpad.TASCNET.tasc.com 2 wpad.tascnet.tasc.com 3 wpad.stafford.net 1). I can compute the number of records that exists for each field value with the following search index=dns* quest_name=wpad* | stats count by quest_name | sort - count The results are quest_name count 1 wpad.TASCNET.tasc.com 5777 2 wpad.tascnet.tasc.com 1324 3 wpad.stafford.net 225 2). I can compute the total number of records for all values for quest_name with the following search index=dns* quest_name=wpad* | stats count(quest_name) AS total The results are total 1 9492 3). Now I want to obtain the percentage of each field value in relation to the "total" value using a single search to show the following calculations: quest_name count percent 1 wpad.TASCNET.tasc.com 5777 5777/9492= 2 wpad.tascnet.tasc.com 1324 1324/9492= 3 wpad.stafford.net 225 225/9492= This I have not been able to do. Your help is requested. Thank you. ... View more

This is related to my DNS index. I need to search all names that start with wpad and to list all the values found. I did this using the command "index=dns* quest_name="wpad* | stats values(quest_name) AS WPAD An excerpt of the results is shown below wpad.NGIT.Northgrum.com wpad.NGMS.Northgrum.com wpad.Northgrum.com .... What I need is to compute the number of times each of the preceding values, say "wpad.NGIT.Northgrum.com", appears in the search. Can it be done using a single search? Thank you. ... View more

I search Netflow firewall denied traffic on port 53 using the netflow index. Based on the IPs found (source and DNS destination server), subsequently and using the DNS index I want to find out the domain name of the specific queries that were blocked. How can this be done by passing the source and destination IPs obtained in the 1st search to the 2nd search? ... View more

I have set up a lookup table that consists of a number of offenses that need to be identified for every daily search. Each search works fine and output the IP along with the offending event. I need to keep track of these violations by IP over the time line. Notably when a threshold count has been exceeded, an alert is generated. Your help is requested (1) in generating a file name that records the IP and the offense. The IP is not known until the violation happens. (2) Append additional violations to an IP is the file already exists and (3) trigger an alert if a threshold has been exceeded ... View more

We have a need to identify the country of origin of IPs that are hitting our firewalls, notably from "unfriendly" countries. To that end, I have collected a list of IPs in CIDR notation for each of these unfriendly countries. The Splunk lookup search works but our 8-CPU server is choking and drastically slows down other searches. One file has about 7k lines (142352 bytes), the other about 11k lines (231628 bytes). Each IP entry in the lookup table is in CIDR notation. I have reason to believe that matching an IP to a subnet in CIDR notation is CPU intensive. Linux "top" and "vmstat" show that the all 8-CPUs remain steady at 99-100%. Memory and swap space usage is quite low. Question: are there ways to improve the searches performance? (1) I can index these cvs tables that are small by setting the "max_memtable_bytes=142350" parameter. This setting is system-wide and may have impact that I not aware about (2) I can develop an external lookup doing binary search rather a sequential search on the lookup files. Your help in clarifying (1), (2) and other options would be much appreciated ... View more