When you do the stats list(domain) as sender_domain, you are creating a single multivalue field. Better to just count by domain, so that each domain is its own event in a single value field. Try this: index=mail sourcetype=xemail [search index=mail sourcetype=xemail subject = "Blah" |stats count by UID| fields UID] | stats list(subject) as subj list(sender) as sender list(recipient) as recp list(vendor_action) as status by UID | search status= "Message done" | rex field=sender "[@\.](?<domain>\w+\.\w+)$" | stats count by domain | rename domain AS sender_domain | lookup watch_list.csv domain as sender_domain OUTPUT flag | eval on_watch_list=if(flag=1,"yes","no") | fields - flag ... View more

Not sure you necessarily need dc. If you wanted count of unique combinations using the 5 fields you listed: index=main auditSource=filing auditType=FilingCompleted OR auditType=FilingSubmissionAcknowledged | stats count by "detail.Filing ID", field1, field2, field3, field4 | table field1, field2, field3, field4 However, if you really only want to use "detail.Filing ID" to remove duplicates: index=main auditSource=filing auditType=FilingCompleted OR auditType=FilingSubmissionAcknowledged | stats latest(*) AS * by "detail.Filing ID" | table field1, field2, field3, field4 You can substitute latest for earliest, first, or last, depending on if you want the first matching field or the last, or you want the earliest chronologically matching field or the latest. http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/CommonStatsFunctions ... View more

You can search an elasticsearch instance from Splunk: https://github.com/hvandenb/splunk-elasticsearch I personally haven't seen any add-ons or customized versions of Kibana that allow it to search Splunk. You could have logstash agents forward data to splunk (probably using syslog or a tcp socket). I'm not sure about forwarding splunk forwarder data into elasticsearch; you'd need some intermediary to pass the data, say syslog/tcp to logstash, then to elasticsearch. ... View more

I think you have a syntax error in your inputs.conf file. You need to specify the key that contains the string "delete". In the inputs.conf example from the docs: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf whitelist = EventCode=%^200$% User=%jrodman% You'll need to look at your specific event to identify the key. For example if the key was "Action": whitelist = Action=%delete% Your second method of using a props/transforms routing rule is also not correct. You have the configurations swapped between the two files. But in general you would use this method of filtering on your indexers or heavy forwarders, not your universal forwarders. Try correcting your whitelist and see if that works for you. ... View more

It depends on how frequently this IP list will change. You could manage the list of IP addresses in a lookup file (ie, a CSV) or you could have a scripted lookup that pulls the address list from an external source. http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/Lookup If you were using a lookup file, you'd probable have it formatted something like this, in a file called admin_ip.csv: admin, allowed_ip joe, 1.2.3.4 bob, 2.3.4.5 Then use a subsearch to look for your admin's activity. Substitute eventtype=login for whatever search criteria you use to match a login event. This also assume that your login event has a field called admin. If not, rename the admin field in the subsearch to whatever the username field is called in your login event. Then lookup the list of allowed IP addresses, see if they match the IP address in the event (assuming its in the ip field), and return the list of events. eventtype=login [| inputlookup admin_ip | dedup admin | fields admin] | lookup admin_ip admin OUTPUT allowed_ip | eval allowed=if(ip==allowed_ip,"Y","N") | where allowed=="N" ... View more

If its an older archive that was not generated from an active cluster member, there's not much you can do to force it to replicate. Index buckets created pre-clustering aren't in the right format to be replicated. If you want to, you could distribute the buckets to other cluster members, but that won't give you redundancy, it will just distribute the search load. You have to make sure to avoid bucket numbering collisions as well as duplicating buckets across the cluster (or you'll see duplicate events in your search). ... View more

Really, the solution here is to look at your syslog config and the splunk config and correct the truncation issue. You should send the events over TCP instead of UDP syslog if you can, because you can send a longer event. Also, make sure you are not truncating the events yourself by inserting newlines in your syslog data. However, if you want to work around the problem, it sounds like you do have control over the script that is writing the data to syslog. What I would suggest you do is actually break up your single event into multiple events that will not be truncated, and record a session id in the event so you know which events belong together. First, to figure out your event size limit do something like: sourcetype=your_sourcetype | eval length=len(_raw) | stats max(length) Then in your script, do something like this (I'm making these extremely short, but you get the idea): <timestamp> id=1 key1=value1 key2=value2 key3=value3 <timestamp> id=1 key4=value4 key5=value5 key6=value6 And so on. You then use the id field to glue your events back together. Then perform a search one of two ways. Either, join all your data by id using stats: sourcetype=my_sourcetype | stats avg(key1) avg(key4) by id sourcetype=my_sourcetype | stats values(*) AS * by id Or use a subsearch to find specific events (you don't have to search in field values of course). sourcetype=my_sourcetype [sourcetype=my_sourcetype key4=value4 | fields id] Of course if you don't need all of the fields, you can just do a normal search. But at least you'll avoid having your data split. ... View more

By index, do you mean you want to create indexed fields from the filename? So that in your example you have: 12345_filename_78910.csv You want the following fields? ID1=12345 filename=filename OR 12345_filename_78910.csv (?) ID2=78910 A search time extraction would look something like this: | stats count | eval source="/some/path/1234567_file_na_454054_me_8910.csv" | rex field=source "(?<ID1>\d+)_(?<filename>.*)_(?<ID2>\d+)\.csv" You could add this to your config, so the extraction is done automatically. In props.conf add: [your_sourcetype] EXTRACT-file_properties = (?<ID1>\d+)_(?<filename>.*)_(?<ID2>\d+)\.csv in source That should get you an automated extraction at search time. You can convert this to an index time extraction if you need to, but you might want to test and see if the search performance is really that bad. Keep in mind you are searching in the source field, which is an indexed field itself. Check this page for information for caveats about indexed time fields: http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/Configureindex-timefieldextraction ... View more