This is not an efficient way to solve this problem. With Splunk, it's much, much better to have the work done in a single search pass with an OR than to run two searches and combine the results. Here you should search: (sourcetype=first_source 404) OR (sourcetype=second_source 303) | fields url (sourcetype=first_source 404) OR (sourcetype=second_source 303) | stats count by url Now the efficient version of intersect is very similar. Here we count the number of sourcetypes per url and only keep those urls with more than 1 sourcetype (hence both the sourcetypes): (sourcetype=first_source 404) OR (sourcetype=second_source 303) | stats count dc(sourcetype) as sourcetypes by url | search sourcetypes > 1 These searches can (and will) return an unlimited number of rows, and statistics will be accurate on the entire result set. Using set will only pull a limited number of rows from each search. ... View more

For the lookup table, I would suggest building the table up front using a search like the one above. If it takes a long time to build, I'd suggest incrementally adding to it via "sourcetype=userAccount earliest=-1h@h | intputlookup usertable.csv append=t | stats first(userType) as userType by userID | outputlookup usertable.csv". If your userActivity search ever only retrieves hundreds of distinct users, and the lookup table doesn't scale well, I'd suggest making a lookup script that executes one or more searches against sourcetype=userAccount to achieve the lookup. ... View more

Is there any guarantee that the specific userID->userType mapping is in recent time or does this need to run over all time? ... View more

You don't have to worry too much about the lookup getting too big. Once it exceeds a certain size, we will index it for fast retrieval. Your other choice here is to create a script based lookup. If the userID->userType mapping lives in an RDBMS, you can write a python script that performs the lookup. The contract for the script is that it will receive CSV with blank values on its stdin, and it should produce CSV on its stdout with the blanks filled in. This should easily scale with the query speed of the RDBMS. Now the last possibility is a script runs searches against sourcetype=userAccount. ... View more

This is a very interesting question and none of the answers actually get you to the full solution, where you find not only the top concurrency during the day, but when that occurred. First you should note that the bin command simply flattens one field into "bins". For example, it turns seconds of the day into hours of the day. It alone cannot do the job here,it must be combined with concurrency to find the actual concurrency at any transition point in the time series (i.e., when a connection comes or goes). Here's the first search, which will find the most recent occurrence of the top concurrency time of the day: ... | eval duration = ReqTime/1000 | concurrency duration=duration | bin span=dh _time as day | dedup day sortby -concurrency | eval when=_time | timechart span=1d max(concurrency) as concurrency list(when) as when | convert ctime(when) Now your actual search is a bit trickier since we may have many points in the day of highest concurrency: ... | eval duration = ReqTime/1000 | concurrency duration=duration | bin span=1d _time as day | eventstats max(concurrency) as max_concurrency by day | eval when=if(concurrency==max_concurrency, _time, null()) | timechart span=1d max(concurrency) as concurrency list(when) as when | convert ctime(when) The general recipe here is: Convert the duration to seconds. Compute the level of concurrency at the start of every event. Create a representation of the day of each request. Either dedup to find the single highest concurrency point, or eventstats+eval to find the many highest equal concurrency points. Use timechart to summarize the data by day. Make the time of occurrence prettier. Note that this technique can be used on splunk_access.log data using the following search: index=_internal source=*/splunkd_access.log earliest=-24h@h | eval duration = spent/1000 | concurrency duration=duration | bin span=1h _time as hour | eventstats max(concurrency) as max_concurrency by hour | eval when=if(concurrency==max_concurrency, _time, null()) | timechart span=1h max(concurrency) as concurrency list(when) as when | convert ctime(when) ... View more

Unfortunately there's no well supported way of doing this, but it's a good enhancement request to forward on to Customer Support. If you are feeling daring, you can modify $SPLUNK_HOME/etc/apps/search/bin/sendemail.py which renders emails. Note that any modifications will be overwritten on upgrade. ... View more

As Michael says, this is certainly a supported deployment. There are three basic ways in which summary indexing can be set up in a distributed environment: Perform summary indexing on the search head. This is the simplest deployment and usually works very well. It has the advantage of making use of the disk resources on the search head, which are typically underused. Perform summary indexing on a separate search head, and distribute search from the primary search head to the summarizing search head. This reduces compute burden on the primary search head. This is almost always the best use of a second search head. Perform summary indexing on either a primary or secondary search head and forward the data to the indexers. This is typically the highest performance deployment. Any summary search will be able to retrieve the summary records in parallel from all the indexers. We don't usually recommend setting up independent summarization on the indexers (as described by zscgeek). This is for both management and performance reasons. From a management perspective, it's harder, but not impossible, to make sure that summaries built on a search head will work perfectly on the indexers. From a performance perspective, we give up some of the "compression" implied by summarization. In any of the above three approaches, for any given time period there will be exactly one summarization. If all the indexers summarize independently, then each time period will have n summaries, one for each indexer. This uses more space and requires computation to recombine. ... View more

Another workaround here is to continue to use file uploads, but manually configure the delimiter based extraction for the source or sourcetype. It should be noted that CHECK_FOR_HEADER doesn't perform any magic beyond setting a per-sourcetype search-time field extraction rule. This is easy to achieve for a person after indexing the data. The documentation at http://www.splunk.com/base/Documentation/latest/Admin/Extractfieldsfromfileheadersatindextime shows the configuration that CHECK_FOR_HEADER makes when a new input comes in. ... View more

Lowell is correct. We intend to fix this problem in 4.2 by recording license usage in a more granular manner. Specifically, we'll capture usage by the tuple (_time, source, sourcetype, host, forwarder) so that you can slice and report by any combination of those fields. As an aside, are you running your search from the "Advanced Charting" view or CLI? These are many times faster than the default "flashtimeline" view for reporting searches. ... View more

This should normally work, and its failure probably has something to do with the heuristic of looking for the value in the index. The first check to make is to not put the field comparison in the initial part of the search. Does a search for just *text2search yield all the results that should match your field search. Another way is instead of: my_field="*text2search" Try: * | search my_field="*text2search" ... View more

Define developer's IP. If you have a mapping of splunk user to IP, then the macro trick is best. ... View more

As gkanapathy said, this is a bug with the specification of the xpath command and its support for multivalued fields. You can fix this in your installation by adding to $SPLUNK_HOME/etc/apps/search/local/commands.conf : [xpath] supports_multivalues = true ... View more

It's unclear to me how the linked question has anything to do with this question. ... View more

You can get this behavior by setting up a custom multivalued extraction for the "type" field. In props.conf set (for your sourcetype or source): REPORT-type = mv-type In transforms.conf set: [mv-type] REGEX = type=(?<type>\S+) MV_ADD = true ... View more

This sounds like SPL-31773, and is fixed in 4.1.5. I'm not sure why it would have worked in 4.1.3, however. The problem is that subsearches aren't run with sufficient information to know about the app that they're in. I'd try gkanapathy's suggestion of sharing the extraction globally, and see if that works. You'll have to add export=system to both [props] and [transforms] . ... View more

Given your reference to exporttool, I'd imagine that the other answers here aren't quite right, as they deal with streaming data out at the same time it's indexed. You probably want to look at the export capabilities added to CLI search in version 4.1. A CLI search like the following will stream all events out of the index to standard out, and can be piped into other programs or redirected into other files: splunk search '*' -maxout 0 splunk search '*' -maxout 0 | wc -l splunk search '*' -maxout 0 > exportfile.txt You can replace the '*' here with any search on keywords or fields. In fact, as long as the search is non-transforming (no stats, sort, timechart, top, ...) you can use other search commands like lookup, eval, where, regex. ... View more

This appears to be a duplicate of: http://answers.splunk.com/questions/5321/indexes-data-retirement-earliest-event ... View more