Transaction or subsearch?

[gnovak]

I have a question. I've been trying to figure this out for a while.

I have a search I'm using to calculate the number of times users looked at specific reports. I have the number of all users per location in a lookup file. I then calculate the number of times a file was actually accessed per user and calculate a percentage to get an idea of which files are the most popular.

However, I'm looking to get away from the lookup file as it's not always an accurate count of the active users. I've created a script that will query a database and dump the number of users into a file each day. This file is logged by splunk.

Here's the problem. The logs now generated by the script are under a different sourcetype then the main search.

This was my search using a lookup file. The fields the lookup used were registry and registrarcount:

sourcetype="EPPWEB" source="/opt/log/*/web_server/info.log" OR source="/opt/log/*/*/web_server/info.log" WAT | dedup registrar,description,registry | stats count(registrar) as numviewed,max(registrarcount) as registrarcount by description,registry | eval percent=numviewed/registrarcount*100| fields registry, description, numviewed, registrarcount, percent

This is the search I now use to just get the count of how many users there are and also the name for each user. This is the information this is being called by the script and dumped in a log now. This isn't using a lookup file:

sourcetype=registrarcount | stats max(registrarcount) by registry

I need to somehow combine them. I tried doing "what i thought" was a subsearch but this didn't seem to work. I got the error "subsearches are only valid as arguments to commands:

sourcetype="EPPWEB" source="/opt/log/dotinfo/web_server/info.log" OR source="/opt/log/dotinfo/*/web_server/info.log" WAT | [ search sourcetype="registrarcount" ] | dedup registrar,description,registry | stats count(registrar) as numviewed,max(registrarcount) as registrarcount by description,registry | eval percent=numviewed/registrarcount*100| fields registry, description, numviewed, registrarcount, percent

Anyone have any ideas how to combine these searches? Should I use transaction?

[landen99]

The problem with your subsearch is that it is not used as an argument to a command. This may make more sense when you consider the following two examples and then imagine that the subsearch is "action" search term.

sourcetype=pan_threat action=allowed
sourcetype=pan_threat | search action=allowed

similarly, the command that you need to precede the subsearch is "search"

sourcetype="EPPWEB" source="/opt/log/dotinfo/web_server/info.log" OR source="/opt/log/dotinfo/*/web_server/info.log" WAT | search [ search sourcetype="registrarcount" ] | dedup registrar,description,registry | stats count(registrar) as numviewed,max(registrarcount) as registrarcount by description,registry | eval percent=numviewed/registrarcount*100| fields registry, description, numviewed, registrarcount, percent

I know this is resurrecting an old thread, but it was never answered and I think that it would help others trying to understand the subsearch.

[kristian_kolb]

I think you should post a few lines of the events for both the 'info.log' and the new 'log'.

Also, dbryan has a good suggestion below..

[dbryan]

A transaction could be used to do what you want, but it's a very expensive operation.

Are you aware of the outputlookup command? You can use it to output the results of a search to a CSV lookup file, and then you can use it like you would use any other lookup.
Just reference it by the filename, or create a new lookup in transforms.conf referencing the file.

If all you're trying to do is join the results of the subsearch to an outer search, you could do something like this:

`some other search` | join type=inner registry [
    search sourcetype=registrarcount | stats max(registrarcount) as max by registry
    | table max, registry
]

I'm not sure if this will help; if you can clarify in simpler terms what you're actually trying to do that'd be good.

[gnovak]

I wasn't aware of the outputlookup command. I'll research that a bit.

[gnovak]

I basically need to run a search and take the results from that search and use it in an additional search. I assume subsearch does that but wasn't able to get it to work right...Does subsearch also carry over fields?

[gnovak]

All that is in the new file I am indexing is a number. A script runs, queries a database, and dumps a number (the number of users) into a file that is named after the location. It does this once a day. The files are indexed under sourcetype=registrarcount. I also have an entry in props.conf to extract the name of each location as "registry" from the actual source path to where the log is located. An example of a log entry for this is here: 2012-10-15 08:00:04+00 104

[kristian_kolb]

transaction is NOT what you need. A typical, though simple, example of where transaction could be used is to group events on a common parameter, such as a JSESSIONID, userID or IP-address, with optional constraints like begin-end, min- and max-times etc etc. See the docs.

I have a bit of trouble understanding what data you have in the 'new' file you're indexing, but it doesn't really sound like events to me - more like a lookup table.

Always provide some sample events, what you've done so far, and the desired output. That makes it easier to get help.

/k