topic Re: Transaction Search Using different fields across multiple logs in Splunk Search

Transaction Search Using different fields across multiple logs

manwin — Thu, 05 Aug 2010 16:53:38 GMT

I am facing a problem with doing a transaction search across multiple logs (11 different sourcetypes) based on the example below.

I will simplify the scenario to just 4 sourcetypes for illustration purposes

sourcetype="web-access-logs" Common fields: Username, IP_address
sourcetype="application-logs" Common fields: Username, transaction_id
sourcetype="third-tier-app-logs" Common field: transaction_id
sourcetype="bmc-error-logs" Common field: Username

Is there a way to craft a transaction search to pull out all events?

In sourcetype="web-access-logs" I will want to pull out all activities based on the IP_address, as there will be no Username before the User logs into the portal and can only be tracked via the IP address until he logs in and the Username will appear, only then will we be able to link in the Username into the search.

Ultimately we are looking at having all the results using the indirect linkages to form a transaction for troubleshooting purposes

I tried a search as below

sourcetype="*" | transaction fields IP_address,Username,transaction_id connected=f maxspan=1h maxpause=35m |search transaction_id="123456789"

Unfortunately it is not working.

Re: Transaction Search Using different fields across multiple logs

Lowell — Thu, 05 Aug 2010 21:40:34 GMT

Do you need to look at many transactions at once for analysis purposes? If not, a much simpler approach may be to build a form-search based view (or sets of views) and let a user drive the investigation instead of trying to do everything in a single search... just a thought.

Re: Transaction Search Using different fields across multiple logs

Lowell — Thu, 05 Aug 2010 21:42:34 GMT

I do see one minor typo in your transaction command.

You can do either:

transaction fields="IP_address,Username,transaction_id" connected=f maxspan=1h maxpause=35m

transaction IP_address Username transaction_id connected=f maxspan=1h maxpause=35m

but you don't want the term fields just sitting there before you list your fields. I'm guessing this would look literally look for a field named "fields", which isn't what you want.

Re: Transaction Search Using different fields across multiple logs

manwin — Fri, 06 Aug 2010 14:59:12 GMT

Thanks I will test that out, the main reason why I need the transaction search is because the results is used to pull a user's transaction from the application which is spanning across different systems (11 of them in total) pulling out the entire transaction into a single view to understand the flow from start to end in a single view.

Re: Transaction Search Using different fields across multiple logs

Lowell — Fri, 06 Aug 2010 20:43:17 GMT

As with any complex search/view, I suggested either breaking the problem into smaller pieces if possible. Also, start with a simple search you can get working, then build on top of it. Only add a single new component (aka sourcetype) at a time, test and review your results and make sure to save off a working copy as you go (I find it helpful to simply copy-n-paste the search in a text editor along with some short notes about what's working and what's not. You can use splunk to find your previous searches, but without knowing which one was working, it can be quite time consuming I've found.)

Re: Transaction Search Using different fields across multiple logs

gkanapathy — Thu, 19 Aug 2010 22:47:58 GMT

As Lowell, says, transaction is for mass assembly of many transactions. If you're trying to look up a single transaction based on a field, it's much more efficient to use subsearches or use forms and drilldown to select the specific events that make up a transaction.

Re: Transaction Search Using different fields across multiple logs

Stephen_Sorkin — Wed, 25 Aug 2010 00:56:51 GMT

If your retrieval task is to see an entire transaction (either one or a handful of them), a subsearch is essential for performance. As gkanapathy and Lowell commented, you're pulling every event off disk and building a bunch of transactions just to throw most of them away.

You probably want a search like:

[search sourcetype="application-logs" transaction_id=<transaction_id> | dedup Username | fields Username] | transaction Username,transaction_id connected=f ...