Splunk Search

Need help combining 2 separate searches on different log files and show the results on a single timechart as 2 separate lines

pdpsplunk100
Path Finder

Hi - I'm having trouble in combining 2 separate searches and displaying the results on a single visualization (timechart). search 1 searches for events in the access log and search 2 searches for events in another log file (process log). Both log files have a field called "responseTime". In the process log the responseTime field represents "Processing Time". I'm able to run the searches independently and plot the timechart but when I combine them using join I don't get back any results. If I use append I get back results only from search 2. I am looking for 2 lines on the same timechart - one that represents processing time (from process log) and the other would be response time (from access log).

search1: index=xxx source=/x/x/x/localhost_access_log* field=value1
search2: index=xxx source=/x/x/x/process_log field=value2

Using join I tried the following queries: (No results)

search1 | join [search2 | rename responseTime as processTime] | timechart avg(responseTime), avg(processTime)

(and)

search1 | timechart avg(responseTime) | join [search2 | timechart avg(responseTime)]

Using append: (got back results only from search2)

search1 | append [search2 | rename responseTime as processTime] | timechart avg(responseTime), avg(processTime)
Tags (1)
0 Karma
1 Solution

pdpsplunk100
Path Finder

This did the trick.

index=xxx (source=/x/x/x/process_log AND request_param=xxx) | rename responseTime as processTime | timechart avg(processTime) as "Processing Time" | join _time [search index=xxx (source=/x/x/x/localhost_access_log* AND request_param=yyy) | timechart avg(responseTime) as "Response Time"]

View solution in original post

0 Karma

pdpsplunk100
Path Finder

This did the trick.

index=xxx (source=/x/x/x/process_log AND request_param=xxx) | rename responseTime as processTime | timechart avg(processTime) as "Processing Time" | join _time [search index=xxx (source=/x/x/x/localhost_access_log* AND request_param=yyy) | timechart avg(responseTime) as "Response Time"]
0 Karma

renjith_nair
Legend

Try this

index=xxx (source=/x/x/x/localhost_access_log* OR source=/x/x/x/process_log )|timechart avg(responseTime) as responseTime, avg(processTime) as processTime
---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma

pdpsplunk100
Path Finder

I get how you use the OR to search both sources. But the field name is "responseTime" in both the log files. I would need to rename "responseTime" in the process_log to "processTime" before the timechart command. Where/How do I place the rename in your suggested query because I need to tell Splunk that the rename applies only to the process_log source.

0 Karma

sundareshr
Legend

Try this

index=xxx (source=/x/x/x/localhost_access_log* OR source=/x/x/x/process_log )| timechart avg(eval(match(source, "access") as  responseTime, avg(eval(match(source, "process") as processTime
0 Karma

pdpsplunk100
Path Finder

Tried using match but that does not seem to work. I get eval expression is malformed and timechart syntax is incorrect.

I was able to get what I wanted using the query below (using join). I'm still looking for ways to optimize the query as join is expensive but this does the job for now.

index=xxx (source=/x/x/x/process_log AND request_param=xxx) | rename responseTime as processTime | timechart avg(processTime) as "Processing Time" | join _time [search index=xxx (source=/x/x/x/localhost_access_log* AND request_param=yyy) | timechart avg(responseTime) as "Response Time"]

Thanks for your guidance folks.

0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...