Splunk Search

Why does my timechart lose data with real-time search? (all data is present and correct on "Last x minutes/hours" type search)

johnraftery
Communicator

Hi,

I have a timechart which appends three types of data into one chart in this way:

eventtype=x sourcetype=x | where ... | table _time series value 
| append [ search eventtype=y sourcetype=y | where ... | table _time series value ] 
| append [ search eventtype=z sourcetype=z | where ... | table _time series value ] 
| timechart bins=1000 max(value) by series limit=0

This works nicely when the start and end time are static, as in when viewing the last 60 minutes for example. However, when I switch to a real-time search like "1 hour window", only the first set of data is displayed ( eventtype=x sourcetype=x). If anyone is able to shed some light, that would be much appreciated.

Thanks for reading!
John

Here is the full search, for reference:

eventtype=mlc2 sourcetype=tps host=SPLUNK_DEMO | where match(operationIdentity, "^[A-Z]")  | eval value = endTime - startTime | rename operationIdentity as series | table _time series value 

| append [ search eventtype=mlc2 sourcetype=lts_timings host=SPLUNK_DEMO | eval series=task_name."; ".duration_seconds." seconds" | eval value=1000 | eval end_timestamp=strptime(end,"%Y-%m-%d %H:%M:%S") | eval end_event=mvappend("",end_timestamp.",".series.",".value,"") | mvexpand end_event | rex field=end_event "(?<_time>[^,]+),(?<series>[^,]+),(?<value>[^,]+)" | table _time series value ] 

| append [ search eventtype=mlc2 sourcetype=vmstat-linux host=SPLUNK_DEMO | rex field=source "/(?&lt;series&gt;[^/]+).{24}$" | eval series=if(like(series, "vmstat_DB_%"), "CPU DB", if(like(series, "vmstat_APP_%"), "CPU APP", if(like(series, "vmstat_ONYX_%"), "CPU ONYX", if(like(series, "vmstat_SM_%"), "CPU SM", series)))) | rename cpu_used as value | table _time series value ] 

| timechart bins=1000 max(value) by series limit=0
0 Karma
1 Solution

hortonew
Builder

You can't use subsearches in realtime searches as the search would have to finish in order to pass the data back to the main search. See the following for more info:

https://answers.splunk.com/answers/77808/how-do-i-run-a-real-time-subsearch.html

View solution in original post

hortonew
Builder

You can't use subsearches in realtime searches as the search would have to finish in order to pass the data back to the main search. See the following for more info:

https://answers.splunk.com/answers/77808/how-do-i-run-a-real-time-subsearch.html

johnraftery
Communicator

Oh, that's a pity. Thanks for the info.

0 Karma

johnraftery
Communicator

FYI, I have tried rearranging the search into a different order:

eventtype=z sourcetype=z | where ... | table _time series value 
| append [ search eventtype=y sourcetype=y | where ... | table _time series value ] 
| append [ search eventtype=x sourcetype=x | where ... | table _time series value ] 
| timechart bins=1000 max(value) by series limit=0

In this case, only the eventtype=z sourcetype=z search is displayed. This tells me that the issue is not related to the content of the subsearches, but maybe something about the append? I have also tried an equivalent search with appendcols. I simplified my original search to the following:

index=mlc_live sourcetype=tps host=SPLUNK_DEMO | where match(operationIdentity, "^[A-Z]")  | eval value = endTime - startTime | timechart bins=1000 max(value) by operationIdentity 
| appendcols [ search index=mlc_live sourcetype=lts_timings host=SPLUNK_DEMO | eval value2=1000 | timechart bins=1000 max(value2) by task_name ]

Still only the first set of search results come through.

John

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...