Why does my timechart lose data with real-time search? (all data is present and correct on "Last x minutes/hours" type search)

[johnraftery]

Hi,

I have a timechart which appends three types of data into one chart in this way:

eventtype=x sourcetype=x | where ... | table _time series value 
| append [ search eventtype=y sourcetype=y | where ... | table _time series value ] 
| append [ search eventtype=z sourcetype=z | where ... | table _time series value ] 
| timechart bins=1000 max(value) by series limit=0

This works nicely when the start and end time are static, as in when viewing the last 60 minutes for example. However, when I switch to a real-time search like "1 hour window", only the first set of data is displayed ( eventtype=x sourcetype=x). If anyone is able to shed some light, that would be much appreciated.

Thanks for reading!
John

Here is the full search, for reference:

eventtype=mlc2 sourcetype=tps host=SPLUNK_DEMO | where match(operationIdentity, "^[A-Z]")  | eval value = endTime - startTime | rename operationIdentity as series | table _time series value 

| append [ search eventtype=mlc2 sourcetype=lts_timings host=SPLUNK_DEMO | eval series=task_name."; ".duration_seconds." seconds" | eval value=1000 | eval end_timestamp=strptime(end,"%Y-%m-%d %H:%M:%S") | eval end_event=mvappend("",end_timestamp.",".series.",".value,"") | mvexpand end_event | rex field=end_event "(?<_time>[^,]+),(?<series>[^,]+),(?<value>[^,]+)" | table _time series value ] 

| append [ search eventtype=mlc2 sourcetype=vmstat-linux host=SPLUNK_DEMO | rex field=source "/(?&lt;series&gt;[^/]+).{24}$" | eval series=if(like(series, "vmstat_DB_%"), "CPU DB", if(like(series, "vmstat_APP_%"), "CPU APP", if(like(series, "vmstat_ONYX_%"), "CPU ONYX", if(like(series, "vmstat_SM_%"), "CPU SM", series)))) | rename cpu_used as value | table _time series value ] 

| timechart bins=1000 max(value) by series limit=0

[hortonew · Accepted solution]

You can't use subsearches in realtime searches as the search would have to finish in order to pass the data back to the main search. See the following for more info:

https://answers.splunk.com/answers/77808/how-do-i-run-a-real-time-subsearch.html

[johnraftery]

Oh, that's a pity. Thanks for the info.

[johnraftery]

FYI, I have tried rearranging the search into a different order:

eventtype=z sourcetype=z | where ... | table _time series value 
| append [ search eventtype=y sourcetype=y | where ... | table _time series value ] 
| append [ search eventtype=x sourcetype=x | where ... | table _time series value ] 
| timechart bins=1000 max(value) by series limit=0

In this case, only the eventtype=z sourcetype=z search is displayed. This tells me that the issue is not related to the content of the subsearches, but maybe something about the append? I have also tried an equivalent search with appendcols. I simplified my original search to the following:

index=mlc_live sourcetype=tps host=SPLUNK_DEMO | where match(operationIdentity, "^[A-Z]")  | eval value = endTime - startTime | timechart bins=1000 max(value) by operationIdentity 
| appendcols [ search index=mlc_live sourcetype=lts_timings host=SPLUNK_DEMO | eval value2=1000 | timechart bins=1000 max(value2) by task_name ]

Still only the first set of search results come through.

John