topic Re: [Studio JSON Source Code] Possible to append two ds.search or ds.chain together WITHOUT subsearches? in Dashboards & Visualizations

[Studio JSON Source Code] Possible to append two ds.search or ds.chain together WITHOUT subsearches?

ClubMed — Thu, 30 May 2024 04:09:16 GMT

Specifically speaking the dataSources section discussed here:
https://docs.splunk.com/Documentation/Splunk/9.2.1/DashStudio/dashDef#The_dataSources_section

Hypothetically, I have two tables, each stored in individual data source stanzas:

Table 1 = ds.search stanza 1
Table 2 = ds.search stanza 2

The goal is to append the tables together, and then use the "stats join" method to merge the two tables together. If possible, this merge could be done as a ds.chain type stanza with two extend options, but it does not appear to be allowed.

Here's the documentation for Data source options.

https://docs.splunk.com/Documentation/Splunk/9.2.1/DashStudio/dsOpt

The document seems to be missing options like "extend", so I'm hoping someone knows if there's any additional options that is hidden.

Now, I am trying to avoid using the [] subsearches because of 50,000 row limit, so the following append command will not be desired:

<base search> | append [search ....]

Anyone with mastery of JSON hacks might know if appending two data sources stanzas together be possible?

Thank you.

Re: [Studio JSON Source Code] Possible to append two ds.search or ds.chain together WITHOUT subsearches?

ClubMed — Thu, 30 May 2024 04:23:43 GMT

Update in case anyone tried testing to see if "append" option exists, the "append" option does actually save but appears to not work.

Re: [Studio JSON Source Code] Possible to append two ds.search or ds.chain together WITHOUT subsearches?

ClubMed — Mon, 03 Jun 2024 23:35:52 GMT

After an investigation long story short, this is not possible and it needs to be a new feature suggestion if someone needs to request it.

The explanation is that I looked at the network logs for the Dashboard Studio and found the payload for base and chain searches.

Base search has their own parameter in the payload called 'search'.

All chained searches are grouped together in a parameter called 'postprocess_searches'.

There's no other parameters that support a third search parameter and call it 'append'. It is in fact non-existent with the payload structure.

Furthermore, based on the name 'postprocess_searches' parameter, it is clear only the base search gets distributable commands. All post-process searches (chained searches) happen on the searchhead only.

That is an important rule to keep in mind. If you want your search to be fast, then all the compute-heavy commands need to be in the base search.

Unfortunately, that means you'd need your base search to be a relatively large table of all sourcetypes appended together into a single table and do whatever necessary aggregation is required. Then use chained searches to slice and dice this large table into small bits, such as dividing by 'sourcetype' to branch out the table into multiple smaller "base" tables as the basis for additional chained searches.

In my case, I formulated my base search to be a merge of 3 different sourcetypes using stats join. It is reasonably fast with the ability to distribute the base search, despite having 15+ chained searches running off of it!

Re: [Studio JSON Source Code] Possible to append two ds.search or ds.chain together WITHOUT subsearches?

PickleRick — Tue, 04 Jun 2024 08:01:43 GMT

There's a portal for such feature requests - https://ideas.splunk.com/