subsearch using inputlookup and then search tags

[sunset25]

I want to select a team on a dashboard and feed that token into a search that will find all the applications they manage in a csv lookup file, then search tags fields in an index of 'metadata' to get all the logs related to those applications. The 

This is my search so far but its provdes blank results.
index=metadata
[

| inputlookup snow_sys_applications.csv
| where support_group="12345"    ```$token$ would go here```
| table u_cloud_domain
| format

]
| eval appdomain = mvfind('Tags{}.Key', "^Domain")          ```find the Domain tag```
| eval AppDomain = mvindex('Tags{}.Value', appdomain) ```find the value of the Domain tag```
| search AppDomain=u_cloud_domain                                 ``` use the value to search the subsearch results```
| rex field=source ".*:(?<awsMetaType>.*)"
| table AppDomain awsMetaType id

The subsearch I developed works separately and will provide a list a list of u_cloud_domain's
0001
0002
0003
0004
0005

The main search will work separately if a search for an AppDomain="0005".

The won't work together. I suspect because u_cloud_domain is a list of values and AppDomain is a single value.
I have tried with and without | format.
What search or where commands do I need to get them to work?
Also is there a way to see the output results from the subsearch to check them?

[sunset25]

The fields in  snow_sys_applications.csv are 
name, support_group, u_cloud_domain, dv_operational_status

the field u_cloud_domain is related to the domain tag in the indexed data.
Tags: [ [-]
{ [-]
Key: Domain
Value: 0005
}

The raw data is

"Tags": [{"Key": "CostCenter", "Value": "12345"}, {"Key": "Owner", "Value": "Smith"}, {"Key": "Availability", "Value": "Tier II A1b"}, {"Key": "AppEnv", "Value": "uat"},  {"Key": "Domain", "Value": "0005"}]

[PickleRick]

OK. So leaving the lookup aside for a while, the problem is still more complicated because you need to combine {"key":something,"value":other_something} into something=other_something.

And it's a non-trivial task.

Yes, I saw that you've been doing a mvfind/mvindex-based evals but they don't have to work properly if your data would have missing/empty fields so I'd approach it a bit differently.

I'd try something like that:

Start with the data from your index

index=metadata

Parse out the list into array of json structures

| spath path="Tags{}" output=tags

now we can find the single json object which has the proper key:

| eval Domain=mvfind(tags,mvindex(tags,"\"Key\"\s*:\s*\"Domain\""))

And now we can parse out our Domain from the "sub-json"

|  spath input=Domain path="Value" output=Domain

After this you'd have your Domain field properly parsed out.

Now there are two approaches you can do. Both will be similar in terms of performance since you have to firstly dig through all your data anyway.

One approach is to use a subsearch to generate a list of conditions for the Domain value

| search [ inputlookup snow_sys_applications.csv
     | where support_group="$your_token$" 
     | rename u_cloud_domain as Domain | table Domain ]

The alternative approach could be to use the lookup and get the support_group field from the lookup based on the u_cloud_domain field looked up from the Domain field and then filter with support_group condition.

Unfortunately, because your data is not well-structured, you can't do a simple search for Domain=something.

You can improve your search efficiency a bit by adding the u_cloud_domain values to the initial search (without any particular field to match) so that Splunk can limit its search only to those events which have this value anywhere. Depending on your data it might give a significant performance boost or none at all.

[sunset25]

I want to select a team on a dashboard and feed that token into a search that will find all the applications they manage in a csv lookup file, then search in an index of 'metadata' to get all the logs related to those applications with tags fields for 'Domain' and 'Valve' (example "0005" OR  "1290") .
This will give me a list of metadata (AWS services/objects) that the team own.

[PickleRick]

OK, wait, back up a little 🙂

What fields do you have in your lookup? And how those fields relate to your data from the index.

[PickleRick]

If I understand correctly what you're trying to do, you misunderstand how subsearches work.

Your subsearch is run first, it returns results and those results are substituted into the "outer" search.

So if your subsearch returns 5 values for u_cloud_domain, the outer search becomes

index=metadata (u_cloud_domain="0001" OR u_cloud_domain="0002" OR u_cloud_domain="0003" OR u_cloud_domain="0004" OR u_cloud_domain="0004")

So if you don't even have a field named u_cloud_domain in your events, results from this search will obviously be empty so subsequent operations won't matter.

[sunset25]

Thanks for the clarification that the field needs to exist.

The problem is that these 2 fields don't exist and the eval creates them.

| eval appdomain = mvfind('Tags{}.Key', "^Domain")

| eval AppDomain = mvindex('Tags{}.Value', appdomain)

The AWS tags are structured with a key and a value so you need to find the key and get the value.

Example

Tags: [ [-]
{ [+]
}
{ [-]
Key: BusinessRegion
Value: Global
}
{ [-]
Key: Domain
Value: 0005
}

There can be 20+ tags.keys and tags.values

So this search works

index=metadata
[
| inputlookup snow_sys_applications.csv
| where u_cloud_domain="0005" OR u_cloud_domain="1290"
| rename u_cloud_domain as Tags{}.Value
| table Tags{}.Value
]

| rex field=source ".*:(?<awsMetaType>.*)"
| table Tags{}.Value awsMetaType id

but its searching all the Tags.Value fields and there may be 0005 or 1290 in other tags.value fields.

Would doing an automatic lookup to extract the AppDomain so it appears as a separate field get round this issue?

[PickleRick]

OK, leave the subsearch for now.

Explain in your own words what you want to achieve.