Splunk Search

Help with spath aws:metadata sorucetype tag.key{} vs Tag.value{}.

youngsuh
Contributor

youngsuh_0-1664595933183.png

 

 

 

 

index=aws sourcetype="aws:metadata" InstanceId=i-* 
| spath Tags{}.key.Name output=Hostname
| mvexpand Hostname
| fieldsummary 
| search field = Hostname

 

 

 

 

The above search give me count of value instead the value itself.  What I am missing? Tag &  AmiLaunchIndex is at same level right?  Splunk extracts "Tags{}.Key"=Name, AmiLaunchIndex at the INTERESTING FIELDS.  I really want to learn spath.  I Know how to do with regex.  I read the documentation but, it doesn't make sense to me.

 

 

| spath Tags{5}.Key output=HN 

 

 

Give me values at the key level but not at the Name values

Labels (2)
Tags (5)
0 Karma
1 Solution

yuanliu
SplunkTrust
SplunkTrust

I get the feeling that the use of fieldsummary is adding to your confusion, as I cannot find how this command is useful if all you want are the actual values.  You thought process is perhaps also thrown off by the node names such as "Key" and "Value", and node values such as "Name".  You'll get used to such when you work with JSON objects more.

Back to the structure of your data. (You should post some sample data - anonymize as needed, in raw JSON text form, preferably with pretty print.  Screenshot is not a good way of sharing.)  In pretty print (as the Splunk screenshot shows), you data is structured like this:

{
 "Attribute1": "Attrib1value",
 "Attribute2": Attrib2value",
 ...
 "Tags": [
  {
   "Key": "Somekey1",
   "Value": "Somevalue1"
  },
  {
   "Key": "Somekey2",
   "Value": "Somevalue2"
  },
  {
   "Key": "Name",
   "Value": "Somename"
  },
  ...
 ]
}

In this structure, Tag{}.Key has no further path.  It is a "leaf node", same as Tag{}.Value, same as Attribute1, Attribute2, and so on.  The string "Name" is the value of Tag{2}.Key, same as "Somename" is the value of Tag{2}.Value.  I have never used aws data.  But if I may speculate, there are other keys under Tag that do not say "Name"?  To further this speculation, the node where Tag{2}.Key's value is "Name" holds the name of the device.  In the above example, that node is Tag{2}.

With JSON, there is no predetermined index number in which the key "Name" appears, so you will need to to do some filtering.  You can do this in one of two ways.  Let's first use mvfind and mvindex.

 

index=aws sourcetype="aws:metadata" InstanceId=i-*
``` | spath can be ignored when Splunk is already been treating data as JSON ```
| eval namenode = mvfind('Tags{}.Key', "^Name$")
| eval Hostname = mvindex('Tags{}.Value', namenode)

 

 In the above, mvfind looks through all values of Tags{}.Key to determine which one, if any, matches the string "Name", and returns the index of that item to become the value of namenode (in our example, it will have value 2); mvindex, then, takes the value of Tags{}.Value at that index. (In our example, it will have value "Somename".)

A more common technique employs mvexpand.

| spath path=Tags{}
| mvexpand Tags{}
| spath input=Tags{}
| where Key == "Name"
| rename Value as Hostname

The first spath command extracts the array Tags{} as a unitary field; after this extraction, each Tags{} value is still a JSON object that you cannot directly dereference.  That's why after mvexpand, you run a second spath using this field as input in order to extract Key and Value into their own fields so you can use "ordinary" search commands.

Hope this helps.

View solution in original post

yuanliu
SplunkTrust
SplunkTrust

Field name in SPL uses dot (".") to segment data paths in JSON, and curly brackets ({}) to represent a JSON array.  Tags{5}.Key means the "Key" node off the 5th (0-based) node of the array Tags.  Tags{}.Key, on the other hand, gives you a multivalue field (think of it as array, too) of all "Key" nodes of the array Tags.  I think Tags{}.Value in your data represents actual host names that you masked out in the screenshot.  So,

 

index=aws sourcetype="aws:metadata" InstanceId=i-* 
| spath Tags{}.Value output=Hostname
| mvexpand Hostname
| fieldsummary 
| search field = Hostname

 

If you work with another language, you can compare how that language represents JSON objects. (For example, you can think of spath as json.loads() in Python.  After this, you then use its nested dictionaries and arrays to work on the data.)

A good way to learn is to run spath without a path argument and observe how Splunk flattens nested data structure.  For example,

 

index="_introspection"
| table *

 

(You don't need spath to extract base data from _introspection index because JSON data is already extracted in this index.) If you use a path argument with spath, you will extract the value of that particular path.

youngsuh
Contributor

@yuanliu Thanks for quick response.  It make more sense now.  The challenge now is the extract the array value on Tags{Name}.Key bring up the count of the values but, not nested values within the Name Field that has the value We want

 

index=aws sourcetype="aws:metadata" InstanceId=i-* 
| spath Tags{}.Value output=Hostname
| mvexpand Hostname
| fieldsummary 
| search field = Hostname

 

Return the results on the screen shot.  Notice that the count of value and field values mixed.  that's the part I don't get and trying to understand.  Hostname only returns the count.

youngsuh_0-1664639540877.png

"Field name in SPL uses dot (".") to segment data paths in JSON, and curly brackets ({}) to represent a JSON array. "  So, should the path Tags.Key.Name{} return the values for Name array not the count of the values

 

| spath Tags.Key.Name{} output=Name

 

 Yes.  I am having problem understanding the path logic based on the results.  The question is how do retrieve the values in 3 depth array into muti-value?

 

index=aws sourcetype="aws:metadata" InstanceId=i-* 
| spath path="Tags{}.Value" output=Name 
| mvexpand Name 
| search Name = I-*
| stats count by Name

 

This get the results in Value but, my understanding of it still rough.  I need extract more tag.values{} in tag.values as fields with value instead of the values to re-do a savesearch that rely tag.$field_name$  Now it's broken out into Value being the field value where key is the field name.  I think, I need to group them.  Right?

0 Karma

yuanliu
SplunkTrust
SplunkTrust

I get the feeling that the use of fieldsummary is adding to your confusion, as I cannot find how this command is useful if all you want are the actual values.  You thought process is perhaps also thrown off by the node names such as "Key" and "Value", and node values such as "Name".  You'll get used to such when you work with JSON objects more.

Back to the structure of your data. (You should post some sample data - anonymize as needed, in raw JSON text form, preferably with pretty print.  Screenshot is not a good way of sharing.)  In pretty print (as the Splunk screenshot shows), you data is structured like this:

{
 "Attribute1": "Attrib1value",
 "Attribute2": Attrib2value",
 ...
 "Tags": [
  {
   "Key": "Somekey1",
   "Value": "Somevalue1"
  },
  {
   "Key": "Somekey2",
   "Value": "Somevalue2"
  },
  {
   "Key": "Name",
   "Value": "Somename"
  },
  ...
 ]
}

In this structure, Tag{}.Key has no further path.  It is a "leaf node", same as Tag{}.Value, same as Attribute1, Attribute2, and so on.  The string "Name" is the value of Tag{2}.Key, same as "Somename" is the value of Tag{2}.Value.  I have never used aws data.  But if I may speculate, there are other keys under Tag that do not say "Name"?  To further this speculation, the node where Tag{2}.Key's value is "Name" holds the name of the device.  In the above example, that node is Tag{2}.

With JSON, there is no predetermined index number in which the key "Name" appears, so you will need to to do some filtering.  You can do this in one of two ways.  Let's first use mvfind and mvindex.

 

index=aws sourcetype="aws:metadata" InstanceId=i-*
``` | spath can be ignored when Splunk is already been treating data as JSON ```
| eval namenode = mvfind('Tags{}.Key', "^Name$")
| eval Hostname = mvindex('Tags{}.Value', namenode)

 

 In the above, mvfind looks through all values of Tags{}.Key to determine which one, if any, matches the string "Name", and returns the index of that item to become the value of namenode (in our example, it will have value 2); mvindex, then, takes the value of Tags{}.Value at that index. (In our example, it will have value "Somename".)

A more common technique employs mvexpand.

| spath path=Tags{}
| mvexpand Tags{}
| spath input=Tags{}
| where Key == "Name"
| rename Value as Hostname

The first spath command extracts the array Tags{} as a unitary field; after this extraction, each Tags{} value is still a JSON object that you cannot directly dereference.  That's why after mvexpand, you run a second spath using this field as input in order to extract Key and Value into their own fields so you can use "ordinary" search commands.

Hope this helps.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...