Splunk Search

Strange behaviour when using dedup.

iet_ashish
Explorer

I have this query which when I run,

index=*aws_config* resourceType=TERM("AWS::EC2::Volume") 
| search ARN="arn:aws:ec2:eu-west-1:848889366260:volume/vol-0ecf419c9cd71857c" 
| table ARN, "tags.Genie.ArchPath" 
| dedup ARN

gives following results. Notice that value of field "tags.Genie.ArchPath" is blank. This is what I expect

+-----------------------------------------------------------------+---------------------+
|                               ARN                               | tags.Genie.ArchPath |
+-----------------------------------------------------------------+---------------------+
| arn:aws:ec2:eu-west-1:848889366260:volume/vol-0ecf419c9cd71857c |                     |
+-----------------------------------------------------------------+---------------------+

However when change the query so that dedup is called earlier, I get strange results,

index=*aws_config* resourceType=TERM("AWS::EC2::Volume") 
| dedup ARN
| table ARN, "tags.Genie.ArchPath"
| search ARN="arn:aws:ec2:eu-west-1:848889366260:volume/vol-0ecf419c9cd71857c"

The results I get are as shown below. Now the value of field "tags.Genie.ArchPath" is not blank. It strangely is a pipe(|) separated concatenation of source, host and sourcetype.

+----------------------------------+--------------------------------------------------+
|               ARN                |               tags.Adobe.ArchPath                |
+----------------------------------+--------------------------------------------------+
| arn:aws:ec2:eu-west-1:8488893662 |                                                  |
| 60:volume/vol-0ecf419c9cd71857c  | source::mavl://adobe-mavlink-prod-confi          |
|                                  | g/AWSLogs/848889366260/Config/eu-west-           |
|                                  | 1/2020/4/26/ConfigSnapshot/84888936626           |
|                                  | 0_Config_eu-west-1_ConfigSnapshot_2020           |
|                                  | 0426T110637Z_6375f945-8932-4196-ab9f-27          |
|                                  | 1c3333c55a.json.gz|host::840136feca32|aws:config |
+----------------------------------+--------------------------------------------------+

I fail to understand shy this is happening. Ideally both the queries should give same results. Would really appreciate if someone can help here.

Thanks,
Ashish

0 Karma

to4kawa
Ultra Champion

In table(Statistics): dedup keep Top
In search(Events): dedup keep First

If there is or not the field, It works differently.

0 Karma

PavelP
Motivator

Hello @iet_ashish ,

what happens if you use tags_Genie_ArchPath instead of tags.Adobe.ArchPath ? By default any field keys which contain dot in the name are being "cleaned":

CLEAN_KEYS = [true|false]
* NOTE: This setting is only valid for search-time field extractions.
* Optional. Controls whether Splunk software "cleans" the keys (field names) it
  extracts at search time. "Key cleaning" is the practice of replacing any
  non-alphanumeric characters (characters other than those falling between the
  a-z, A-Z, or 0-9 ranges) in field names with underscores, as well as the
  stripping of leading underscores and 0-9 characters from field names.
* Add CLEAN_KEYS = false to your transform if you need to extract field
  names that include non-alphanumeric characters, or which begin with
  underscores or 0-9 characters.
* Default: true

Let me know if it worked for you

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...