topic Strange behaviour when using dedup. in Splunk Search

Strange behaviour when using dedup.

iet_ashish — Sun, 26 Apr 2020 12:34:00 GMT

I have this query which when I run,

index=*aws_config* resourceType=TERM("AWS::EC2::Volume") 
| search ARN="arn:aws:ec2:eu-west-1:848889366260:volume/vol-0ecf419c9cd71857c" 
| table ARN, "tags.Genie.ArchPath" 
| dedup ARN

gives following results. Notice that value of field "tags.Genie.ArchPath" is blank. This is what I expect

+-----------------------------------------------------------------+---------------------+
|                               ARN                               | tags.Genie.ArchPath |
+-----------------------------------------------------------------+---------------------+
| arn:aws:ec2:eu-west-1:848889366260:volume/vol-0ecf419c9cd71857c |                     |
+-----------------------------------------------------------------+---------------------+

However when change the query so that dedup is called earlier, I get strange results,

index=*aws_config* resourceType=TERM("AWS::EC2::Volume") 
| dedup ARN
| table ARN, "tags.Genie.ArchPath"
| search ARN="arn:aws:ec2:eu-west-1:848889366260:volume/vol-0ecf419c9cd71857c"

The results I get are as shown below. Now the value of field "tags.Genie.ArchPath" is not blank. It strangely is a pipe(|) separated concatenation of source, host and sourcetype.

+----------------------------------+--------------------------------------------------+
|               ARN                |               tags.Adobe.ArchPath                |
+----------------------------------+--------------------------------------------------+
| arn:aws:ec2:eu-west-1:8488893662 |                                                  |
| 60:volume/vol-0ecf419c9cd71857c  | source::mavl://adobe-mavlink-prod-confi          |
|                                  | g/AWSLogs/848889366260/Config/eu-west-           |
|                                  | 1/2020/4/26/ConfigSnapshot/84888936626           |
|                                  | 0_Config_eu-west-1_ConfigSnapshot_2020           |
|                                  | 0426T110637Z_6375f945-8932-4196-ab9f-27          |
|                                  | 1c3333c55a.json.gz|host::840136feca32|aws:config |
+----------------------------------+--------------------------------------------------+

I fail to understand shy this is happening. Ideally both the queries should give same results. Would really appreciate if someone can help here.

Thanks,
Ashish

Re: Strange behaviour when using dedup.

PavelP — Wed, 30 Sep 2020 05:13:40 GMT

Hello @iet_ashish ,

what happens if you use tags_Genie_ArchPath instead of tags.Adobe.ArchPath ? By default any field keys which contain dot in the name are being "cleaned":

CLEAN_KEYS = [true|false]
* NOTE: This setting is only valid for search-time field extractions.
* Optional. Controls whether Splunk software "cleans" the keys (field names) it
  extracts at search time. "Key cleaning" is the practice of replacing any
  non-alphanumeric characters (characters other than those falling between the
  a-z, A-Z, or 0-9 ranges) in field names with underscores, as well as the
  stripping of leading underscores and 0-9 characters from field names.
* Add CLEAN_KEYS = false to your transform if you need to extract field
  names that include non-alphanumeric characters, or which begin with
  underscores or 0-9 characters.
* Default: true

Let me know if it worked for you

Re: Strange behaviour when using dedup.

to4kawa — Sun, 26 Apr 2020 20:06:08 GMT

In table(Statistics): dedup keep Top
In search(Events): dedup keep First

If there is or not the field, It works differently.