Where are the strange text artifacts included in search results coming from?

[bensec01]

Hey folks,

   Here's a weird one...  I just added a new data source (Windows share permissions) into our Splunk environment, and I'm working on some views to visualize this data for IT staff.

   This isn't rocket surgery - this is pretty simple.  Here's an example event which is created by a PowerShell script that runs every 12 hours on Windows systems:

2022-10-07 09:31:54 DataType="SharePermissions" ShareName="Users" Account="Everyone" Type="Allow" Right="Read"

   That's pretty simple.  However, with at least one system, I'm getting crazy data back when I search for it in the Splunk web UI:

`all_windows_index` sourcetype="PowerShell:SMBShares" host=my_hostname_here DataType="SharePermissions"
| stats values(SharePath) as SharePath list(Account) as Account list(Type) as Type list(Right) as Right by host ShareName
| search ( ShareName="Users" )
| search `filter_no_admin_shares`
| rename host as Server

  This should display a simple line, with each group or user and the rights they have on this share.  No witchcraft here...  But, when I run the search, in the visualization (a table with zero customizations), I get something like:

   (In the above, I intentionally cropped the hostname from the left side of the table's row)

   That text doesn't appear anywhere in the event.  The event looks exactly like the example given above, plain text, single words, nothing odd.  And what's even weirder, it's not consistent.  Here are three more refreshes *of exactly the same view*, no changes to inputs, one right after another.  One of them does the right thing.  The other two have more random artifacts:

   Between these refreshes, there were no changes in the data.  

   The text in these artifacts is obviously from Splunk (a lot of it looks like it comes from stuff I see in the job inspector), but it appears nowhere in the event itself, nor in the macros (simple index definitions or filters), nor in the SPL.  For some reason, Splunk is doing this itself, I have no idea why.

   I *have* restarted Splunk just to make sure something didn't go sideways on me...  This is Splunk Enterprise v8.2.4, on-prem.  I would LOVE it if someone could explain this behavior.  This is the first time I've seen this with *any* of my data.

   Help?

   Thanks so much!

Chris

[yuanliu]

Have you expanded all macros for testing?  Also, you can add index, sourcetype, etc., to groupby to help diagnosis

[bensec01]

I have expanded all macros, yes.  I added the remaining metadata to groupby, and it doesn't really point to a pattern either.  The only pattern I see is that it's always the 'Type' column that gets screwed up.  But the Account isn't consistent, nor any other things.  Some shares are hidden ('$' appended, some aren't).  Different Accounts (some are AD groups, some are users).  They're all the same index, but that's OK...

This one is driving me nuts.  Well, nuttier.  🙂 

[bensec01]

Oh, and this happens on multiple search heads too.  No SHC, just standalone search heads talking to a multisite indexing cluster.