topic Re: Transforming commands no more work after upgrade to Splunk 8 in Splunk Search

Why do transforming commands not work after upgrade to Splunk 8?

sistemistiposta — Thu, 17 Feb 2022 16:14:29 GMT

Hello,

I have recently upgraded from Splunk 7 to Splunk 8.2.4.

After the upgrade, I noticed that some transform commands such as chart or stats do not work in smart and fast mode.

For instance:

index=main | chart count by host

returns the expected results in detailed mode. It returns 0 results in smart and fast mode.

Ps:

The transaction command still works, but I have to select the fields I want with fields in place of table. In Splunk 7 table works too.

I would like that stats and chart commands still work in fast search mode, as it happened in Splunk 7. Could you help me to revert the Splunk 7 working mode?

Thank you very much

Kind Regards

Marco

Re: Transforming commands no more work after upgrade to Splunk 8

gcusello — Thu, 17 Feb 2022 11:54:13 GMT

Hi @sistemistiposta,

I have Splunk 8.2.4 and commands as chart or stats run with the host field.

Are you meaning that you haven't results in the Events tab or that you haven't resuls in Events and in Statistics tab?

Obviously being chart a streaming command I have results in the Statistics Tab in all the modes (Fast. Smart and Verbose) but I have events only in Verbose in Events Tab.

Ciao.

Giuseppe

Re: Transforming commands no more work after upgrade to Splunk 8

isoutamo — Thu, 17 Feb 2022 11:56:24 GMT

probably something related your other configurations / your environment as this works ok in my environment.

When you you have run this what you found from Job inspector's search.log or Job Details Dashboard (easier to read)?

I got e.g.

r. Ismo

Re: Transforming commands no more work after upgrade to Splunk 8

sistemistiposta — Thu, 17 Feb 2022 15:41:59 GMT

Hello,

thank you for the interesting. It's not easy to debug, because I found it happens only sometime. I can see this with timechart in place of chart. See here:

detailed search: it works always

fast search: it works sometime during time

I can't understand why.

The search log doesn't show errors:

normalizedSearch	litsearch index=main \| addinfo type=count label=prereport_events \| fields keepcolorder=t "_time" "prestats_reserved_" "psrsvd_" \| prebin _time span=rtspan \| prestats count by _time
numPreviews	8
optimizedSearch	\| search index=main \| timechart count
phase0	litsearch index=main \| addinfo type=count label=prereport_events track_fieldmeta_events=true \| fields keepcolorder=t "_time" "prestats_reserved_" "psrsvd_" \| prebin _time span=rtspan \| prestats count by _time
phase1	timechart count
pid	674509
priority	5
provenance	UI:Search
reduceSearch	bin _time span=rtspan \| sistats count AS "count" by _time
remoteSearch	litsearch index=main \| addinfo type=count label=prereport_events track_fieldmeta_events=true \| fields keepcolorder=t "_time" "prestats_reserved_" "psrsvd_" \| prebin _time span=rtspan \| prestats count by _time
reportSearch	timechart count

Thank you very much

Kind Regards

Marco

Re: Transforming commands no more work after upgrade to Splunk 8

sistemistiposta — Thu, 17 Feb 2022 15:45:10 GMT

Ciao Giuseppe,

thank you for the reply. Yes I know that chart or stats commands hide events in fast or smart mode. It's not here the problem.

Thank you

Kind Regards

Marco

Re: Transforming commands no more work after upgrade to Splunk 8

isoutamo — Thu, 17 Feb 2022 15:49:39 GMT

This is quite interesting 😞

How about if you are adding

index=main host=* | chart count by host ```or timechart ...```

This should found all events where host is defined (which should be true for all events).

Is this issue only in main and some special sourcetype or for all indexes and source types?

Maybe it's time for splunk support to find if this is bug or some configuration issue?

r. Ismo

Re: Transforming commands no more work after upgrade to Splunk 8

sistemistiposta — Thu, 17 Feb 2022 16:00:36 GMT

Oh yes, I tried the by-clause. It's the same: your search

index=main host=* | timechart count by host

reports results by hosts until 13:20 only in "fast" mode.

Yes, I tried other analogue searches changing index and sourcetype: it's the same for all them.

Thank you very much

Kind Regards

Marco

Re: Transforming commands no more work after upgrade to Splunk 8

isoutamo — Thu, 17 Feb 2022 17:17:35 GMT

How about changing host to e.g. source? Did it work then or not?

Just try to wondering if issue is your host definition (if you have any props/transforms which override default host definition).

Are you sure that there is ingested events or could there be some delays?

Re: Transforming commands no more work after upgrade to Splunk 8

sistemistiposta — Fri, 18 Feb 2022 07:55:33 GMT

Hello,

yes, if I count by source the statistics stop at 13:20 too.

There are events, of course, because if I search in detailed mode I see the full statistics.

I can try to ask for the official support... I can't understand how to debug... The search log doesn't show me errors or timeouts...

Thank you

Marco

Re: Transforming commands no more work after upgrade to Splunk 8

landster — Thu, 24 Feb 2022 22:02:26 GMT

I'm curious if you have a resolution yet?

I recently came across a different, but similar issue and it was resolved when I added "| dedup _raw" to my code. I am still uncertain why it made a difference though.

Failed:
index=myindex eventName=myevent hostName=myhostname
| timechart limit=0 span=1h avg(value) as value by eventType
| table...

Success:
index=myindex eventName=myevent hostName=myhostname
| dedup _raw
| timechart limit=0 span=1h avg(value) as value by eventType
| table...

Re: Transforming commands no more work after upgrade to Splunk 8

gcusello — Fri, 25 Feb 2022 06:25:04 GMT

Hi @sistemistiposta,

tel me if I can help you more, or, please accept an answer for the other people of Community.

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Transforming commands no more work after upgrade to Splunk 8

gcusello — Fri, 25 Feb 2022 06:28:25 GMT

Hi @landster,

this is adifferent situation because , with your solution, you don't consider in your results the duplicated events you have, my hont is to try to understand why you have duplicated events!

Anyway, it's better to put this question in a separated post so more people can help you to solve your problem.

Ciao.

Giuseppe

Re: Transforming commands no more work after upgrade to Splunk 8

sistemistiposta — Fri, 25 Feb 2022 08:00:04 GMT

Hello @landster,

yes, If I add "| dedup _raw" it works!

but I suspect this could be a kind of trick in order to make the search as verbose.

It's like you add "| fields *" in fast mode. You really perform a smart search.

Adding

| dedup _raw

to all my search unfortunately is not a solution for me... and sometimes I could have expected duplicated events (such as when I use mvcombine). I haven't found a solution yet.

This doesn't happen (or I haven't never noticed) in Splunk 7.x.

Thank you for all the hints.

Marco

Re: Transforming commands no more work after upgrade to Splunk 8

isoutamo — Fri, 25 Feb 2022 09:34:51 GMT

Definitely you should create a support case for this even you can avoid it with that dedup as it's only a workaround.

Re: Transforming commands no more work after upgrade to Splunk 8

landster — Fri, 25 Feb 2022 13:21:56 GMT

Hello @sistemistiposta,

It is not a permanent solution for me either, and I have a support ticket opened with Splunk. I will let you know what i find out. I was mainly curious if we were experiencing the same problem.

Re: Transforming commands no more work after upgrade to Splunk 8

sistemistiposta — Fri, 25 Feb 2022 13:28:51 GMT

Aaaah yes, it seems the same issue.

Many days ago I opened a support ticket, as @isoutamo suggested.

We will see...

Kind Regards

Ciao

Marco

Re: Transforming commands no more work after upgrade to Splunk 8

sistemistiposta — Tue, 01 Mar 2022 13:49:03 GMT

Hello @landster and all community,

I noticed that also verbose searches can't work, if they contain a subsearch with transforming commands.

Ie:

[ search host=alice* | stats count by host | fields host ] ...

could not work. Instead

[ search host=alice* | dedup host | fields host ] ...

always works.

Just a question. Maybe it is not relevant. Do you have already upgraded the kvstore to wiredTiger?

This change and the upgrade to Splunk 8.2.5 are my only upgrade which I have done recently.

I'm still waiting news from Splunk support.

Thank you

Marco

Re: Transforming commands no more work after upgrade to Splunk 8

landster — Tue, 01 Mar 2022 14:15:44 GMT

Hello @sistemistiposta,

Yes, we completed that update immediately following the update to 8.2.5. That is interesting... It also seems the issue may exist beyond transforming commands as I have seen it on a simple table command. I am also waiting eagerly on a response.

Re: Transforming commands no more work after upgrade to Splunk 8

landster — Tue, 01 Mar 2022 15:50:01 GMT

@sistemistiposta

We upgraded the storage engine on both our test instance and our production instance, along with the update to 8.2.5. Only the production environment is exhibiting the symptom, however.

Re: Transforming commands no more work after upgrade to Splunk 8

sistemistiposta — Tue, 01 Mar 2022 16:16:37 GMT

@landster , the same for me.

In our test environment I didn't notice the problem.

In production we have a large amount of data, so I could suspect that also the data volume can affect this issue.

At this point I hope for news from support...

Kind Regards

Marco