Splunk Search

Why is my query terminating with "unexpected error"?

akasthi
New Member

Hello,

I am running a query to analyse 1 year of data and find out the number of users that used the application per day. But the below query is getting timeout and terminated with the error "unexpected error"

index=myIndex | dedup user_id _time | timechart span=1d dc(user_id) as Users | *outploutlookup ysers.csv

Could you please help with optimizing the above query ?

0 Karma
1 Solution

ashutoshab
Communicator

I see you are running the 'Dedup' command on a large data set with a huge time range. This is directly impacting your search performance and the query fails. Never run 'Dedup' Command directly over a search string. When you run a Dedup Command the text of every event in memory is retained which impacts your search performance.

If you run this search for a short time duration it might work and produce results. But if you run the same search over larger time duration such as 1 year, it will require to retain the text for a long time in the memory and eventually search will fail to complete.

This is the nature of the Dedup Command and this can not be an error. The dedup command is a streaming command or a dataset processing command, depending on which arguments are specified with the command.

To fix this, You have to modify your search to restrict only limited dataset to be pulled out. There are multiple ways to modify your search based on your data and make the search fast.

View solution in original post

0 Karma

ashutoshab
Communicator

I see you are running the 'Dedup' command on a large data set with a huge time range. This is directly impacting your search performance and the query fails. Never run 'Dedup' Command directly over a search string. When you run a Dedup Command the text of every event in memory is retained which impacts your search performance.

If you run this search for a short time duration it might work and produce results. But if you run the same search over larger time duration such as 1 year, it will require to retain the text for a long time in the memory and eventually search will fail to complete.

This is the nature of the Dedup Command and this can not be an error. The dedup command is a streaming command or a dataset processing command, depending on which arguments are specified with the command.

To fix this, You have to modify your search to restrict only limited dataset to be pulled out. There are multiple ways to modify your search based on your data and make the search fast.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You don't need that dedup command in there.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

@akasthi

Does your search work fine with different time range? like last 7 Days, last 30 days, last 2months, etc

0 Karma

akasthi
New Member

Yes, it works for the fewer time range, say 30 days, 7 days, etc

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Can you please inspect Job for that??

  1. Run the search.

  2. From the Job menu, select Inspect Job.

https://docs.splunk.com/Documentation/Splunk/7.2.5/Search/ViewsearchjobpropertieswiththeJobInspector

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...