the search job terminated unexpectedly

[srobinsonxtl]

This search take only a few second to come back index=* sourcetype=* (source="/opt/data/-AA_.csv" OR source="/opt/data2/-AA_.csv") | fields - field1 and return 81,000 records but once I add below to the search I get the search job terminated unexpectedly.

index=* sourcetype=* (source="/opt/data/-AA_.csv" OR source="/opt/data2/-AA_.csv") | fields - field1 | timechart span=5min sum(field*) as AA* | addtotals | table _time,Total | timechart span=1h max(Total) as Total | eval Total = Total/1000 | timechart span=1mon sum(Total) as Total

Here is a sample of the data and is written every 5 minutes. When I was writing the data every 15 minutes, it seem to work ok.
2017-12-31 23:55:00.001+00:00,695,0,733,0,817,0,1078,0,987,0,1004,0,1983,0,1744,0,1236,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

Any help to help improve my search is very much appreciated.

Thanks,

Stephen Robinson

[srobinsonxtl · Accepted solution]

This appeared to improve my search dramatically, thank you for the help

Thanks,

Stephen Robinson

[mayurr98]

can you please write search query which is getting terminated in 101010 sample code for a better understanding of query. Also, i think you need to optimize your search query
give us sample input with the field name and what output you want?

[srobinsonxtl]

index= sourcetype= (source="/opt/data/-AA_.csv" OR source="/opt/data2/-AA_.csv") | fields - field1 | timechart span=5min sum(field*) as AA* | addtotals | table _time,Total | timechart span=1h max(Total) as Total | eval Total = Total/1000 | timechart span=1mon sum(Total) as Total

This provides me the output I want, but the job doesn't seem to efficient. The search creates fields AA1-121 per event and then does all the calculations it needs to do.

This is the sample of data that I have.
2017-12-31 23:55:00.001+00:00,695,0,733,0,817,0,1078,0,987,0,1004,0,1983,0,1744,0,1236,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2018-01-01 00:00:00.001+00:00,695,0,733,0,817,0,1078,0,987,0,1004,0,1983,0,1744,0,1236,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
2018-01-01 00:05:00.001+00:00,695,0,733,0,817,0,1078,0,987,0,1004,0,1983,0,1744,0,1236,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
....
Thanks,

Stephen Robinson

[somesoni2]

See if this helps

index= sourcetype= (source="/opt/data/-AA_.csv" OR source="/opt/data2/-AA_.csv") | fields - field1 
| eval Total=0 | foreach field* [eval Total=Total+'<<FIELD>>' ]
| timechart span=5min sum(Total) as Total | timechart span=1h max(Total) as Total | eval Total = Total/1000 | timechart span=1mon sum(Total) as Total