scheduled report searches no longer perform similar to ad-hoc searches

[ErikStedin]

We use Splunk to report on daily smartGrid meter data.
We use 1 indexer, 1 searchhead and 1 heavy forwarder
We have observed that since the upgrade to 7.3.3 in december 2019 the results of scheduled searches no-longer contain all The expected fields/Field values.
When running these same searches manually we do see all the fields and field values.
These Searches are used in population of kvstore lookup tables which do not get populated properly. The search ends with a collect to a kvstore and returns around 3mln records.

Allthough our search query and volumes are large (3M records) according to the search logs there are no errors and they complete succesfully.

[woodcock]

Are you certain that your manual run is in the same app context as the scheduled search? If you go to Settings -> Searches, Reports, and Alerts and search for yours and then click Open in search, does it run correctly? My suspicion is that it will not. If not, then the problem is that it needs some application's Knowledge Objects that exist in some other app and which do not have global permissions. If so, then the solution is to move the scheduled search into that other application OR upgrade the required Knowledge Objects to global.

[woodcock]

The documentation is somewhat ambiguous but most of us believe that scheduled and dashboard searches run in smart mode which means that if you do not explicitly use a field name in your search, it will get optimized out. The easy way out of this is to simply add a final |table All of the fields that you care about listed here in the order that you prefer to the end.

[ErikStedin]

When you look at the search I posted you can see it ends with a |table to list all the fields so I do use a table. When scheduled, and I look at the results I find some fields/columns complete empty, no values.
When run it manualy, I do see the values in the fields.

[richgalloway]

Please explain how the actual results differ from the expected results. Share your search, if you can.

[ErikStedin]

Some fields are empty with values if run in a schedule.
If run manualy the are not empty, the exact same search.

The search:

((index=dc_cts sourcetype=submeter description!="Meter is verwijderd") OR
(index=dc_cts sourcetype=meterlocation endTs=9223372036854775807 (smartMeterType=D* OR smartMeterType=SLM)) OR
(index=dc_cts sourcetype=meter removedTs=0) OR
((index=dc_mfe* OR index=dc_xfe) sourcetype=submeter) OR
((index=dc_mfe* OR index=dc_xfe) sourcetype=meter) OR
(index=dc_mfe*smr* sourcetype=submeterKey))
|rex "^[^|\n]|(?P[^|]+)"
|rex "^(?:[^|\n]|){2}(?P[^ ]+)"
|eval equipmentId=if(isnotnull(equipmentId),equipmentId,serialNumber)
|eval submeterId=if(sourcetype="submeter" AND index="dc_cts",id,submeterId)
|eval meterLocationId=if(sourcetype="meterlocation",id,if(isnull(meterLocationId),NULL,meterLocationId))
|eval ctsMeterId=if(sourcetype="submeter" AND index="dc_cts",meterId,NULL)
|eval mfeMeterId=if(sourcetype="submeter" AND (like(index,"dc_mfe%") OR like(index,"dc_xfe")),meterId,NULL)
|eval submeterMfeId=if(sourcetype="submeter" AND like(index,"dc_mfedsmr%"),id,NULL)
|eval submeterState=if(sourcetype="submeter" AND index="dc_cts",state,NULL)
|eval locationState=if(sourcetype="meterlocation" AND index="dc_cts",state,NULL)
|eval meterState=if(sourcetype="meter" AND index="dc_cts",state,NULL)
|eval submeterMfeState=if(sourcetype="submeter" AND like(index,"dc_mfedsmr%"),state,NULL)
|eval meterMfeState=if(sourcetype="meter" AND like(index,"dc_mfe%"),state,NULL)
|eval submeterLastModifiedTs=if(sourcetype="submeter" AND index="dc_cts",lastModifiedTs,NULL)
|eval meterLastModifiedTs=if(sourcetype="meter" AND index="dc_cts",lastModifiedTs,NULL)
|eval locationLastModifiedTs=if(sourcetype="meterlocation" AND index="dc_cts",lastModifiedTs,NULL)
|eval submeterMfeLastModifiedTs=if(sourcetype="submeter" AND like(index,"dc_mfedsmr%"),lastModifiedTs,NULL)
|eval meterMfeLastModifiedTs=if(sourcetype="meter" AND like(index,"dc_mfe%"),lastModifiedTs,NULL)
|eval submeterDescription=if(sourcetype="submeter" AND index="dc_cts",description,NULL)
|eval locationDescription=if(sourcetype="meterlocation" AND index="dc_cts",description,NULL)
|eval submeterEquipmentId=if(sourcetype="submeter" AND index="dc_cts",equipmentId,NULL)
|eval locationEquipmentId=if(sourcetype="meterlocation" AND index="dc_cts",equipmentId,NULL)
|eval lastCommunicationTsCTSsub=if(sourcetype="submeter" AND index="dc_cts",lastCommunicationTs,NULL)
|eval lastCommunicationTsMFEmtr=if(sourcetype="meter" AND (like(index,"dc_mfe%") OR index="dc_xfe"),lastCommunicationTs,NULL)
|eval lastCommunicationTsMFEsub=if(sourcetype="submeter" AND (like(index,"dc_mfe%") OR index="dc_xfe"),lastCommunicationTs,NULL)
|eval phoneNumber=if(index="dc_mfesmr5" AND networkType="GPRS",networkId, if(index="dc_mfesmr5" AND networkType="CDMA",ipv4, if(index="dc_mfedsmr40" AND networkType="GPRS",networkId, if(index="dc_mfedsmr40" AND networkType="CDMA",ipv4,phoneNumber))))
|eval supplyModeTs=if(sourcetype="submeter" AND index="dc_cts",supplyModeTs,NULL)
|eval supplyTypeMeter=if(sourcetype!="meterlocation",supplyType,NULL)
|eval supplyTypeLocation=if(sourcetype="meterlocation",supplyType,NULL)
|eval supplyTypeSub=if(sourcetype="submeter" AND index="dc_cts",supplyType,if(sourcetype="submeter" AND like(index,"dc_mfe%") AND TYPE="ELECTRA","EL",if(sourcetype="submeter" AND like(index,"dc_mfe%") AND
TYPE="GAS",TYPE,NULL)))
|eval supplyType=if(sourcetype="meter","EL", if(isnotnull(supplyTypeMeter),supplyTypeMeter, if(isnotnull(supplyTypeLocation),supplyTypeLocation, if(isnotnull(supplyTypeSub),supplyTypeSub, if(sourcetype="submeter" OR sourcetype="meter", if(isnotnull(phoneNumber),"EL","GAS"), if(sourcetype="submeterkey" AND isnotnull(keyType), if(substr(keyType,len(keyType),1)="E","EL","GAS"),"NA"))))))
|eval keyString=keyType+","+reneuwTs+","+ageIndex
|eval initTs=if(sourcetype="submeter",initTs,NULL)
|eval cityName=mvindex(split(cityName," deployment"),0)
|eval dsmrVersion = if(index="dc_mfedsmr22","20200",mvindex(dsmrVersion,mvcount(dsmrVersion)-1))
|eval equipmentId = if(length(equipmentId)>10,substr(equipmentId,len(equipmentId)-11,10), if(length(equipmentId)<10,substr(tostring(pow(10,10-length(equipmentId))),2,10-length(equipmentId))+equipmentId, equipmentId))
|table equipmentId submeterEquipmentId locationEquipmentId *Id *State *LastModifiedTs *Description supplyTypeMeter appliance networkType phoneNumber supplyTypeLocation supplyType supplyTypeSub lastCommunicationTs* initTs errorCode zipCode switchProfile street startTs startDeploymentTs smartMeterType physicalState nextProcessingTs locationEan houseNumber deploymentState cityName administrativeOn nextCouplingTs nextActionTs mfeAction mbusMeterStatus deviceVersion deviceType communicationInterval logicalDeviceName loglevel modemVersionNumber dsmrVersion receivedInvocationCounter transmitInvocationCounter
|inputlookup meterTypeFullMeterGeneration.csv append=true
|stats values(*) AS * BY equipmentId supplyType
|eval equipmentId=if(isnotnull(submeterEquipmentId) AND isnotnull(locationEquipmentId) AND len(submeterEquipmentId)>len(locationEquipmentId),submeterEquipmentId, if(isnotnull(equipmentId) AND isnotnull(locationEquipmentId) AND len(equipmentId)>len(locationEquipmentId),equipmentId, if(isnull(locationEquipmentId),equipmentId,locationEquipmentId)))
|where isnotnull(meterLocationId) OR isnotnull(meterId)
|eval equipmentId=if(isnotnull(submeterEquipmentId),submeterEquipmentId,if(isnotnull(locationEquipmentId),locationEquipmentId,equipmentId))
|eval lastCommunicationTsCTSsub=mvindex(lastCommunicationTsCTSsub,mvcount(lastCommunicationTsCTSsub)-1)
|eval lastCommunicationTsMFEsub=mvindex(lastCommunicationTsMFEsub,mvcount(lastCommunicationTsMFEsub)-1)
|eval lastCommunicationTsMFEmtr=mvindex(lastCommunicationTsMFEmtr,mvcount(lastCommunicationTsMFEmtr)-1)
|fields - *EquipmentId manufacturerId meterLocation
|rename locationEan AS eanCode
|rename supplyTypeMeter AS supplyType
|metertype(equipmentId)
| eval meterTypeFull = if(isnull(meterTypeFullTwo), meterTypeFull, meterTypeFullTwo)
|rename supplyType AS supplyTypeMeter
|eval check=1
|table equipmentId meterTypeFull appliance networkType phoneNumber lastCommunicationTsCTSsub lastCommunicationTsMFEsub lastCommunicationTsMFEmtr initTs receivedInvocationCounter transmitInvocationCounter smartMeterType deviceVersion deviceType submeterDescription mbusMeterStatus supplyTypeMeter eanCode zipCode houseNumber street cityName startTs locationDescription supplyTypeLocation administrativeOn switchProfile deploymentState startDeploymentTs locationState meterMfeState physicalState submeterMfeState submeterState locationLastModifiedTs meterLastModifiedTs meterMfeLastModifiedTs submeterLastModifiedTs submeterMfeLastModifiedTs nextCouplingTs nextProcessingTs mfeAction nextActionTs communicationInterval logicalDeviceName loglevel meterLocationId mfeMeterId submeterId ctsMeterId submeterMfeId dsmrVersion