After some more poking around it seems like the searches are NOT failing at all. They are running and completing but it just instantly times out when loading the search. If I go to Activity > Jobs and click any search I ran it gives me the results and everything works as expected. Its just the initial search that is causing this error. If I click a job that is not finished and still running it gives the same error but shows some results with the greyed out bottom portion (see screenshot above). It also says the job has failed in the activity while its running but once it finishes it changes to done.   

@ITWhisperer I want to compare two field values match.If does not match i want to colour both the fields. index="mulesoft" applicationName="s-concur-api" environment=PRD priority timestamp (tracePoint="EXCEPTION") OR ("Concur Ondemand Started*") OR (message="Expense Extract Process started for jobName :*") OR ("Before Calling flow archive-Concur*") OR (message="*(SUCCESS): Concur AP/GL Extract V*") OR (message="Records Count Validation*") OR (message="API: START: /v1/expense/extract/ondemand*" OR message="API: START: /v1/fin*") OR (message="Post - Expense Extract processing to Oracle*") OR (message="Concur AP/GL File/s Process Status") OR (message="*(ERROR):*") | search NOT message IN ("API: START: /v1/expense/extract/ondemand/accrual*") | spath content.payload{} | mvexpand content.payload{} | transaction correlationId | rename content.SourceFileName as SourceFileName content.JobName as JobName content.loggerPayload.archiveFileName AS ArchivedFileName content.payload{} as response content.Region as Region content.ConcurRunId as ConcurRunId content.HeaderCount as HeaderCount content.SourceFileDTLCount as SourceFileDTLCount content.APRecordsCountStaged as APRecordsCountStaged content.GLRecordsCountStaged as GLRecordsCountStaged content.TotalAPGLRecordsCountStaged as TotalAPGLRecordsCountStaged content.ErrorMsg as errorMessage content.errorMsg as error content.errorMsg as error "content.payload{}.AP Import flow processing results{}.requestID" as RequestID "content.payload{}.GL Import flow processing results{}.impConReqId" as ImpConReqId | rex field=message max_match=0 "Expense Extract Process started for (?<FileName>[^\n]+)" | rex field=message max_match=0 "API: START: /v1/expense/extract/ondemand/(?<OtherRegion>[^\/]+)\/(?<OnDemandFileName>\S+)" | eval OtherRegion=upper(OtherRegion) | eval OnDemandFileName=rtrim(OnDemandFileName,"Job") | eval "FileName/JobName"= coalesce(OnDemandFileName,JobName) | eval JobType=case(like('message',"%Concur Ondemand Started%"),"OnDemand",like('message',"%API: START: /v1/expense/extract/ondemand%"),"OnDemand",like('message',"Expense Extract Process started%"),"Scheduled") | eval Status=case(like('message' ,"%Concur AP/GL File/s Process Status%"),"SUCCESS", like('tracePoint',"%EXCEPTION%"),"ERROR") | eval Region= coalesce(Region,OtherRegion) | eval OracleRequestId=mvappend("RequestId:",RequestID,"ImpConReqid:",ImpConReqId) | eval CheckMatch = if(isnull(SourceFileDTLCount) OR isnull(TotalAPGLRecordsCountStaged), "not matched", "matched") | eventstats min(timestamp) AS Logon_Time, max(timestamp) AS Logoff_Time by correlationId | eval StartTime=round(strptime(Logon_Time, "%Y-%m-%dT%H:%M:%S.%QZ")) | eval EndTime=round(strptime(Logoff_Time, "%Y-%m-%dT%H:%M:%S.%QZ")) | eval ElapsedTimeInSecs=EndTime-StartTime | eval "Total Elapsed Time"=strftime(ElapsedTimeInSecs,"%H:%M:%S") | eval sign=if(SourceFileDTLCount == TotalAPGLRecordsCountStaged,"GREEN", "YELLOW") | rename Logon_Time as Timestamp | table Status JobType "FileName/JobName" Timestamp Region ConcurRunId HeaderCount SourceFileDTLCount APRecordsCountStaged GLRecordsCountStaged TotalAPGLRecordsCountStaged ArchivedFileName ElapsedTimeInSecs "Total Elapsed Time" OracleRequestId correlationId | join correlationId type=left [ search index="mulesoft" applicationName="s-concur-api" (message="*(SUCCESS): Concur AP/GL Extract V.3.02 - *. Concur Batch ID: * Company Code: * Operating Unit: *") | eval Message=case(like('message',"%(SUCCESS): Concur AP/GL Extract V.3.02 - %. Concur Batch ID: % Company Code: % Operating Unit: %"),message) | table Message correlationId ] | eval Response= coalesce(error,errorMessage,Message) | table Status JobType "FileName/JobName" Timestamp CheckMatch Region ConcurRunId HeaderCount SourceFileDTLCount APRecordsCountStaged GLRecordsCountStaged TotalAPGLRecordsCountStaged ArchivedFileName ElapsedTimeInSecs "Total Elapsed Time" sign OracleRequestId Response correlationId | fields - ElapsedTimeInSecs priority | where JobType!=" " | search Status="*"

It looks like there are no newlines between events so the LINE_BREAKER is not matching.  Try these settings: [iis] LINE_BREAKER = ([\r\n]*)\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} SHOULD_LINEMERGE = false TIME_PREFIX = ^ TIME_FORMAT = %Y-%m-%d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 19

If the system logs have been ingested into Splunk, you need to identify which events in those logs include the information you are looking for. You can then tell Splunk how to pull out those events so you can report on them in your dashboard. We do not have access to your data, it is only something that you can determine