Combining searches that look for a lack of data without using append

[hoangtony]

Hi All,

Hoping someone can point me in the right direction with this one.  The use case is there are some processes that I need to be checking if data is being written to their logs (that is the easy part) and I also need to note if there is a lack of data by host.

I used a lookup file that I add to the search in the scenarios where there is a potential issue and I need to indicate that the host has no data.  I managed to get it working and I've combined a number of different processes and use cases together into one search however I've used the append command.  Unfortunately the Splunk admins in my company do not allow for appends in any search (it's a big no no) regardless of the data size which in this case isn't large.  

This is what the search looks like currently:

index=test_index sourcetype=process_a_log "Success Message" earliest=-2h
| inputlookup append=t hosts.csv | fields host
| stats latest(_indextime) as indexedTime by host
| eval count=if(isnull(indexedTime),null,(now()-indexedTime))
| eval process="ProcessA"
| table host,process,count 
| append
    [ search index=test_index sourcetype=process_b_log "Generation completed" earliest=-1h
    | inputlookup append=t hosts.csv | fields host
    | stats latest(_indextime) as indexedTime by host
    | eval count=if(isnull(indexedTime),null,(now()-indexedTime))
    | eval process="ProcessB"
    | table host,process,count ]
| append
    [ search index=test_index earliest=-5m
    | inputlookup append=t hosts.csv | fields host
    | stats latest(_indextime) as indexedTime by host
    | eval count=if(isnull(indexedTime),null,(now()-indexedTime))
    | eval process="Data"
    | table host,process,count ]

I've omitted parts at the bottom where I do an evaluation on thresholds and output severity.

I attempted to do something like this:

index=test_index (sourcetype=process_a_log "Success Message" earliest=-2h) OR (sourcetype=process_b_log "Generation completed" earliest=-1h) OR (sourcetype=* earliest=-5m)
| inputlookup append=t hosts.csv | fields host
| stats latest(_indextime) as indexedTime by host
| eval count=if(isnull(indexedTime),null,(now()-indexedTime))
| eval process=case(
  match(_raw,"Success Message"),"ProcessA",
  match(_raw,"Generation completed"),"OrocessB",
  1=1,"Other")
| table host,process,count 

However it doesn't produce the outcome I require given all the events for all the processes are together and while it appends the host, I need it to append by process.  Basically I need something along the lines of 'inputlookup append=t by process' but unsure how to achieve it.

Any help would be greatly appreciated.  Thanks.

[bowesmana]

You seem on the right track with setting Process= to determine origin, although I would set the case statement as 

| eval process=case(
  match(_raw,"Success Message"),"ProcessA",
  match(_raw,"Generation completed"),"ProcessB",
  isnull(source),"Lookup",
  1=1,"Other")

If you put this before the stats command, then do the

| stats ... by host process 

BTW, the | fields host will remove everything other than _* and host, so would need to handle allowing process through.

Also, not sure that _raw is still valid after the stats, so that won't work.

 I don't know if that will get you to where you want to