Not sure if it would be more efficient, but maybe eventstats could be helpful here? index=abc | eventstats values(message) as all_messages by requestId | where sourcetype="payment" AND log_level="Error" AND NOT match(all_messages,"insufficient balance") | ...     ... View more

You may be able to do a subsearch, but not sure if that's needed here.  If you simply search the past 3 days (or however long in the past you need to go), then you should be able to filter only those events where step 6 occurred today.  Not looking to type it all out again, but something like this once you have all of the fields you need and are ready to actually check timestamps. ... | where step6_time > relative_time(now(),"@d")  That should eliminate any events where  step 6 happened before today.   ... View more

Not sure if I fully understand the requirements, but maybe something like this could get you started?  I can't really test so this could be all sorts of wrong - but I think it seems reasonable?  Include all events in base search  create some new vars based on the sourcetype (so kinda like the rename you want to do) get the detection time and process time for each snsm id (use eventstats to keep the rest of the event) get the step6 time for each prodnr  (think this should put it on the nau/nas events too) Now just look at the NAS events as they should have the detection time, process time and step 6 time and the prod nr as well check the timestamps to see which thing happened when and provide the relevant output index=whatever sourcetype IN ("MP","NAS","NAU") | eval step6_time = case(sourcetype="MP" AND meldepunkt="6",strptime(zeitstempeli,"%d.%m.%Y %H:%M")) | eval detection_time = case(sourcetype="NAS",strptime(errfaast,"%d.%m.%Y %H:%M")) | eval process_time = case(sourcetype="NAU",strptime(errfaast,"%d.%m.%Y %H:%M")) | eventstats values(detection_time) as detection_time, values(process_time) as process_time by snsm | eventstats values(step6_time) as step6_time by prodnr | where sourcetype="NAS" | eval result = case(isnotnull(process_time) AND process_time < step6_time,"fault processed before",isnotnull(process_time),"fault processed after step 6",isnull(process_time),"fault not processed")   ... View more

It's helpful if you post your search inside the code tags like below.  That may help you troubleshoot as well.  At a glance, I would say whatever is happening after your last eval is the problem.  Looks like maybe some tokens from the dashboard are just shoved in a random condition down there? And you can always start eliminating pipes until the error goes away which can help you determine which one is problematic.  For example, remove the timehart and run the search.  Then the last eval, run the search....then next to last eval and so on.   index="salesforce" source="/informatica/pmrootdir/Splunk_SFDC_Logs/*-eventlogfile-splunk.csv" host="louapplps2024" CASE(xxxxxxxxxxxxxx) earliest=-7d latest=now (EVENT_TYPE="ApexExecution") QUIDDITY IN (R,V,W,H,X,M,L,K,I,E) IS_LONG_RUNNING_REQUEST=1 | dedup REQUEST_ID | eval oraDbTime=IF(!isnull(sdbDbTime),round(oraDbTime/1000000,0),oraDbTime) | eval dbCpuTime=IF(!isnull(sdbDbTime),round(dbCpuTime/1000000,0),dbCpuTime) | eval sdbDbTime=round(sdbDbTime/1000000,0) | eval reqIsNull=IF(isnull(requestId) AND entryPoint="TRIGGERS","YES","NO") | where reqIsNull="NO" | eval permitTime= (runTime-5000) | eval permitTimeInSecs= (permitTime/1000) | eval runTimeInSecs= (runTime/1000) | eval endTime=_time | eval startTime=(_time-runTimeInSecs) | eval conTime=(_time-permitTimeInSecs) $org_token$ EVENT_TYPE="LightningPageView" OR EVENT_TYPE="LightningPerformance" APP_NAME="siteforce:communityApp" $userId$ $app$ | timechart span=$timespan$ dc(USER_ID) as Views   ... View more

Been out of the game for a couple years, so I am sure that others here can provide better/updated information, but I'll still give this a quick go.  First, if you can literally only have a single HF, then I don't think you can have H/A.  It just doesn't make any sense.  You may be able to to try to limit data loss but I don't think you can ever call it highly available.  And how much data loss you can avoid I think depends on what roles the HF is playing. I don't recall the details, but I believe a UF has queuing technology.  So if its target isn't available, it can queue the data.  How much and how long it can queue, I no longer know but I think it is configurable but limited.  So if it's a pretty busy UF (like a domain controller) and/or a long downtime, then likely there will be data loss. If the HF is a HEC endpoint, then it would be up to the code sending data to it to handle that logic I think.  But again, it is unlikely that would be able to wait say 12 hours to send data and likely have data loss again. If the HF is the source of data itself, then it would depend on where it is collecting data I think.  Meaning, if the source has rolled past its last checkpoint then there would be data loss.  If not, then it should be able to consume where it left off. I am curious what that recommendation is from Splunk to only have a single HF as alternate "h/a" strategy.   ... View more

It's been a minute since I've been a splunk admin, so fairly rusty here....but I think you might want something like this in your props.   [your_sourcetype] MAX_TIMESTAMP_LOOKAHEAD = 25 TIME_FORMAT = %Y.%m.%d - %H:%M:%S.%3N TIME_PREFIX = Timestamp: SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)(?=-{10})   ... View more

There used to be a really good .conf presentation on regex, but I guess Splunk decided to remove all of the old .conf content from their site....such a shame.  Glad you were able to make it work! ... View more

I am not sure about the udp input stanza and whether you can blacklist there or not.  But a common approach is to use props/transforms to filter matching events to the "nullQueue" so that they are not ingested.  That is something that can be done on the heavy forwarder. You can find details in the docs about it but also probably quite a few posts out here detailing the steps... Route and filter data - Splunk Documentation ... View more

Not sure if you need help coloring the tile or help with the search or both.  But here is an idea for the search at least.   index=msexchange sourcetype="MSExchange:2013:MessageTracking" subject="**** - Success" directionality=Originating psender="********" | earliest(_time) as _time stats dc(message_subject) as message_count | eval now = now(), cutoff = relative_time(now,"@d+11h+30m") | eval color = case(cutoff > now,"green",message_count=0, "red",_time < cutoff,"green",true(),"red")     So we add the earliest time for the matching even into the search.  Then we create a variable for now (when the search is running) and then use that to determine 11:30am today.  Then we can apply the logic.  The case() statement will run through each condition/value pair until it finds one that is true.   So if it is now before 11:30am, then green.  Otherwise, it's past 11:30am.  In that case, if we have 0 messages, then red.  Otherwise if we do have at least one message and it's timestamp is before 11:30am, then green.  And then the default condition will be red - at least one message but the timestamp is after 11:30. I did assume that it's running today for today but that can be more generlalized if needed I think. ... View more

I believe if you want to use join, then the field names need to be the same.  So since you rename sourceHost to uf_hostname, then joining on sourceHost won't work.  You would need to join on uf_hostname instead and then also rename the "join" field in the second search to also be called uf_hostname. That said, if I understand what you're trying to do (which i may not) and you have dns available, then you could try using a dns lookup instead....so just lookup the ip that is in the Source_Network_Address to get the hostname directly?  No need to join to internal logs then... index=* EventCode=4624 | lookup dnslookup clientip AS Source_Network_Address OUTPUT clienthost AS uf_hostname_or_whatever   Or something to that effect.  ... View more

been a while since i've admin'd a splunk instance, so the details escape me and I probably cannot help. But might be useful to know if the app is "invisible" or "inaccessible".  Meaning, is it just that they can't see it in Splunk or is it that even given the link to it, they cannot get to it? ... View more

This seems doable.  Let's assume you create a lookup table called senders with fields "ip" and "email", where you may have multiple rows for the same IP Address since you could have multiple emails for each ip. With that assumption, I think you could handle it two ways.  One, you could get all of the events and then filter for the ones you want.  Second, you could use a subsearch to help build a search that only got you the events you were after in the first place. Examples, not really tested: The first way.  Find all of the events, get the email addresses for the ip in the event, and include events where the Sender isn't in that list   index="myindex" | lookup senders ip AS OriginalClientIp OUTPUT email | where isnull(mvfind(email,Sender))     The second way, build the search condition from the lookup first, then run the search.  This should resolve to something like:   index=myindex (OriginalClientIp=x.x.x.x AND NOT (Sender="user1@isp.com" Sender="user2@isp.com)) OR (OriginalClientIp=x.x.x.y AND NOT (Sender="user3@isp.com" Sender="user4@isp.com)) ....   Not sure if all of the parens/quotes are right, but the basic idea is here   index=myindex [ | inputlookup senders | eval email = "Sender=\"" . email . "\"" | stats values(email) as emails by ip | eval emails = mvjoin(emails," OR ") | eval conditions = "(OriginalClientIp=" . ip . " AND NOT (" . emails . "))" | stats values(conditions) as conditions | eval conditions = mvjoin(conditions, " OR ") | return $conditions ]     ... View more

I'm not sure if I understand exactly what you're asking, but maybe you can use addtotals after your timehart?     ... | addtotals row=t col=f labelfield="Total"     That should calcualte the total sum of any number columns in each row and store in a field called "Total", which should be present on your chart then. ... View more

is that your entire splunk search?  or is that just a portion of a larger search that is trying to filter results?  i've been out of the splunk game for a bit, but i don't think the search command supports regex.  if this is filtering results in a larger search, then maybe using the where command with the match function is more appropriate. ... View more

if you're looking to do this in search, then something like this might work - wouldn't be surprised if there was a better way.  But essentially an array for each letter, then loop over the recordid and replace letters with associated index of that letter in the letters array | stats count | eval recordId="abc4efg" | eval mvRecordId = split(recordId,"") | eval letters=split("|,a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z",",") | eval serviceNowId = mvmap(mvRecordId,if(mvfind(letters,lower(mvRecordId))>0,mvfind(letters,lower(mvRecordId)),mvRecordId)) | eval serviceNowId = mvjoin(serviceNowId,"") ... View more