I'd suggest using a stats to accomplish this instead of an append: index="test" sourcetype=test_events "loginSuccess" OR ("loginFailed" 404) | eval AuthenticationStatus=if(match(_raw, "loginSuccess"), "Success", "Failed") | eval FirstDayOfTheWeek=relative_time(_time, "@w+1d") | stats values(AuthenticationStatus) as AuthenticationStatus by FirstDayOfTheWeek currentUsername | rename currentUsername as User | fieldformat FirstDayOfTheWeek=strftime(FirstDayOfTheWeek, "%d-%m-%Y") | sort 0 + FirstDayOfTheWeek User The OR statement on the first line will have you search both of the conditions you want and then the values command on the stats transformation on line 4 will report the different Authentication statuses as a multivalued field. Give that a try and let me know if it works for you. edited it to account for the currentUsername field instead of User ... View more

Can you define what you mean by "unknown fields"? You set the fields in the data model when you create it. Here's a Splunk query that can pull everything from the json that it saves the data model under. It will show all of the fields that are present in the data model under the objects array: | rest splunk_server="local" "/servicesNS/-/-/data/models" | search title="datamodel_title" | table eai:data | spath input="eai:data" Is that what you are looking for? ... View more

Can you define what you mean by "unknown fields"? You set the fields in the data model when you create it. Here's a Splunk query that can pull everything from the json that it saves the data model under. It will show all of the fields that are present in the data model under the objects array: | rest splunk_server="local" "/servicesNS/-/-/data/models" | search title="datamodel_title" | table eai:data | spath input="eai:data" Is that what you are looking for? ... View more

You can put this in your search or set the equivalient time on the alert under the advanced portion of the time range selector: earliest=-1d@d+21h latest=now ... View more

Here's an updated run anywhere example that will use spath in the evals to try to limit the amount of memory you need when running the query. It will be difficult to deal with the multiple events in a single event until you get your conf line breaks fixed on these events. This query is assuming the ID on the message is what is used to unite the two messages. If that is not the case let me know: | makeresults count=1 | fields - _time | eval data="<Message> <ID>00000000-0000-0000-0000-000000000000</ID> <Date>2019-04-27 19:33:27.06</Date> <Text>Launched application: MMViewer, PID: 7988</Text> <Category></Category> <Source>Workflow</Source> <Level>Event</Level> <Class>General</Class> <Module>WorkflowHost</Module> <Method></Method> <FileName></FileName> <LineNumber>0</LineNumber> <ProcessID>6836</ProcessID> <User>e1e2f1a39246e0cc7e611c8d180e8f6c</User> <Parameters></Parameters> </Message> <Message> <ID>00000000-0000-0000-0000-000000000000</ID> <Date>2019-04-27 19:34:42.04</Date> <Text>Application: MMViewer, PID: 7988 failed</Text> <Category></Category> <Source>Workflow</Source> <Level>Warning</Level> <Class>General</Class> <Module>WorkflowHost</Module> <Method></Method> <FileName></FileName> <LineNumber>0</LineNumber> <ProcessID>6836</ProcessID> <User>e1e2f1a39246e0cc7e611c8d180e8f6c</User> <Parameters></Parameters> </Message>" | rex mode=sed field=data "s/ +</</g" | rex mode=sed field=data "s/\n +\n/\n\n/g" | rex mode=sed field=data "s/\n\n/█/g" | makemv data delim="█" | mvexpand data | eval "Message.Date"=strptime(spath(data, "Message.Date"), "%Y-%m-%d %H:%M:%S.%2N") | eval Launched=if(match(spath(data, "Message.Text"), "Launched"), 'Message.Date', null()) | eval LaunchName=if(match(spath(data,"Message.Text"), "Launched"), spath(data,"Message.Text"), null()) | eval Failed=if(match(spath(data,"Message.Text"), "failed"), 'Message.Date', null()) | eval "Message.ID"=spath(data, "Message.ID") | stats values(LaunchName) as "Launched Application" values(Launched) as Launched values(Failed) as Failed by "Message.ID" | fieldformat Launched=strftime(Launched, "%Y-%m-%d %H:%M:%S.%2N") | fieldformat Failed=strftime(Failed, "%Y-%m-%d %H:%M:%S.%2N") | eval duration=Failed-Launched ... View more

Can you run this search and post the results for just those three domains you have listed and post the results? | inputlookup domain.csv | eval hostname=domain | table hostname | format It should return results that look like this: ( ( hostname="yyy.us" ) OR ( hostname="yyy.at" ) OR ( hostname="yyy.ae" ) ) I got this to work with a run anywhere example using the data you provided. Here's the query that shows your main query and the results: | makeresults count=1 | eval _raw="10.154.248.238 - - [03/May/2019:06:49:35.367 -0500] GET /xxx/rest/services/public/abc?ERROR_CODE=0x00000000&ERROR_TEXT=BA0521I%20%20%20Successful%20completion&HOSTNAME=www.yyy.us&AUTHNLEVEL= HTTP/1.1 307 - https://www.yyy.us/logout Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) userid=- sec=0 usec=13508 fwd=,- route=route0" | rex field=_raw "\[(?<timestamp>[^\]]+)" | eval _time=strptime(timestamp, "%d/%b/%Y:%H:%M:%S.%3N%z") | fields - timestamp | eval hostname=mvindex(split(mvindex(split(_raw,"&HOSTNAME="),1),"&AUTHNLEVEL="),0) | eval hostname=trim(replace(hostname,"www.","")) Your results should look like this: _raw 10.154.248.238 - - [03/May/2019:06:49:35.367 -0500] GET /xxx/rest/services/public/abc?ERROR_CODE=0x00000000&ERROR_TEXT=BA0521I%20%20%20Successful%20completion&HOSTNAME=www.yyy.us&AUTHNLEVEL= HTTP/1.1 307 - https://www.yyy.us/logout Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) userid=- sec=0 usec=13508 fwd=,- route=route0 _time 2019-05-03 07:49:35.367 hostname yyy.us This run anywhere query mimics your lookup file and produces the results I would expect with the format command: | makeresults count=1 | fields - _time | eval domain="yyy.us~yyy.at~yyy.ae" | makemv domain delim="~" | mvexpand domain | eval hostname=domain | table hostname | format You should see this under the search header: ( ( hostname="yyy.us" ) OR ( hostname="yyy.at" ) OR ( hostname="yyy.ae" ) ) Combining them into a single search that will return any hostname that exists in that lookup: | makeresults count=1 | eval _raw="10.154.248.238 - - [03/May/2019:06:49:35.367 -0500] GET /xxx/rest/services/public/abc?ERROR_CODE=0x00000000&ERROR_TEXT=BA0521I%20%20%20Successful%20completion&HOSTNAME=www.yyy.us&AUTHNLEVEL= HTTP/1.1 307 - https://www.yyy.us/logout Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) userid=- sec=0 usec=13508 fwd=,- route=route0" | rex field=_raw "\[(?<timestamp>[^\]]+)" | eval _time=strptime(timestamp, "%d/%b/%Y:%H:%M:%S.%3N%z") | fields - timestamp | eval hostname=mvindex(split(mvindex(split(_raw,"&HOSTNAME="),1),"&AUTHNLEVEL="),0) | eval hostname=trim(replace(hostname,"www.","")) | search [| makeresults count=1 | fields - _time | eval domain="yyy.us~yyy.at~yyy.ae" | makemv domain delim="~" | mvexpand domain | eval hostname=domain | table hostname | format] You can change the data after HOSTNAME=www to confirm that the filter is functioning with any three of the domains in the lookup as well as see it return no results if you make it something not in that lookup. ... View more

gotcha, a longer term solution would be to have your conf files adjusted so that your line breaks are on each <message> xml flag and that your event time is taken from the date portion of the message. That will make it easier to perform your requested statistical analysis to derive the duration. That being said, the answer I provided in the answer section can still work, but it will need to be tweaked to be more efficient as using sdata on every event plus a mvexpand will blow up the memory utilized in your query quite a bit. I'll comment on my solution below a more efficient query to extract the required fields via a spath eval. ... View more

If could provide an example of the events that you are looking at (with any private information removed) form your main query and some examples of the events in your lookup file I can give you a more details query using a run anywhere example. ... View more

If I'm reading his original post correctly, the domain lookup file equates to the hostname eval he is extracting in his search, not the host that the access log originated from. ... View more

Glad I could help! ... View more

I have appeared to identify a defect with the markup code. If you post a code snippet after a ordered list, it breaks the code snippet functionality. ... View more

It appears to have stripped the xml markup I copy/pasted from your original example. I'll work to get that fixed. ... View more

Here's a run anywhere example using the data you provided that accomplishes you goal. This is making a couple assumptions that I would need you to confirm: | makeresults count=1 | fields - _time | eval data="<Message> <ID>00000000-0000-0000-0000-000000000000</ID> <Date>2019-04-27 19:33:27.06</Date> <Text>Launched application: MMViewer, PID: 7988</Text> <Category></Category> <Source>Workflow</Source> <Level>Event</Level> <Class>General</Class> <Module>WorkflowHost</Module> <Method></Method> <FileName></FileName> <LineNumber>0</LineNumber> <ProcessID>6836</ProcessID> <User>e1e2f1a39246e0cc7e611c8d180e8f6c</User> <Parameters></Parameters> </Message> <Message> <ID>00000000-0000-0000-0000-000000000000</ID> <Date>2019-04-27 19:34:42.04</Date> <Text>Application: MMViewer, PID: 7988 failed</Text> <Category></Category> <Source>Workflow</Source> <Level>Warning</Level> <Class>General</Class> <Module>WorkflowHost</Module> <Method></Method> <FileName></FileName> <LineNumber>0</LineNumber> <ProcessID>6836</ProcessID> <User>e1e2f1a39246e0cc7e611c8d180e8f6c</User> <Parameters></Parameters> </Message>" | rex mode=sed field=data "s/\n\n/~/g" | makemv data delim="~" | mvexpand data | spath input=data | eval "Message.Date"=strptime('Message.Date', "%Y-%m-%d %H:%M:%S.%2N") | eval Launched=if(match('Message.Text', "Launched"), 'Message.Date', null()) | eval LaunchName=if(match('Message.Text', "Launched"), 'Message.Text', null()) | eval Failed=if(match('Message.Text', "failed"), 'Message.Date', null()) | stats values(LaunchName) as "Launched Application" values(Launched) as Launched values(Failed) as Failed by "Message.ID" | fieldformat Launched=strftime(Launched, "%Y-%m-%d %H:%M:%S.%2N") | fieldformat Failed=strftime(Failed, "%Y-%m-%d %H:%M:%S.%2N") | eval duration=Failed-Launched The ID in the message always matches between the failed/launched events The fields are automatically extracted by Splunk. I used the spath command to manually have it extracted, but that is not recommended to run on all your events as it can consume a lot of memory on your search head at execution time. If that's not the case I can show you how to run spath in an eval so it only pulls that specific field. ... View more

Is each message xml a seperate event in Splunk or are they in the same event? The answer to this will determine how best to answer your question. ... View more

Sorry, I misread the initial request and the first result was providing _raw values but you wanted specific fields. I have just adjusted it for that. ... View more

You can manually perform the top command after a stats that pulls the latest _raw event using the below query: index=omi_Uat host=DEFRNCMP* sourcetype=all_events_attributes | eval {idx} = elt | fields ID,UMN,TicketID,node | stats count values(ID) as ID values(TicketID) as TicketID values(node) as node by UMN | eventstats sum(count) as TotalCount | eval percent=round((count/TotalCount)*100,6) | sort 0 - count | streamstats count as counter | where counter<=10 | fields - TotalCount counter If you require an other field that will make it a bit tricker, but it is possible: index=omi_Uat host=DEFRNCMP* sourcetype=all_events_attributes | eval {idx} = elt | fields ID,UMN,TicketID,node | stats count values(ID) as ID values(TicketID) as TicketID values(node) as node by UMN | eventstats sum(count) as TotalCount | eval percent=round((count/TotalCount)*100,6) | sort 0 - count | streamstats count as counter | eval UMN=if(counter>10, UMN, "Other") | fields - TotalCount counter | stats sum(count) as count sum(percent) as percent values(ID) as ID values(TicketID) as TicketID values(node) as node by UMN | sort 0 - count ... View more

Can you provide some examples of the different queries that you need to join? Instead of doing things with join or lookups, you could use stats command on the field you were using to join. Lookups are fast, but require maintenance and can take up large amounts of disk space on your search head if you have an extremely large data set you are working with. Here's an extremely high level concept of using stats to combine data from different sources: (search string 1) OR (string string 2) | eval joiner=if(criteria unique to search string 1, field from search string 1 that you join on, field from search string 2 that you join on) | stats values("Field from Search string 1") as "Field from Search string 1" values("Field from Search string 2") as "Field from Search string 2" by joiner ... View more

Done. Thank you for reminding me to do this. ... View more

This won't fix the root cause of the problem that @somesoni2 mentioned, which is that you the sourcetype is not setup with an offset as explained in the documentation, but it can make the _time appear accurately as a search time work around. This can be used to show results from the last hour Search with UCT earliest=now latest=+4h | eval _time=strptime(strftime(_time, "%Y-%m-%dT%H:%M:%S.%3N")."UCT", "%Y-%m-%dT%H:%M:%S.%3N%Z") | where _time>relative_time(now(), "-1h") AND _time<now() ... View more

It depends on what you are monitoring, if it's just the operating system being up/down I have an example I can give you that I use to monitor our Linux servers OS status: index=linux ("Startup finished" OR "shutting down for system halt" OR "shutting down for system reboot" OR "systemd-shutdownd: Shutting down" OR "Starting * about System Reboot/Shutdown" OR ("kernel time sync status")) | eval Status=case(match(_raw,"Startup finished") OR match(_raw, "kernel time sync status"),"Up",NOT match(_raw,"Startup finished") AND NOT match(_raw, "kernel time sync status"), "DOWN!") | stats max(_time) as _time latest(Status) as Status by host | table _time host Status If you are monitoring if an application is running or not that would be a different query. The basic principles is that you need to identify the specific logs for what you want to monitor that tell that it's down and then a log that tells you when it's back up. If you have that, you then can use the above query, but pointed at the events that you need. Without you providing me scrubbed of anything sensitive examples of those events, I cannot really help much more than at a conceptual level. ... View more

Hi @jip31 , I recognize this query. 🙂 The join is case sensitive so if the data of the HOSTNAME has different case than the host in the other query it won't match. This should fix that if that's the case: index="X" sourcetype="Y" source="Z" EventCode=6008 | dedup host | eval host=lower(host) | table _time host EventCode | join type=left host [ search index=master-data-lookups sourcetype=":view_splunk_assets" | stats count by HOSTNAME SITE ROOM TOWN CLIENT_USER COUNTRY OS | fields - count | rename HOSTNAME as host | eval host=lower(host)] | table _time host COUNTRY TOWN SITE ROOM CLIENT_USER OS | sort -_time limit=10 If that works, I'll convert this to an answer. ... View more

Is the goal to have a dashboard that shows the last status reported by a server? If so, you'll need to use a lookup table as your stateful table and craft a scheduled query that updates that table with the most recently reported status of the server. If you give some more details as to the type of queries you are using to identify server state I can at least give you some ideas on how to accomplish that. ... View more

Is this logging format some kind of application standard or is this something that someone made in house. I ask because the way it's currently formatted makes it difficult for index time field extractions. I have a search time extraction that can be used to accomplish what I believe you are trying to do (create key value pairs) using a run anywhere example with your data that was provided on your question: | makeresults count=3 | streamstats count as counter | eval _raw=case(counter=1, " isDraggingObject : True id : afbbdeb7-9fd4-4b53-ab17-742809154ba9 condition : {or, matches System.Object[] (?i)(^.*?host failure alert.*?www\.jennycraig\.com\.au.*?$), matches System.Object[] (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com\.au.*?$)} catch_all : False advanced_condition : {} actions : {route PVG22KK, severity warning}", counter=2, " isDraggingObject : True id : 3b5aa785-b854-4e43-900a-225da5786a27 condition : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com\.au.*?$)} catch_all : False advanced_condition : {} actions : {severity critical, route PVG22KK}", counter=3, " **isDraggingObject : True id : a8420998-fbca-486b-9ff7-d03b9e16536e condition : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com$), matches System.Object[] (?i)(^.*?\bcritical\b.*?locations\.jennycraig\.com)} catch_all : False advanced_condition : {} actions : {severity critical, route PW0VV83}**") | fields - counter _time | rex field=_raw mode=sed "s/(\n|^)\s+(\w+\s+):/█\1\2:/g" | makemv _raw delim="█" | rex mode=sed field=_raw "s/█//g" | rex mode=sed field=_raw "s/\n/ /g" | extract kvdelim=":" pairdelim=" " The first sed statement is placing a unique character to be inserted into the event, which we can then use to make it a multi-valued results that splits each key value pairing into a unique value of the raw data. It then removes the unique character and then removes all line breaks and makes them spaces. After that is done you can run the extract command and it will produce proper key value pairing. If this is a custom generated log event, I would suggest that they consider making it key value format by wrapping their values with quotes and changing the colon to an equal sign or follow the Windows events formatting standard if you want line breaks without quotes in your values. ... View more