Is it possible to use the internal logs instead?? That way you can check for a phonehome event within your polling interval. If the event exists then the device is on, if not the device is off. index=_* host=yourHostFilter phonehome | timechart span=30m count by host | foreach * [eval <<FIELD>>IsOn = if(<<FIELD>> > 0, 1, 0) | eval <<FIELD>>IsOff = if(<<FIELD>> = 0, 1, 0) ] | fields + *IsOn + *IsOff As long as you have a span that is >= your phone home interval this should work. Thanks. ... View more

That works but there are 100's of "instance" values which makes it hard to view the data in a table and impossible to view in a chart. Ideally I would get the top 10 counter + instance combinations. ... View more

One thing that I found out the hard way. In my sample I was using the makeresults command to simulate data and test the changes to limits.conf. Rolled the change to production and it is failing...because the rex command is running against the event data on the Indexer, not on the Search Head. :angry: I'd recommend adding the limits.conf change to both sides to keep them in synch because depending on the query the rex might run on the SH or the Indexer. Thanks. ... View more

YES!!!! That's a winner, thank you so much. Well done sir, post that as answer so I can accept. ... View more

Looks like I was using a non-compatible stanza, the version of my sandbox server is 6.6.4 so uses an older stanza: [rex] recursion_limit = 999999 match_limit = 999999 Was able to get a bit more processing but now getting generic failure: Job terminated unexpectedly The search.log doesn't have an error in it, couple unrelated warnings about lookup tables. Checked the _internal and _audit logs for the SID and the error text, nothing there either. Looks like I'm butting up against a framework limit on the system, which is disappointing when regex101 shows that it should work just fine. :sadpanda: ... View more

Found the edge of the limit by slimming down to just the field with a bunch of escaped quotes in it. This will fail: | makeresults | eval quotedJSON = "{ \"property25\": \"<?xml version=\\\"1.0\\\" encoding=\\\"utf-16\\\"?><Error ServerName=\\\"xxxxxx\\\" DateTime=\\\"6/28/2018 1:49:33 PM\\\"><Message Description=\\\"Object reference not set to an instance of an object.\\\" Number=\\\"\\\" /><Methods><Method Dll=\\\"PPUI.Widgets, Version=0.18.1.1652, Culture=neutral, PublicKeyToken=__SECURITYMASK__\\\" Class=\\\"classname.xPSLayoutWidget\\\" Name=\\\"ProcessInstruction_DisplayFieldLabel\\\" LineNumber=\\\"42\\\"><Details><Detail Name=\\\"DisplayFieldLabel\\\" Description=\\\"Object reference not set to an instance of an object.\\\" /><Detail Name=\\\"Inner XML\\\" Description=\\\"\\\" /><Detail Name=\\\"fieldName\\\" Description=\\\"\\\" /><Detail Name=\\\"labelType\\\" Description=\\\"\\\" /><Detail Name=\\\"suppressColon\\\" Description=\\\"\\\" /><Detail Name=\\\"lengthAttr\\\" Description=\\\"\\\" /><Detail Name=\\\"mainLabel\\\" Description=\\\"Is Unknown\\\" /><Detail Name=\\\"subLabel\\\" Description=\\\"Is Unknown\\\" /></Details></Method><Method Dll=\\\"PPUI.Widgets, Version=0.18.1.1652, Culture=neutral, PublicKeyToken=__SECURITYMASK__\\\" Class=\\\"classname.Widgets.xPSLayoutWidget\\\" \" }" | rex field=quotedJSON "(?x-i) (?(DEFINE) (?<ws>[\r\n\t\x20]*) (?<str>\"(?:\\[rntbf\\\/] | \\\\\" | [[:xdigit:]]{4} | [^\\\"[:cntrl:]])*\") (?<bool>true|false) (?<nil>nil) (?<num>-?\d+(?:\.\d+)?) (?<elem>(?:(?&str)|(?&bool)|(?&nil)|(?&num))(?&ws)) (?<comma>,(?&ws)) ) (?<extractedJSON> \[ (?&ws) (?: (?: (?&elem) | (?R)(?&ws) ) (?(?=(?&comma)(?:(?&elem)|[\[\{]))(?&comma)) )* \] | \{ (?&ws) (?: (?&str) (?&ws) : (?&ws) (?: (?&elem) | (?R)(?&ws) ) (?(?=(?&comma)[\"\[\{])(?&comma)) )* \} )" | eval rawSameAsExtracted = if(_raw=extractedJSON, "true", "false") | table quotedJSON, extractedJSON, rawSameAsExtracted But if you pull off the last quoted element it will work: Class=\\\"classname.Widgets.xPSLayoutWidget\\\" Honestly I don't think this should fail. When putting the same object in a regex 101 test it completes in just 4ms, I'm thinking the limits.conf settings aren't working... ... View more

Got access to a sandbox server and added this stanza in the limits.conf: [rex] depth_limit = 9999999 match_limit = 9999999 Couldn't find anything in the documentation that specified the real limits. Tried zero (0) and that didn't work at all. Also tried 100000 and 999999. Error is slightly different on the sandbox server than what is running in cloud: has exceeded the configured recursion_limit, consider raising the value in limits.conf I also tried adding a "recursion_limit" line in the limits.conf, that gave a generic error: search failed unexpectedly The search.log file didn't have anything helpful in it...coming up empty. ... View more

Okay, the data is too big to paste in this system but I did store it as a regex101 example: https://regex101.com/r/No8Xnc/2 The real issue is "property25" which in the real world is an application stack trace formatted as xml. This data has a ton of double quotes and other other escape characters and is really long. If I remove that field the regex works just fine. If you pull the example into a query you'll have to find/replace single quotes with \" and then replace \" with \\". This sample without property25 works just fine: | makeresults | eval quotedJSON = "{ \"InstrumentationLogDateTime\": \"2018-06-28T13:49:33.7895781-04:00\", \"property1\": \"00000000-1111-2222-3333-444444444444\", \"property2\": \"00000000-1111-2222-3333-444444444444\", \"property3\": \"my application\", \"property4\": \"00000000-1111-2222-3333-444444444444\", \"property5\": \"V0300\", \"property6\": \"999918\", \"property7\": \"system\", \"property8\": \"Info\", \"property9\": \"123456\", \"property10\": \"xx\", \"property11\": \"xx\", \"property12\": \"page name\", \"property13\": \"class name\", \"property14\": \"method name\", \"property15\": \"123456\", \"property16\": \"123456\", \"property17\": 42, \"property18\": \"xx\", \"property19\": \"xx\", \"property20\": \"123456\", \"property21\": \"trans name\", \"property22\": \"42\", \"property23\": \"2018-06-28T13:49:28\", \"property24\": \"Object reference not set to an instance of an object.\", \"CustomFields\": { \"property26\": \"\", \"property27\": \"xx\", \"property28\": \"anotherbiglongvalue\", \"property29\": \"xx\", \"property31\": \"\", \"property32\": \"123456\", \"property33\": \"anotherbiglongvalue.anotherbiglongvalue.anotherbiglongvalue\", \"property34\": \"anotherbiglongvalue,anotherbiglongvalue,anotherbiglongvalue\", \"property35\": \"anotherbiglongvalueanotherbiglongvalueanotherbiglongvalue\", \"property36\": \"this is a big ol thing\", \"property37\": \"this is a big ol thing\", \"property38\": \"this is a big ol thing\", \"property39\": \"2018-06-28T13:49:33.0000000-04:00\", \"property40\": \"2018-06-28T13:32:24.0000000-04:00\" } }" | rex field=quotedJSON "(?x-i) (?(DEFINE) (?<ws>[\r\n\t\x20]*) (?<str>\"(?:\\[rntbf\\\/] | \\\\\" | [[:xdigit:]]{4} | [^\\\"[:cntrl:]])*\") (?<bool>true|false) (?<nil>nil) (?<num>-?\d+(?:\.\d+)?) (?<elem>(?:(?&str)|(?&bool)|(?&nil)|(?&num))(?&ws)) (?<comma>,(?&ws)) ) (?<extractedJSON> \[ (?&ws) (?: (?: (?&elem) | (?R)(?&ws) ) (?(?=(?&comma)(?:(?&elem)|[\[\{]))(?&comma)) )* \] | \{ (?&ws) (?: (?&str) (?&ws) : (?&ws) (?: (?&elem) | (?R)(?&ws) ) (?(?=(?&comma)[\"\[\{])(?&comma)) )* \} )" | eval rawSameAsExtracted = if(_raw=extractedJSON, "true", "false") | table quotedJSON, extractedJSON, rawSameAsExtracted ... View more

Okay, got an updated rex that is working well. Had an issue with matching on \" but that has been solved. | makeresults | eval quotedJSON = "{\"foo\":\"bar\", \"works\":\"say, maybe \r \b \\" \"}" | rex field=quotedJSON "(?x-i) (?(DEFINE) (? [\r\n\t\x20]) (? \"(?:\[rntbf\\/] | \\\" | [[:xdigit:]]{4} | [^\\"[:cntrl:]]) \") (? true|false) (? nil) (? -?\d+(?:.\d+)?) (? (?:(?&str)|(?&bool)|(?&nil)|(?&num))(?&ws)) (? ,(?&ws)) ) (? [ (?&ws) (?: (?: (?&elem) | (?R)(?&ws) ) (?(?=(?&comma)(?:(?&elem)|[[{]))(?&comma)) )* ] | { (?&ws) (?: (?&str) (?&ws) : (?&ws) (?: (?&elem) | (?R)(?&ws) ) (?(?=(?&comma)[\"[{])(?&comma)) )* } )" | eval rawSameAsExtracted = if(quotedJSON=extractedJSON, "true", "false") | table quotedJSON, extractedJSON, rawSameAsExtracted Still puking on the PCRE limits, trying to see if I have access to a sandbox to fiddle with those settings in limits.conf ... View more

The rex is in the original question and it's a beast. It uses a lot of recursion which is necessary for hierarchical object structures so I'm not sure there is anything I can do there. Likely means I'll have to up the PCRE limit but would love a set of skilled eyes on it. Thanks. ... View more

True, but unfortunately I don't have this: consistent field that should be there I did figure out why the extraction didn't work, was missing a top level capture group, added this right after the DEFINE group: (?<extractedJSON> Now I am getting the correct JSON and can do a simple comparison at the end: | eval rawSameAsExtracted = if(_raw=extractedJSON, "true", "false") Working wonderfully...except it isn't. Now I'm getting an error for very large JSON objects: exceeded the PCRE recursion limit Found another answer with a similar issue: https://answers.splunk.com/answers/581183/is-my-rex-right-rex-has-exceeded-configured-match.html But that one was using lookbacks, this monstrosity...I actually don't even understand it. 😉 Ideas?? ... View more

Found the top tools container, it toggles a class called "affix-top" when scrolling vertically: Normal: <div class="statistics-controls-inner"> Scrolled: <div class="statistics-controls-inner affix-top"> Also found the column header, looks like a copy of the header is toggled when scrolling because the visibility goes from hidden to block: Normal: <div class="header-table-docked" style="right: 0px; left: 0px; display: none; top: -1000px;"> Scrolled: <div class="header-table-docked" style="right: 0px; left: 0px; display: block; top: 36px;"> But no idea how to get these added to my dashboard. 😞 ... View more

After some further research this approach isn't going to work. Our summary jobs complete in just a few seconds with file sizes well under 1 MB, so unless we were to change the autoLBFrequency property to something freakishly small like 2s we will still have everything go to a single indexer. Furthermore, even if we were to change the value it would send to data to maybe two indexers...we have 18. Two is 100% better than One, but still not good. ... View more

From what I gather in your response this would impact scheduled Summary Jobs as well as any one time back-fill, as they both use stash files?? Thanks for the tip, I'll look into using that property and come back to accept if it works. Given my sample set you are likely right that the generated file is processed well below the default 30s threshold. ... View more

You could use a combination of a Summary Index then supplement the data with what hasn't been summarized yet. If you are running for "All Time" then you are likely doing some type of | stats avg(field1), count(field2) etc. You could keep this query, replace the stats with sistats and save the search as a summary index job that runs daily. Then when you are in your dashboard you can query the summary index for any data that is greater than one day old and append that to today's data. Something like this: index=summary source="YouSummaryData" earliest=-1w@d latest=-1d | bucket _time span=1d | stats count as TransactionCount by _time | append [ search index=YourOperationalData earliest=@d latest=now | bucket _time span=1d | stats count as TransactionCount by _time ] The trick here is to control the earliest and latest fields to rely mostly on the Summary data and only pull one day of your Operational data. ... View more

I don't want the transcount by the other fields, I want it to be the total transcount for the whole server and only break the "multipleValues" field out by instance. ... View more

Hmmm, I think this is getting closer. Tried the mvexpand and it fixed the issue of the visualization, but it still has the values across all the "instance" groups in one field. Ideally it would be grouped by the instance. Tried adding the instance to the "by" and it is grouping all the fields by instance now, but I really only want the single field grouped by the instance. In a perfect world it would be something like: | chart count(eval(like(sourcetype, "iis"))) as transCount , values(value1) as valueToCheck by instance , values(networkUtilizationValue) as networkUtilization by _time I did figure out that I can fix the issue by adding eval statements for each instance...but that is tedious and boring. Thanks. ... View more