On the Splunk Usergroup Slack Chat (Need an invite? https://splk.it/slack ), Martin pointed out that a sum of nothing would be null not zero :), so quick adding an appropriate fillnull in here. ... View more

foreach is not needed in this case... you would use foreach when you need to do something to each distinct field... but in your case you have a bunch of values of the same field. I think if you know first that timechart will create a different field per value of the by clause field, and that untable can take a tabular format back out to a statistical format so that each of your events gives the bytes by ip for each 10 minute window (this is the opposite of the xyseries command), and that streamstats has a global parameter by which you can ensure the the window of 200 events (~30/40 minutes) is by each ip instead of by each set of events, your search is rather easy to translate to work across multiple ip addresses: index="proxies" earliest=-7d@d [inputlookup lu_inventory where function="web_server" | rename ip as c_ip | fields c_ip] |timechart limit=0 span=10m sum(bytes_out) by c_ip |fillnull |untable _time ip bytesout |streamstats window=200 current=true global=false median(bytesout) as median by ip |eval absDev=(abs(bytesout-median)) |streamstats window=200 current=true global=false median(absDev) as medianAbsDev by ip |eval lowerBound=(median-medianAbsDev*exact(20)), upperBound=(median+medianAbsDev*exact(20)) |eval isOutlier=if(bytesout < lowerBound OR bytesout > upperBound, 1, 0) |where isOutlier=1 The reason you'd do timechart ... | untable ... is that way you get those 10 minute spans where a particular IP has 0 bytes... another option would be to do bin _time span=10m | stats sum(bytes_out) as bytesout by c_ip but this would not include those 10 minute spans where an IP address has no bytes out, and thus throw off the running median... there's another command you would need to include to fill those time periods but offhand I'm not sure what it is. ... View more

Log analysis through Splunk does not seem like the correct approach to answering to the question of if accounts are currently disabled or not... instead you should be looking at the source of truth that maintains the accounts and seeing the state of the accounts. (Now if you were asking when accounts were disabled and by whom, then Splunk is your tool, assuming you're capturing and retaining those audit logs for long enough). With windows systems in a reasonably sized organization, i would guess your accounts are maintained in an Active Directory domain. To answer your question we'll need to query the state of those objects in AD, which doing so in bulk the best way is through LDAP queries.. if you absolutely need to keep results in Splunk, then SA-ldapsearch will help here, but honestly I would rather use the ldapsearch CLI utility from OpenLDAP, or if you prefer a GUI to perform LDAP searches, I'd recommend Apache Directory Studio. Now to build your LDAP Search. Often, when people say they have a userid for a domain account, they often mean that they have a list of SAMAccountNames. But it could be other naming attributes, or even email address. Whichever attribute it is, you'll then build an OR list of them in your LDAP filter to find all of the accounts based on these identifiers... Something like: (| (samaccountname=firstusername) (samaccountname=secondusername) ... (samaccountname=lastusername) ) Which while I'm writing on multiple lines for clarity I should note that could be collapsed into a single line just as easily thanks to the parenthesis being groupers and delineators. Next the question is which of these accounts are disabled, and which are enabled, and to answer this in Active Directory you'll need to be familiar with the userAccountControl attribute. This is a bit flag of statuses of the user accounts in AD... including if the account is disabled or not. Knowing that the 2 bit is the flag for the account being disabled or not...we could just download this attribute with our previous filter, and sort it out... But knowing that there are bitwise search operators for LDAP filters, we can actually build upon our previous filter with two different searches to answer that question for us from LDAP. To get the set of accounts that are disabled (where the disabled flag is set... we need an AND filter with our previous query... like so: (& (userAccountControl:1.2.840.113556.1.4.803:=2) (|(samaccountname=firstusername)...) ) And to get the set of accounts that are not disabled, we add a NOT filter around our check for the 2 bit on the user account control. (& (!(userAccountControl:1.2.840.113556.1.4.803:=2)) (|(samaccountname=firstusername)...) ) And if an account is returned by neither search, then it's not in Active Directory anymore. I hope this gives you an idea of how to do such, and I hope I've impressed upon you the need to use the right tool for the question you're trying to answer (and that Splunk may not be the answer all the time!). If you're not using Active Directory at your organization, then the steps will vary, but the general idea is similar: Look up the account status in the source of truth for the accounts, don't rely on the set of captured logs to tell you the current status. ... View more

So one major assumption that I'm making here, is how to pair up Start and End times. My answer assumes that the first Start time goes with the first End time for each job: |inputlookup f1.csv |stats list(*_Time) as *_Time by "JOB NAME" |mvexpand Start_Time |streamstats count by "JOB NAME" |eval End_Time=mvindex(End_Time,count-1) |fields - count So what we're doing here, is we're collecting all the times preserving order and duplicate values by using list() instead of values() Using mvexpand, We then expand out any duplicate Start_Times making a row for each. We then have to figure out which End_Time goes with each Start_Time, here we use streamstats to count the entries for each job, and then using eval we can pull the Nth End_Time out. (We wouldn't want to use mvexpand on both Start and End times because that would give us the cross product, which doesn't seem to be what you're after.) For a similar run anywhere example |makeresults | eval _raw="Job,Start,End Job1,S11, Job2,S2, Job3,S3, Job1,S12, Job4,S4, Job1, ,E11 Job2, ,E3 Job3, ,E3 Job1, ,E12 Job4, ,E4" | multikv | fields Job Start End | eval Start=if(trim(Start)=="",null(),trim(Start)),End=if(trim(End)=="",null(),trim(End)) | stats list(*) as * by Job | mvexpand Start | streamstats count by Job | eval End=mvindex(End,count-1) ... View more

Using the built-in webhook action, no. This is explicitly stated in the dev guide: http://dev.splunk.com/view/dev-guide/SP-CAAAE7A At this time, the JSON payload format can't be changed. That said, the custom alert action framework / Modular alert framework, means that anyone can create their own custom alert action, and do whatever you want with results, including invoke a webhook with a customized payload. Splunkbase has a number of custom alert actions, some of which invoke webhooks for different platforms and if none of these are suitable, you can always develop your own following the guide and resources in the docs: http://docs.splunk.com/Documentation/Splunk/7.0.2/AdvancedDev/ModAlertsIntro The Splunk Add-on Builder may also help in provide some utilities for building your custom alert action. ... View more

So tags, like event types are a type of knowledge object and can be created and managed in the UI through the settings menu or through tags.conf files. Some useful docs: https://docs.splunk.com/Documentation/Splunk/7.0.2/Knowledge/Defineandusetags https://docs.splunk.com/Documentation/Splunk/7.0.2/Knowledge/CurateSplunkknowledgewithManager https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Tagsconf ... View more

Another interesting statistic along these lines from that same Metrics log may be values(sourceIp) which I wonder if the assets table would help you or not... ... View more

As you're looking for a simple moving average with a window, instead of the trendline command, what about using the streamstats command ? You can set the window / time window appropriately, and just have average(close) by symbol ... View more

So if you are going HF- HF- Indexer, the place where these would have any effect is the first HF. In props.conf, the stanza names e.g. [CAMrouting] are the sourcetype, host or source that the attributes underneath apply to. So you have currently defined two sourcetypes each applying one transform. Spend some time with the spec file. You will also find that you can specify multiple TRANSFORM-x lines (so long as x varies), or specify multiple transforms in one attribute as that attribute is a comma separated list. ... View more

I think I can see a few potential issues here... First, it sounds like you're routing based on a field value, but your props.conf has stanzas for two different sourcetypes. If your inputs.conf is actually reading two distinct sources, then assign index at input time separately as well and you're done. (I would note that Xrouting are terrible sourcetype names, you should think about following Splunk's naming conventions for sourcetypes. Even if same sourcetype but two different inputs can be sent to different indexes on the forwarders. Much easier than applying a regex to each event) If you have a single stream of events, then your props stanzas are likely incorrect as they should be named for sourcetype. Barring HEADER_MODE, HEC (if you look at it the right way), and some esoteric setups that I'll get into in a minute, A single stream of data would not have differing sourcetypes at this point in the parsing process therefore the separate props conf stanzas is likely incorrect that way as well. Now your architecture... You describe a sequence of "Relay Forwarders." If these are Heavy Forwarders (I.e. full Splunk package as opposed to the Splunk universal forwarder package), then the first one to see your data is executing the regexReplacement processor, not your indexers and your appropriate props and transforms need to be there. (And the appropriate restarting of Splunk after landing config of course).This comes from Splunk trying to be efficient in ingestion processing allowing fully parsed data to skip parsing steps when forwarded to other Splunk instances. With some adjustments to inputs.conf on your indexers you can force re-baking of the data, but very rarely is that the correct answer and misconfiguration there can really screw things up. I'm not going to go deep into it but intermediate heavy forwarders are also usually not the right answer, and present a number of issues. Talking about the steps of ingestion, I usually keep this page bookmarked from the Splunk wiki: https://wiki.splunk.com/Community:HowIndexingWorks Hopefully this gives you some ideas that may or may not be applicable to your specific scenario. ... View more

The free Splunk Fundamentals 1 course doesn't get into field extraction... but rather assumes fields are already extracted for you. Fundamentals 2 (which is paid training) picks up where Fundamentals 1 leaves off and gets into the basics of field extractions among other topics. I have to admit I took the older courses and not the Fundmentals series (I don't remember if this was a normal Searching and Reporting class topic, or an Advanced S&R topic, or if rex landed squarely in the Admin course... but different aspects were covered in different parts of my Splunk class journey). ... View more

The simple answer is extract the data into a field. The documentation goes into this in great depth and length, about using the field extractor and many other options: http://docs.splunk.com/Documentation/Splunk/7.0.1/Knowledge/Aboutfields With a quick one-off though you may consider using the rex command to quickly extract the numeric portion and then use it like so: <base search> | rex "duration (?<duration_ms>\d+)ms" | timechart count min(duration_ms) max(duration_ms) avg(duration_ms) p95(duration_ms) I should note, I only had your one partial sample to work with, therefore milage may vary but that's the general idea... Build a regular expression to extract the data you want from your events, with appropriate anchoring. If you need help developing regular expressions, I recommend playing with regex101.com or any number of other interactive regex testers. If you are taking formal courses from Splunk it looks like this is started to be covered in Splunk Fundamentals 2 and later on with Splunk Data Administration. ... View more

Now I'm not too familiar with SplunkMINT but I suspect this is actually an issue with the Android app that you are using. As you mentioned that you haven't used this app in over a year I would first make sure that your app is up to date as the developer may have fixed the issue in the meantime. If the issue is still present, I would then contact the developer of said app to report the issue to them. I suspect this is a bug in how the developer is using the MINT SDK, as I found a GitHub Issue on another developer's repository. That developer seems to have fixed this issue by updating the version of the MINT SDK that they were using and ensuring that they were not attempting to set a null object somewhere. GitHub Issue: https://github.com/domoticz/domoticz-android/issues/262 Related Commit: https://github.com/domoticz/domoticz-android/commit/f3a5713a5d00cbbe492c269b63e3510632b42d1c ... View more

I can tell you it's definitely possible, as I do remember seeing a demo of data about a fleet of shuttle cars that Splunk had live dashboards on for a conference. In there they had a heatmap that generated around which routes the cars were taking. I think that they were using a custom visualization built in HTML on top of the web framework at the time, but other custom visualizations may be able to help you here. I will note I have not personally tried this particular visualization but it seems like it may be along the lines of what you may be looking for: https://splunkbase.splunk.com/app/3217/ I don't know the exact specifications of their query in the demo either... but I suspect you will want to look into SPL commands like geostats to get to the count of events in each geographic bin based on your data as well (it might be different depending on what the visualization you choose needs) ... View more

If this is the only date in your event, you should be configuring your sourcetype to properly parse this timestamp as _time . This comes with benefits of a lot of niceties in terms of becoming the thing Splunk searches by, built in commands like timechart and others. The docs go a bit into parsing time values: http://docs.splunk.com/Documentation/Splunk/7.0.1/Data/Configuretimestamprecognition With a value that is properly parsed (to epoch time), formatting it is just a function of the strftime eval function, or the convert search command. (In parsing and formatting you'll want to familiarize yourself with the Time format variables if you're not already familiar with them). If this is not the only date in your event, and there is a better one that fills the role of event time, so you need to extract it as a string... first extract the entire field between the brackets (as that timezone offset is important). Then you can use either strptime or convert to turn that string into an Epoch time value. Once that's done then you can again use strftime or convert to format the timestamp how you wish. (I mention the eval functions in addition to the commands, because the eval functions can be your friend in setting up Calculated Fields on your sourcetype). ... View more