I can't think of any way to inherently accomplish that. Typically that kind of logic would get applied at search time. But if that's not an option for you, then maybe just blacklist that event id (see blog post below). And then you could probably script something to accomplish your goal for that event id. Event Logs in Splunk 6 ... View more

So you're still getting that message? I'm not sure, honestly. Typically that message means that the index doesn't exist on your indexer. Maybe there is something wrong with index configuration though? ... View more

I would recommend against that. It should be easy enough to create a new app and copy the files over there. That said... If not, then i would suggest you don't manage those files from the deployer. Instead just copy the files directly down to the search app on the members (may have to do a rolling restart before they're recognized, not sure). And then manually create the dashboards in Splunk Web on one of the cluster members (just copy/paste dashboard source to a new dashboard in the search app). The dashboards themselves will get replicated to the other members. Technically, you probably could copy the search app from a member to the deployer. Place your files and dashboards there. Then push the entire search app back down to the members. But that just seems like a really bad idea and would likely cause you quite a headache later on. ... View more

On this forum, don't forget to put your splunk search code inside the code blocks - if you just paste it in the comment box, a bunch of stuff gets parsed out for some reason. But back to the question. Instead of rex'ing twice, can we generalize our regex to grab service no matter what those letters are. For example, if the pattern is always "[info] xxxx(service)", then something like this might work. \[info\][^\(]+\((?<service>[^\)]+).+Latency:\s+(\d+\s+){7}(?<cool_num>\d+) ... View more

Well, what you have there isn't a valid a command. Are you trying to filter your results at that point? If so, then you probably want to use where. | where idaFullName=$username$ If you're not trying to filter, what are you trying to do? ... View more

Sorry, I don't think I get alerted/emails on these comments so I tend to miss them (I'll check my settings). Anyway, that looks good to me. Did you restart splunk on the indexer since installing the SCCM app? ... View more

So you just want to create two pie charts - one for prod indexes and one for non-prod? And the slices would be actual indexes based on how much was indexed? If so maybe something like these: Non-prod index=_internal source=*license_usage.log* type=Usage idx=np_* | stats sum(b) as bytes by idx | eval GB = round(bytes/1024/1024/1024,5) | fields idx GB Prod index=_internal source=*license_usage.log* type=Usage idx!=np_* | stats sum(b) as bytes by idx | eval GB = round(bytes/1024/1024/1024,5) | fields idx GB ... View more

I don't think Splunk tells you when data is truncated or whether any of the normal "health" apps like SoS, DMC, Fire Brigade, etc have panels with that info. We use a simple search below in one of our own dashboards to help identify which sourcetypes are being truncated, so we can address appropriately in props.conf. The splunkd logs know how long the event was and the limit that it exceeded which caused it to be truncated. Maybe you could find it helpful.... index=_internal sourcetype=splunkd truncating | rex "limit of (?<limit>\S+).+\>=\s+(?<length>\S+)" | stats max(length) as length max(limit) as limit count by data_source data_sourcetype | sort - count ... View more

something like this? ... | stats count(eval(type1_if=1)) as type1_if count(eval(type2_if=1)) as type2_if count(eval(type1_if=1 OR type2_if=1)) as type_if by _timestamp user ... View more

Just a quick comment on the monitor format. This is from the inputs.conf spec file on splunk docs: [monitor://<path>] * This directs Splunk to watch all files in <path>. * <path> can be an entire directory or just a single file. * You must specify the input type and then the path, so put three slashes in your path if you are starting at the root (to include the slash that goes before the root directory). So the first 2 slashes are actually part of monitor definition. And then typically 3rd slash (on nix) is the start of the path to the file/directory to monitor. On windows, instead of having a 3rd slash, you'd start your path there, like this [monitor://C:\program files\blah\blah.txt] Not sure if that's where you've seen the difference between 2 and 3 slashes, but hopefully it helps a little ... View more

I believe those props settings are processed during the parsing phase. So if your forwarder is actually a heavy forwarder, then that's where you'd make the changes. If it's a universal forwarder, then you'll want to move those changes to your indexer. Where do I configure my Splunk Settings? ... View more

I think I understand what you're asking. The "service" referenced in the search isn't a literal string. We're actually using the rex command to extract whatever is in the parentheses after "wsgw" and storing that value in a new field named "service". Then in the timechart command, we're telling splunk to group the results by the various values in that "service" field. Also a quick note on the max aggregate. Usually when you plot against time in Splunk, you will have multiple values for a chunk of time. So you have to use some sort of aggregate function to tell Splunk how to combine all of those values - min, max, avg, sum, etc. And you can specify multiple aggregates if you want too. For example, if you wanted to see both the min and max values. ... View more

thanks, great info! ... View more

one thing to note, if you're trying to monitor the path /var/log/.... then you should have three slashes after monitor: in your stanza. For example: [monitor:///var/log/remote/*dlpmgrv/*.log] I'm wondering if that's somehow causing those symptoms of seemingly indexing the data but in the wrong index/sourcetype. Also, I think the nix app uses an os index. If that's installed on that box, maybe there are some conflicts there? ... View more

I'm not sure what kind of chart you're looking at or how you want it to appear. Are you trying to plot the numbers by the service over time? If so, something like this might work. ... | rex "wsgw\((?<service>[^\)]+).+Latency:\s+(\d+\s+){7}(?<cool_num>\d+)" | timechart max(cool_num) by service ... View more

just a thought,....you don't possibly have a copy of an older indexes.conf in a local directory somewhere on your search heads do you? splunk btool indexes list _blocksignatrue --debug also may be worth checking to see if it's specifically listed in an authorize.conf somewhere too? Not sure if that has an effect on search. ... View more

Were you planning to push those changes to the search app in the shc or to another app? or multiple different apps maybe? In general, I would say don't use the search app. With that said, whatever other app you choose to push those changes to in the shc, I would have the same exact app on the dev box. And then once the changes look good, copy it over to the deployer and push out to the cluster. ... View more

Are you running that search outside of the palo alto app? If so, do you get the same results if you run it inside the palo alto app? I'm wondering if maybe the lookups aren't shared globally? Of course maybe make sure the lookups are actually there too, Settings -> Lookups -> Lookup definitions. Then choose the "Splunk for Palo Alto" app in the dropdown. ... View more

it does, thanks! ... View more

It sounds like you need the index to be on host2. when you say you see it in the gui, do you mean you see it on host2 in the gui? If you run the following from the cli on host2 (your indexer), what do you get? splunk btool indexes list sccm --debug | grep -v system/default ... View more

It might be more efficient to use stats. I think table/dedup/sort can get expensive. Maybe just stats values(source_ip) would work? index=pan_logs earliest=-30d host="10.10.10.10" Destination_IP=1.2.3.4 Rule_Name=VPN sourcetype=pan_traffic | stats values(Source_IP) as Source_IPs We had to resort to running searches on the cli recently because of how long some searches were running. Even some of those timed out, so we ended up scripting it to run separate searches in different chunks of time. Which might be another options for you? Meaning, run two 15 day searches or 3 10 day searches? And export the results? ... View more