Not sure i understand the scenario. When you say it's there if you just search the data, you mean like you search index=whatever and the results have the pool field extracted? And when you say USE it, you mean like index=whatever | table pool .... or something like that? And there's nothing there? which version of splunk? ... View more

some thoughts: is this a shc? if it is, then keep in mind that the deployer will merge default/local appropriately and push the "winning" changes into default on the members. So if this is a member in a shc, being in default doesn't necessarily mean it is in default in the app. If that is the case, check to see if there is a local version of checklist.conf on your deployer if you download 3.1.0 and just look at the files (w/o installing), does that key exist in the default checklist.conf? that might help you verify whether it was something from your migration. If it is, is it also defined appropriately in the checklist.conf spec file in the README folder. And then if needed, do the same check for 3.1.1 (if it was maybe caught and addressed) ... View more

Not sure about getting your config to work (we struggle with o365 connection/input with that add-on). But my understanding is that the plan for now at least (ms always seems to be changing it), is to send everything to Azure Monitor. Maybe that's not the case for your data? But they do have a TA specifically for Azure Monitor now. We struggled getting that add-on to work as well, but the developer was very responsive and helped us through installing/configuring it. Azure Monitor Add-on For Splunk And here is their session at .conf this year: Monitor and Manage Your Cloud Environment with Azure Monitor and Splunk ... View more

On Server B, you need to install the SCOM console. Just the console. Not a management server, just the console. The same thing you would install on your desktop so you could manage your SCOM env. I believe the link that jplumsdaine22 provided shows you how to do that. Does that make sense? ... View more

you need a heavy forwarder because the add-on does more than a uf can do. But it doesn't need to be installed on a SCOM server. I believe you just need the SCOM console installed on the heavy forwarder, so the add-on can access the powershell cmdlets. ... View more

hmm, what is it set to global as well? honestly, i'm not sure exactly which permissions will it available but assume if it's global and read/write for everyone, then everyone should see it from anywhere. side note, you can respond with comments instead of answers. answers are typically set aside as for when somebody thinks they have an answer to the question. ... View more

If they go to Settings -> Alert Actions, is the alert action listed there? If so, are the permissions set correctly? ... View more

dumb question, does it matter if you throw the --answer-yes at the end? I mean "no target specified" kinda makes it sound like it's not even parsing the command correctly. just a thought.... ... View more

You should be able to create another tcp group in outputs.conf and then add that group to your _TCP_ROUTING setting in your inputs. It sounds like you already have a tcpout group created for your indexers. So just create another for the other splunk environment. The _TCP_ROUTING setting in inputs.conf accepts a comma-delimited list of those groups, so just add that new group to the list. At that point, the eventlog data should get routed to both groups/environments. If the other environment needs it in a different index, then they should be able to update their props.conf to set the right index before indexing on their side. outputs.conf [tcpout:<target_group>] server = [<ip>|<servername>]:<port>, [<ip>|<servername>]:<port>, ... * Required if indexerDiscovery is not set. * Takes a comma separated list of one or more systems to send data to over a tcp socket. * Typically used to specify receiving splunk systems, although it can be used to send data to non-splunk systems (see sendCookedData setting). * For each mentioned system, the following are required: * IP or servername where one or system is listening. * Port on which syslog server is listening. inputs.conf _TCP_ROUTING = <tcpout_group_name>,<tcpout_group_name>,<tcpout_group_name>, ... * Comma-separated list of tcpout group names. * Using this, you can selectively forward the data to specific indexer(s). * Specify the tcpout group the forwarder should use when forwarding the data. The tcpout group names are defined in outputs.conf with [tcpout:<tcpout_group_name>]. * Defaults to groups specified in "defaultGroup" in [tcpout] stanza in outputs.conf. * To forward data to all tcpout group names that have been defined in outputs.conf, set to '*' (asterisk). * To forward data from the "_internal" index, _TCP_ROUTING must explicitly be set to either "*" or a specific splunktcp target group. ... View more

I'm not sure if the search results are available to just be accessed. But in the _audit index, there should be events where action=search and info=granted. The sid from the alert event should match the search_id from that one. And in that search event, you should have the search itself as well as earliest/latest. So you might be able to build a drilldown to rerun the search over the timeframe of the triggered alert...maybe? ... View more

I think you're thinking like a programmer and not a splunk'er...but not sure if I understand the scenario exactly. Can you elaborate on what 'events' you're referring to? Can you provide example events and also what the lookup contains? That said, something like "... | stats dc(keyword)" might get you what you want, but will need that context about your data to help further. ... View more

I think it's a regex thing, not a splunk thing. This is when regex starts to hurt my head but the following seems to work...just took a little playing around on regex101. node=(?<node>[^\s]+) a0=(?<cmd>[^\s+]+)|\sa[1-9]=(?<args>\S+) Ultimately, if you want regex to match multiples, each one would need to match what leads up to it. So I think this is telling regex that the node/a1 stuff can be there (a1) or even just space (a2-xxx). That's my understanding, but could also be making it up. ... View more

It's for the following 3 hours. The 12am interval includes, 12am, 1am and 2am. The 3am interval includes 3am, 4am, and 5am...and so on. If span worked, i think that's the way it would work. if you want it to be the other way, a few more commands can probably handle that. ... View more

If you're referring to pushing parsing changes to the indexer cluster, then you can review this documentation to understand what will cause a restart vs a reload Restart or Reload Also, in 6.6 they finally added the ability to check whether the cluster will restart or not. So you should at least be able to determine whether one will be needed - if not, then you might be able to make the changes during business hours. We're not on 6.6 so I haven't had a chance to verify it works as expected. Validate the Bundle To validate the bundle and check whether a restart is necessary, include the --check-restart parameter: splunk validate cluster-bundle --check-restart This version of the command first validates the bundle. If validation succeeds, it then checks whether a peer restart is necessary. If you're on an earlier version of Splunk, be sure to choose your version in the documentation. Changes have become less impactful in recent versions, but older versions will probably require a restart more often. ... View more

I get the impression that you may not know how join works. Typically, you use join to glue to result sets together by one or more fields. It looks like your search is trying to join a result set of just one field called Version to a result set of just two fields called count and Systems. So there's no field for splunk to use to join those together? Anyway, you may be able to get away with just stats and not use join. Note: it's really hard to come up with a working search without actually seeing your data, but maybe something like this. earliest=-24h latest=now index=abc (source=xyz OR (source=pqr AND (Systems="I" OR Systems="X"))) | stats count(Systems) as system_count, values(RUNNING_VER) as product_version by ID | stats sum(system_count) by product_version So get data from both sources initially (assuming the Systems filter is needed for pqr from your search?) . Then use stats to count events from the pqr data and get version from xyz source by the common ID field. And since you may have duplicate versions across IDs (???), just sum the counts from the pqr source by version from xyz. Hope that helps somewhat. ... View more

Another option might be to use eventstats to get the total number of events, streamstats to get a running count, and then just keep the first and last ... | eventstats count as total | streamstats count | where count=1 OR count=total ... View more

where is the data coming from? where is the props? and where are you running oneshot? Is it all happening on the search head? Is your search head also your indexer? Just want to be sure you're putting your props in the right place in your infrastructure - wherever parsing happens (the first instance running a full version of splunk enterprise) Side note, I think using LINE_BREAKER and SHOULD_LINEMERGE=false is typically more efficient than trying to merge lines then break them. ... View more

Is that example scenario based on a real one? You should be able to change ownership via REST I think....which will propagate the changes. Manually modifying a conf file will not. I haven't run into many scenarios in my clusters where i created something that Splunk didn't let me edit, so was curious how common this for you. ... View more

pretty sure your role needs to be able to search the indexes - otherwise, the searches behind the panels won't find any data. I don't think you can have a user just see the results of a search w/o giving them access to the data gathered by the search. ... View more

differentiate how? do you just need to identify for a given alert, where it came from? Ore are there different security teams or SOCs that need to work/document the incdients? Or different thresholds/rules between the domains? If it's just a matter of the data will be in different indexes and that's it, then you should be able to get away with one instance of ES. If there are more complex requirements around roles, accessibility, permissions, etc, then you might want to elaborate on those here. ... View more