Hello, I believe I know the problem based on some testing we did at my company.  I believe the root issue is when executing powershell with the powershell stanza it is invoking splunk-powershell.exe and that executable is not managing powershells memory at all.  In order to have the memory clear after your script completes you need to put this command at the end of your script: [System.gc]::Collect() This will force a garbage collection of powershell.exe's memory. If you do not do this then each iteration of your script will continually add to powershell's memory footprint since the memory is never cleared.  This article was very enlightening in regards to memory management with powershell: https://www.cloudsparkle.be/2021-05-06-PowerShellMemory/ Personally I believe this should be handled by the splunk-powershell.exe process and I believe it should be treated as a defect by splunk. ... View more

I recently had to deal with this too and found this answer, but it's out of date of the current webhook.py.  I figured having a more recent example that is python3 compatible would be beneficial.  Replace 127.0.0.1 with the proxy host you need to use.   import sys import json import csv import gzip from collections import OrderedDict from future.moves.urllib.request import urlopen, Request, ProxyHandler, build_opener, install_opener from future.moves.urllib.error import HTTPError, URLError def send_webhook_request(url, body, user_agent=None): if url is None: sys.stderr.write("ERROR No URL provided\n") return False sys.stderr.write("INFO Sending POST request to url=%s with size=%d bytes payload\n" % (url, len(body))) sys.stderr.write("DEBUG Body: %s\n" % body) try: proxy = ProxyHandler({'http': 'http://127.0.0.1:80', 'https': 'https://127.0.0.1:80'}) opener = build_opener(proxy) install_opener(opener) if sys.version_info >= (3, 0) and type(body) == str: body = body.encode() req = Request(url, body, {"Content-Type": "application/json", "User-Agent": user_agent}) res = urlopen(req) if 200 <= res.code < 300: sys.stderr.write("INFO Webhook receiver responded with HTTP status=%d\n" % res.code) return True else: sys.stderr.write("ERROR Webhook receiver responded with HTTP status=%d\n" % res.code) return False except HTTPError as e: sys.stderr.write("ERROR Error sending webhook request: %s\n" % e) except URLError as e: sys.stderr.write("ERROR Error sending webhook request: %s\n" % e) except ValueError as e: sys.stderr.write("ERROR Invalid URL: %s\n" % e) return False if __name__ == "__main__": if len(sys.argv) < 2 or sys.argv[1] != "--execute": sys.stderr.write("FATAL Unsupported execution mode (expected --execute flag)\n") sys.exit(1) try: settings = json.loads(sys.stdin.read()) url = settings['configuration'].get('url') body = OrderedDict( sid=settings.get('sid'), search_name=settings.get('search_name'), app=settings.get('app'), owner=settings.get('owner'), results_link=settings.get('results_link'), result=settings.get('result') ) user_agent = settings['configuration'].get('user_agent', 'Splunk') if not send_webhook_request(url, json.dumps(body), user_agent=user_agent): sys.exit(2) except Exception as e: sys.stderr.write("ERROR Unexpected error: %s\n" % e) sys.exit(3)   ... View more

Additionally if you want a command line python script that can poll your splunk environment's api on some kind of cron to pull each object individually into git, you can do so with this script I wrote: https://github.com/paychex/splunk-python/tree/main/Splunk2Git It is functionally similar to @gjanders app but doesn't involve installing an app if you have users who individually want to maintain their objects in whatever repository they want instead. ... View more

Thank you @guilmxm!  I did see the linked app in splunkbase which we are ultimately going to move to it, but there are a lot of processes already using that old app and its syntax that will require rework to make that transition.  I am glad that we have definitive answer though on if they plan on supporting the app or not so we can plan that work. ... View more

The joiner eval doesn't have double quotes around "outer" so it's trying to say when the value in field type equals the value in field outer, then use the value in field _id.  What you want to do is wrap outer with double quotes so the eval command knows you are looking for the field type to have a value of "outer." index=my_index (type=outer OR type=inner) | rename msg.* AS * | eval joiner=if(type="outer", _id, source_id) | stats values(*) as * by joiner | where ticket_num=TicketNum ... View more

In order to know the amount of down ports as compared to the total ports and assuming the show_interface sourcetype is a streaming input of the interface's status, you will need to obtain the latest status of each interface before you calculate a rate.  Your current query is passing in the values of all fields and then deduping, but values returns it in lexicographical order, not time order.  This will produce inaccurate results.  This is my suggestion to accomplish what you are looking for in a single query:   (index=71412-cli sourcetype=show_interface) OR (index=71412-np sourcetype=device_details) | eventstats values(deviceName) as deviceName by deviceId | stats latest(adminStatus) as adminStatus values(deviceName) as deviceName by deviceId interface | search deviceName ="BLV2-TI-SW_WAS18-01" | eval rate=if(adminStatus="down", 1, 0) | stats avg(rate) as "Down Rate" count(eval(adminStatus="down")) as "Down ports" count as "Total no of ports" by deviceName deviceId | eval "Down Rate"='Down Rate'*100   Line 2 is just appending in the deviceName from the device_details sourcetype to the show_interface sourcetype events so we can have that information when we use stats on line 3 to provide the latest adminStatus of each interface.  This could also be accomplished with a join after the stats line instead, but I tend to avoid joins if at all possible.  Once the latest adminStatus is obtained from line three you can apply your deviceName filter and create a "rate" field that is assigning a 1 or a 0 depending on if the latest adminStatus that returns is down or not.  Finally on line 6 you can average the "rate" field we crated on line 5 and also count the down and total ports on each device.  Line 7 is unnecessary but makes the percentage a bit easier to consume in my opinion. ... View more

Thank you for the details on the expected output of your data with examples.  Without seeing what the output that the query I proposed is doing it's difficult to know what is going on but I am going to assume that this data changes over time due to you using dedup and using values on the stats was causing multiple events to show up on each deviceId row.  This can be solved by simply using latest instead of values on those fields:   index=168347-np ([| `last_np_sourcetype("index=168347-np", "hw_eox")`] currentEoxMilestone=EoSCR OR currentEoxMilestone=LDoS (physicalType=*)) OR ([| `last_np_sourcetype( "index=168347-np", "group_members")` ] groupId=290681) | fields deviceId deviceName physicalElementId hwEoxId sourcetype | stats latest(deviceName) as deviceName latest(physicalElementId) as physicalElementId values(sourcetype) as sourcetype latest(hwEoxId) as hwEoxId latest(groupName) as groupName latest(groupId) as groupId by deviceId | search sourcetype=hw_eox sourcetype=group_members | stats count as Devices     If you ever need more fields you just add them to the stats line using latest so you always get the most current results for the deviceId.  Here's a run anywhere example demonstrating this using fake data:   | makeresults count=1 | eval data="cpyKey: 168347, currentEoxMilestone: LDoS, currentEoxMilestoneDate: 2018-01-31T00:00:00, deviceId: 18468220, deviceName: 10.24.13.240, hwEoxId: 286609, nextEoxMilestone: null, nextEoxMilestoneDate: null, physicalElementId: 362026445, physicalType: Power Supply, productId: C3K-PWR-265WAC, timeStamp: 2020-12-15T12:57:03 cpyKey: 168347, currentEoxMilestone: LDoS, currentEoxMilestoneDate: 2018-01-31T00:00:00, deviceId: 18468220, deviceName: 10.82.112.71, hwEoxId: 266079, nextEoxMilestone: null, nextEoxMilestoneDate: null, physicalElementId: 341757347, physicalType: Power Supply, productId: C3K-PWR-265WAC, timeStamp: 2020-12-15T13:57:03 deviceId:18468220, groupId:309425, groupName:Cisco 3900 ISR Routers, timeStamp:2020-12-15T12:56:41" | makemv data tokenizer="(?<data>[^\n]+)" | mvexpand data | rex mode=sed field=data "s/, /\n/g" | rex field=data "timeStamp: ?(?<timeStamp>[^\s]+)" | eval _time=strptime(timeStamp, "%FT%T"), sourcetype=if(match(data, "cpyKey"), "hw_eox", "group_members") | fields - timeStamp | rex field=data "deviceId: ?(?<deviceId>\d+)" | rex field=data "deviceName: (?<deviceName>[^\n]+)" | rex field=data "physicalElementId: (?<physicalElementId>[^\n]+)" | rex field=data "hwEoxId: (?<hwEoxId>[^\n]+)" | rex field=data "groupId:(?<groupId>[^\n]+)" | rex field=data "groupName:(?<groupName>[^\n]+)" | rex field=data "cpyKey: (?<cpyKey>[^\n]+)" | rex field=data "currentEoxMilestone: (?<currentEoxMilestone>[^\n]+)" | rex field=data "currentEoxMilestoneDate: (?<currentEoxMilestoneDate>[^\n]+)" | rex field=data "nextEoxMilestone: (?<nextEoxMilestone>[^\n]+)" | rex field=data "nextEoxMilestoneDate: (?<nextEoxMilestoneDate>[^\n]+)" | rex field=data "physicalType: (?<physicalType>[^\n]+)" | rex field=data "productId: (?<productId>[^\n]+)" | stats latest(deviceName) as deviceName latest(physicalElementId) as physicalElementId values(sourcetype) as sourcetype latest(hwEoxId) as hwEoxId latest(groupName) as groupName latest(groupId) as groupId latest(cpyKey) as cpyKey latest(currentEoxMilestone) as currentEoxMilestone latest(currentEoxMilestoneDate) as currentEoxMilestoneDate latest(nextEoxMilestone) as nextEoxMilestone latest(nextEoxMilestoneDate) as nextEoxMilestoneDate latest(physicalType) as physicalType latest(productId) as productId by deviceId | search sourcetype=hw_eox sourcetype=group_members | fields - sourcetype deviceId     You'll see in the above run anywhere example only the most current values are showing for each field except the sourcetypes because we want to ensure both sourcetypes are being represented in this data. ... View more

Hello, Without knowing your underlying data it will be hard for me to know what is wrong.  The execution of the query is what I would recommend to remove a join.  I did notice something odd on the final two lines of your non-join query.  You are counting the rows by hwEoxId then summing that count.  It's possible you could be double counting rows when doing it by hwEoxId if there are multiple hwEoxId  per deviceId.  If that is the case you can correct that by doing this query instead: index=168347-np ([| `last_np_sourcetype("index=168347-np", "hw_eox")`] currentEoxMilestone=EoSCR OR currentEoxMilestone=LDoS (physicalType=*)) OR ([| `last_np_sourcetype( "index=168347-np", "group_members")` ] groupId=290681) | fields deviceId deviceName physicalElementId hwEoxId sourcetype | stats values(deviceName) as deviceName values(physicalElementId) as physicalElementId values(sourcetype) as sourcetype values(hwEoxId) as hwEoxId by deviceId | search sourcetype=hw_eox sourcetype=group_members hwEoxId=* | stats count as Devices ... View more

I'm glad it's working.  Please mark the solution as accepted to help future individuals.  Thank you! ... View more

In order to properly answer this question we need to know how the "hardware" and "group_members" are being used in the last_np_sourcetype macro.  Is there a specific field where those values are being searched?  Is it just in the raw event somewhere?  If it is in the raw event then the below query would work but it's not as efficient as it would be if we knew the exact field that these values are expected in: index=168347-np ([| `last_np_sourcetype("index=168347-np","hardware")`] (physicalType=*)) OR ([| `last_np_sourcetype( "index=168347-np", "group_members")` ] groupId=290681) | fields physicalElementId deviceId _raw | stats values(_raw) as raw values(physicalElementId) as physicalElementId by deviceId | search raw=*hardware* raw=*group_members* | stats dc(physicalElementId) as Devices ... View more

If you just want the latest _raw value in a time period you can just use latest(_raw) in your stats.  Here's an example:   | rename Issues{}.details AS details Issues{}.file AS file Issues{}.severity AS severity Issues{}.confidence AS confidence Issues{}.line AS line | eval tempField=mvzip(mvzip(mvzip(mvzip(details, file), severity), confidence), line) | stats count latest(_raw) as example by _time, service, source, tempField | eval details=mvindex(split(tempField,","),0), file=mvindex(split(tempField,","),1), severity=mvindex(split(tempField,","),2), confidence=mvindex(split(tempField,","),3), line=mvindex(split(tempField, ","),4) | stats max(_time) AS latest, count AS Issues latest(example) as example by _time, severity | sort - _time ... View more

When you pop your query out of the dashboard and you are looking at the table instead of the visualization what is the name of the column that has the data you want in the token?  Also please post the full drilldown part of your xml on the panel just in case. ... View more

click.name and click.value are not the appropriate tokens to use when you are using a cluster map, which is what I am assuming you are using since you are using the geostats command to generate the data for the visualization.  You can simply use $row.name$ as your drill down token since behind the visualization is just a big table and each cluster represents a row.  Now this gets complicated when you have multiple names in a single tile, but I believe it should still be functional depending on your data. ... View more

I like it! 🙂 ... View more

If you modify the iis log configuration so it has the missing IdentityClaim field you will need to adjust your field extractions I wrote a bit to this in search: | rex "\s((?<IdentityClaim>[ic]):)?0(?<ClaimType>[\!\"\#\$\%\&\'\(\)\*\+\-\.\0\1\2\3\4\5\6\7\8\9\<\=\>\?\@\[\]\^\_\`\a\b\c\d\e\f\g\\\])(?<ClaimValueType>[\.\+\)\"#\$\&\!0])(?<AuthMode>[wstmrfc])\|((?<OriginalIssuer>[^\|]+)\|)?(?<ClaimValue>[^\s]+)" and to this if you make it an extracted field in props.conf for the ms:iis:auto sourcetype: \s((?<IdentityClaim>[ic]):)?0(?<ClaimType>[\!\"\#\$\%\&\'\(\)\*\+\-\.\0\1\2\3\4\5\6\7\8\9\<\=\>\?\@\[\]\^\_\`\a\b\c\d\e\f\g\\])(?<ClaimValueType>[\.\+\)\"#\$\&\!0])(?<AuthMode>[wstmrfc])\|((?<OriginalIssuer>[^\|]+)\|)?(?<ClaimValue>[^\s]+) ... View more

Once you feel comfortable with all of that you can take it a step further by turning that rex to pull out each of those fields in the final search into an extracted field for the ms:iis:auto sourcetype.  You need to adjust it slightly for that though:   \s0(?<ClaimType>[\!\"\#\$\%\&\'\(\)\*\+\-\.\0\1\2\3\4\5\6\7\8\9\<\=\>\?\@\[\]\^\_\`\a\b\c\d\e\f\g\\])(?<ClaimValueType>[\.\+\)\"#\$\&\!0])(?<AuthMode>[wstmrfc])\|((?<OriginalIssuer>[^\|]+)\|)?(?<ClaimValue>[^\s]+)   Once that is done you can then create lookup definitions for those lookup files you created and use automatic lookups so the fields that are in those lookups already show in searches and can be referenced by other users without the need to do the lookup command manually in your search. ... View more