@genesiusj  Then you may have some opportunities to cleanup, but just combining my logic with yours current one, I have this: | rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/glass_table report_as=text fields="_key,identifying_name,description,acl,definition" ```1``` | eval value=spath(value,"{}") | mvexpand value ```14 glass tables``` | eval glass_table_name = spath(value, "identifying_name") | search glass_table_name = "gex ecosystem - jp tes" ```1 glass table``` | eval description = spath(value, "description") ```this is blank ???``` | eval key = spath(value, "_key") ```is this the glass table ky: 2913043e-76e1-11f0-abf2-0050568f9bbf``` | eval acl = spath(value, "acl") ```what is acl? an access control list listing what permissions are used on the glass table?``` | rex field=acl "\"owner\"\:\s\"(?<owner>\w+)\"" ```glass table owner/creator``` | eval definition = spath(value, "definition") ```what is definition?``` | rex field="definition" max_match=0 "(?s)\"(?<ds_key>ds_[^\"]+)\"\\s*:\\s*\\{.*?\"name\"\\s*:\\s*\"(?<ds_name>[^\"]+)\"" | eval pair=mvzip(ds_key, ds_name, "||") | mvexpand pair | eval key=mvindex(split(pair,"||"),0), name=mvindex(split(pair,"||"),1) | table key name | sort key But you can append more values to it such as the GT name or even filtering it by name so your result will always be the one that you are filtering. Also, to your questions there: "Description" can be blank if malformed or not informed. If that is important to you, you can review your specific GTs and work on the extraction properly. I don't think it is as you never actually use it in your results in this example. "acl" is an object containing access control list fields to map ownership, permissions and storage app of the object. It is a common object for Splunk knowledge objects.  "definition" is the actual source code behind the GT, so there is where all the panels, queries, dropdowns, styles, etc are stored.   ... View more

I endorse what the folks shared above. MCD alone does a lot of noise across the board due to its REST searches that comes packaged to make it works on its nature. In my organization, what I did was to get a second MCD box in another datacenter for HA, but splunk service is always stopped there.  In a very high level description, I have an rsync script that keeps both on sync on a given interval, along with a custom automation in place (using SOAR in this case) that checks whenever the main MCD stops responding REST, causing the playbook to check host availability, and in case of failures it then starts the service on the second box. At the load balancer, the health check also monitors REST so if it is not responding it marks the box as offline, so after starting the service in the second the HC will mark it as up and bring it back to live. Everything ends with an email communication so we know that the flip was done and the proper status of each step. Kinda alternative to keep MCD structure alive, but this is just in case you actually need it otherwise it may not worth the stress. ... View more

Hey @genesiusj , Yeah, those ids are not as good as names for tracking when it comes to reading by source, which is mostly my case. What I do is whenever building dashboards I always change the DS generated ID and its references downstream to named instead, like ds_panela or ds_panelb, so I know by looking where they are coming from. But in your case, as you are already with hundreds created such action is way too much to think of, so let's try this: | rest /servicesNS/-/-/data/ui/views/ splunk_server=local | search title="<your dashboard title>" | fields title eai:acl.app eai:data | rex field="eai:data" max_match=0 "(?s)\"(?<ds_key>ds_[^\"]+)\"\\s*:\\s*\\{.*?\"name\"\\s*:\\s*\"(?<ds_name>[^\"]+)\"" | eval pair=mvzip(ds_key, ds_name, "||") | mvexpand pair | eval key=mvindex(split(pair,"||"),0), name=mvindex(split(pair,"||"),1) | table key name | sort key   This should give you what you need, the key (id) and the "name" of each data source:   ... View more

@learningmode I'm sorry, I completely overlooked that, you are right, you are using $host_list as a parameter to get the field content, not a token value... Try this instead: index=_internal earliest=-15m group=queue name=* [ | makeresults | eval host_list="$tok_MultiSearch1$" | makemv delim="," host_list | eval host_list=mvmap(host_list, "\"" . trim(host_list) . "\"") | eval search = "host IN (" . mvjoin(host_list, ",") . ")" | return $search ] | fields name | dedup name sortby name | stats values(name) as Queues   What I'm suggesting here: Run the makeresults that gets the token value to run as subsearch, which takes priority and runs first. The "return" clause will return a field to the parent search context, that can then be accessed and evaluated properly. Give it a try and see if that works. Also, you can customize the "search" field's return format by editing line 7 to adhere to your expectations. ... View more

Hi @learningmode, Let me try to help you here... As per your description the solution should be really straightforward so I'm sure it is just something small left behing. Starting with your last search there, the token is like that exactly as you pasted? If so, you missed a "$" after "$host_filter": ... | fields - quoted_hosts | search index=_internal earliest=-15m group=queue host IN ($host_filter) | fields name ...   Did you noticed that? ... View more

Oh, I love when simple workarounds do the trick. Well, glad you got that one sorted. If I can help with anything else, even for a brainstorming session, lemme know 🙂 ... View more

Hi @Kyles, What version of DBConnect are you using? The 3.* versions allow you to pass the field named params with values to be passed to the variables in the query/procedure: dbxquery connection=<string> [fetchsize=<int>] [maxrows=<int>] [timeout=<int>] [shortnames=<bool>] query=<string> OR procedure=<string> [params=<string1,string2>] Sample query: dbxquery procedure="{call sakila.test_procedure(?, ?)}" connection="mysql" params="150,BOB"  Please try that out if you are running DBConnect 3.x and let me know how it goes ... View more

Okay, The OAUTH integration message asks for credentials and says that that account will be able to interact as if it was you, right? And based on what you said, this is what happening (assuming that whenever that ServiceNow screen shows up you are adding your own credentials to allow OAUTH, instead of the credentials of the local API account you should have on ServiceNow for this purpose as you also mentioned that the account does NOT have interactive UI access). IMO and based in your statement, I believe in that authorization part you entered your own credentials and now during your tests this is the reason why ServiceNow is showing Splunk interactions as if it was you. So, though I'm just repeating myself, I guess the path is: If using basic auth: Create a new ServiceNow account config in the Splunk TA as you shown before, define it as basic On your alerts, whenever configuring the send event / incident action, pass the name of the account you created here You're done! If using OAUTH: Create a new ServiceNow account config in the Splunk TA as you shown before, define it as OAUTH While being redirected to ServiceNow OAUTH authorization page, insert the API account user name and password (not your own) - At this step, if your API account isn't accepted, then you cannot use it as it lacks those interactive permissions I mentioned According to the message, whatever communication between Splunk and ServiceNow with that OAUTH channel will use THAT logged account during auth phase  On your alerts, whenever configuring the send event / incident action, pass the name of the account you created here You're done!   ... View more

Hi @elcuchi , Ok, I use basic auth instead of OAUTH so different scenario, OAUTH was not available on our first tested TA versions and we never moved away from it (which I should prioritize now). Did you test basic or that is not an option? Thing is, for basic auth: Whenever you configure the ServiceNow account in the TA, you'll have to pass that account as parameter for the ServiceNow action commands OR reference it in the alert action (it is the first field it asks you to fulfill). That is the account the TA will use to open the REST connection with ServiceNow and push the data there (either event or incident). AFAIK, there is no configuration on the TA that uses the actual Splunk logged in user in the authentication context to ServiceNow to trigger those actions. Behind the scenes, every communication is done via the account configured in the TA, at least this is how it works for me while using this TA for the past 4-5 years.   So, question: How are you testing this? (Based in your "when we test the creation of an incident from splunk interface" statement) For OAUTH it may be different, but according to the documentation I don't think it actually is. Documentation says that Oauth requires UI access to SNOW instance, which you mentioned you don't have: OAuth Authentication configuration requires UI access to your ServiceNow Instance. User roles that do not have UI access will not be able to configure their ServiceNow account to use OAuth.   If this is using the person logged in to access ServiceNow instead of using whatever OAUTH config, it makes no sense for the TA to ask clientID and clienteSecret as the main purpose for those is to authenticate. ... View more

Hi @roopeshetty ,   The proxy config should be in its own stanza, not the [general] one.     [proxyConfig] http_proxy = <string that identifies the server proxy. When set, splunkd sends all HTTP requests through this proxy server. The default value is unset.> https_proxy = <string that identifies the server proxy. When set, splunkd sends all HTTPS requests through the proxy server defined here. If not set, splunkd uses the proxy defined in http_proxy. The default value is unset.> no_proxy = <string that identifies the no proxy rules. When set, splunkd uses the [no_proxy] rules to decide whether the proxy server needs to be bypassed for matching hosts and IP Addresses. Requests going to localhost/loopback address are not proxied. Default is "localhost, 127.0.0.1, ::1">   Once you make the changes and restart, run a btool to make sure the server is getting it correctly from your configset: /<splunk_home>/bin/splunk btool server list --debug | grep proxy All the configurations returned are the ones being used by the system, confirm if all your custom configs are here and if there are not overlays taking precedence over them.   ... View more

Hi @meshorer,   You can stop that by clicking in your name (top-right) and going to Account Settings. There, you click on Notifications tab and turn off the notifications you won't need, such as Event Reassigned in this example. Once you save it, you won't receive those emails anymore.   ... View more

Hi @saraomd93 , This is pretty generic and can be happening for many different reasons, so trying some: - Maybe there is a PG instance that failed to halt and is still alive. Run a ps -ef | grep postgres and see if you get any process running. If so, kill the process - Maybe there is a problem on the password set during the upgrade process. Review that against your current configuration and try again - tail the <SOAR_DIR>/var/log/pgbouncer/pgbouncer.log for some hints about what is going wrong. - tail the <SOAR_DIR>/data/db/pg_log/<todays_file>.log for some hints about what is going wrong. - Check if you have enough space on disk on the partition where SOAR is installed (may look a bit dummy but I got surprised a few years back when my disk got full during the upgrade caused by DB backup that was done there). ... View more

Hi @roopeshetty , Can you elaborate more about what did you try already when you mentioned " We tried many options using proxy settings but none of them are working."?   Also, it is not sure if you are running in a standalone environment or a clustered one, and if the proxy configs you tried were in conf files or added via REST. Check this documentation about some good example on how to configure proxy and non-proxy addresses, and make sure that you define the http/https_proxy correctly (use the same config mentioned in your browser for reference if that is using a direct proxy address instead of a auto-discovery script.) Configure splunkd to use your HTTP Proxy Server - Splunk Documentation Notice that you must pass the authentication in the URL if your proxy requires it. ( like http://user:pass@myproxy.com:80) ... View more

Okay, in this case, please share the search that renders the datamodel. Perhaps you can do the replace there to make sure there are no double quotes returned on Climate.air_temp field. ... View more

Hi @lyngstad , Based in your search results screenshot, you get the values from the by clause but lack the calculation of air temp, so I guess the trick is to format the datamodel to be measured (so yes, it may be related to some double quotes around the values). Can you try this then? Trying to get it via "values" so we can later convert to number and remove the quotes, which will allow average metric. | tstats values(Climate.air_temp) as air_temp_raw from datamodel="Climate" where sourcetype="TU_CLM_Time" host=TU_CLM_1 by host _time span=60m | eval air_temp_numeric = tonumber(trim(air_temp_raw, "\"")) | stats avg(air_temp_numeric) as air_temp by host _time    ... View more

Hi @santoshpatil01 , Your request is lacking useful information for anyone to help. It is not clear the query format that will serve as your base search so maybe you'll have to adjust that to your reality. Basically you'll need to have a base query that returns all raw data for the tokens wherever they are, and then you create the panels accordingly. If you don't have the input fields to set the tokens, you'll need to set them as well on each panel OR in the dashboard header depending on the filter active necessity. In each panel, mention the base search making this a linked search, and use as query something like this: Total request number for security token/priority token filtered by partner name | search partner=$token.partner$ | stats count as "Total Requests" by security_token, priority_token Duplicate request number filtered by partner name and customer ID (to check if current expiration time for both tokens are appropriate) | search partner=$token.parner$ AND customerId=$token.customerId$ | stats count by parner, customerId | where count>1 Priority token usage filtered by partner name | search partner=$token.parner$ | stats count by token_name Response time analysis for security token/priority token | stats avg(response_time) as response_time by security_token, priority_token Or if you need 90th percentile instead: | stats p90(response_time) as response_time by security_token, priority_token Again, this is just a scratch in the surface as I don't know your query, field names and additional information, but it should be enough for you to kick this off and play around. ... View more

Have you tried to consolidate then via stats command and then configure your alert to trigger for each result and tokenize the email parameter? Try this (adjust to your reality): <your search> | stats values(event_field) as events by user, email Then in your alert configuration, set trigger conditions: Number of results > 0 Trigger: For each result And add the email action with To with token $result.email$ That will make each email receive their group of events Give it a try and let me know  ... View more

Hi @hazem., Is this [DDMMYYYY] just a placeholder for an actual date in this example or this is the literal string being monitored in the monitor stanza and also the literal text in the filename? I ask that because if what you wanna do is to monitor C:\Program Files (x86)\dir1\log\name_CRT_<any date>.log then you can use * at that part like: C:\Program Files (x86)\dir1\log\name_CRT_*.log This way the monitor stanza will know what to do. Anyways, always make sure that in order for the forwarder to proper monitor something, that file must have the right read permissions to be read. Usually some applications under Program Files may be locked to administrators and that may cause SplunkForwarder service not to have the right permission to read the particular log. A good indication for that is to check the _internal index for logs related to that and see if they are logging Access Denied somewhere. The below search may give you some heads up on hits; Restart splunk forwarder and keep eyes on that log for last 5 min range or something as forwarder will evaluate the monitors at the startup and you'll find it easier. index=_internal host=<my_forwarder_host> "C:\Program Files (x86)\dir1\log\" ... View more

Hey @dswoff , AFAIK there is a problem in your logic. The | iplocation command accepts a few arguments, but not like key:value pair as the IP. I believe in your case you want to pass the IP and get the Country as result, then try this: index="eventlog" EventCode=1309 | rex field=Message "User host address:\s(?<ip_address>.*)" | iplocation ip_address | table ip_address, Country OR for fixed IP index="eventlog" EventCode=1309 | iplocation "<your_ip_here>" | table ip_address, Country The iplocation accepts an IP and will give you as response the fields: City, Continent, Country, MetroCode, Region, Timezone, lat and lon. Give it a try and let me know ... View more

Hi @abi2023 , It is not so clear to me if you want to apply color to the cells in the "user" column only if in a specific format and shape, so if you can clarify it would be nice. You can apply that to the table section in your XML code by adding the Format tag with type color. Under that, you specify the type of coloring logic that can be dynamic, range, scale, etc... For example, for a rule that will color the cells of User column whenever there is a value with a light green cell color: <format type="color" field="user"> <colorPalette type="expression">if (isnotnull(value), "#00ff3c", "#f24949") </colorPalette> </format> Green will be valid values, red will be empty/null value cells. Is something like that you're looking for? Full sample: <dashboard version="1.1" theme="dark"> <label>My Dashboard</label> <row> <panel> <table> <search> <query>MySearchString</query> <earliest>-15m</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">100</option> <option name="dataOverlayMode">none</option> <option name="drilldown">none</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <format type="color" field="user"> <colorPalette type="expression">if (isnotnull(value), "#00ff3c", "#f24949") </colorPalette> </format> </table> </panel> </row> </dashboard>   ... View more

Hi @darkins , It is actually simple, as long as you are comfortable with regex syntax. It will be like this: | eval condition=case(match(_raw, "thisword"), "first_condition", match(_raw, "thisotherword"), "second_condition", 1=1,"default_condition") | rex field=_raw "<rex_pattern>" if condition=="first_condition" | rex field=_raw "<rex_pattern>" if condition=="second_condition" | rex field=_raw "<rex_pattern>" if condition=="default_condition" Give it a try and let me know how it goes. ... View more

Hey @tread_splunk , Not gonna lie, it seems a bit confuse to understand your goal here. Both actions search and GET_PASSWORD only resides in _audit index, while internal index will have other kind of information. IF what you want is just use the internal logs to get the source clientip for that user (not exactly related to the action calls though) you can try something like this: index=_audit (action=search OR action=GET_PASSWORD) | stats count as audit_count by user | join user [ search index=_internal sourcetype=splunkd_access user=* clientip=* | stats count as internal_count by user clientip] | table user clientip audit_count internal_count The counts on audit and internal is the part that doesn't make much sense to me unless you want to filter the URI in the internal logs to something that is triggered during action=search or action=GET_PASSWORD, so you can customize my query a bit more. If I'm tripping, please help me understanding your goal so I can try to give you more insights if any. ... View more