Hi!  Most things are possible. Let's try to figure what we are trying to achieve.  "| eval Created=substr(Created, 1, 7)" Is this generating a state i.e. "created" or is this a user_id or similar with multiple combinations of values?  "| eval a=if(State="Closed",1,0)"  Do you want to count the number of occurrences something was created  and closed?  maybe source="a.csv" OR source="b.csv" OR source="c.csv" Company="x" | eval created_by=substr(Created, 1, 7) | eval is_closed=if(State="Closed",1,0) | eval user_source = created_by.":".source | chart sum(is_closed), count by user_source OR  source="a.csv" OR source="b.csv" OR source="c.csv" Company="x" | eval created_by=substr(Created, 1, 7) | eval is_closed=if(State="Closed",1,0) | eval user_source = created_by.":".source | chart sum(is_closed), count by created_by, source   /Seb          ... View more

In the settings of the visualization you can choose to set 0 for null values.    ... View more

Is there any clue on what crashes the page?  I had problems with browsers struggling when 1000+ or more items are used in dropdowns. Easiest and most sensible way to solve it is to use a free text field to filter in these type of tokens.  /Seb  ... View more

In the user preferences you can specify the time zone. After selected all searches and results will be based on you selected time zone.  Pro tip is to always expose the time zone when working with this kind of topics.  time_with_timezone = strftime(_time, "%+") /Seb  ... View more

Maybe something like:    source="a.csv" OR source="b.csv" OR source="c.csv" Company="x" ```Expection State is either "Created OR Closed" ``` | eval state_source = State.":".source | chart count by state_source     /Seb  ... View more

It the problem only happening for the pdf? Is the dashboard loading as expected when you are opening it?  Could not reproduce any issues created a dashboard below:  <dashboard version="1.1"> <init> <set token="field1.earliest">-7d@d</set> <set token="field1.latest">now</set> </init> <label>time_selector</label> <row> <panel> <chart> <title>Events by sourcetype</title> <search> <query>| tstats count where index=_internal by sourcetype</query> <earliest>$field1.earliest$</earliest> <latest>$field1.latest$</latest> </search> <option name="charting.chart">line</option> <option name="charting.drilldown">none</option> </chart> </panel> <panel> <chart> <title>Events by sourcetype</title> <search> <query>| tstats count where index=_internal earliest=$field1.earliest$ latest=$field1.latest$ by sourcetype</query> </search> <option name="charting.chart">line</option> <option name="charting.drilldown">none</option> </chart> </panel> </row> </dashboard> ... View more

Hi!  Cooked connection to ip=x.x.x.x:9997 timed out.  That would be the connection to the indexer the UF is forwarding too. Is it not attempting to connect to the next indexer in the cluster?  Yes you might be onto something if indexers are at their limit the ingestion queues will fill up and eventually data from UFs will be blocked or at least delayed until the indexer can work. through the queues.  Use the monitoring console to check if the indexing queues are getting blocked could be a good start.  /Seb  ... View more

Hi!  Is the problem that the affected nodes stops forwarding data? or that they are flagged as unsatable?  Check if the affected nodes have the splunk UF running or not. To exclude whether the issue is due to the Splunk process not running or whether there is a data forwarding issue (firewall / network / ports etc).  If the Splunk process is not started when a server is rebooted have a look here on how to enable splunk boot start https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/ConfigureSplunktostartatboottime.  Also if these are Linux hosts note that there is a slightly different setup if the boot on start should be enabled with systemd: See https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/RunSplunkassystemdservice#Configure_systemd_using_enable_boot-start  /Seb  ... View more

@vinothnaga no not to my knowledge. Think your best option is partial backup.  Not that I know of,  since this is in regards to configure ITSI to be easier to migrate to another environment, not part of ITSI best practice itself.  Here you can find some useful information on ITSI in general https://www.splunk.com/pdfs/getting-started/splunk-getting-started-with-itsi.pdf   /Seb  ... View more

@vinothnaga  If you do a partial backup the entities will not be included. Try it out! You can also unzip the backups to have a look what is included.    /Seb  ... View more

I have come across this many times as well. Your best option in my opinion is to move the savedsearches, macros etc. from the /itsi/local app context to your self managed application context. for example project-x_itsi_addon. Then promote the configurations from this app to a higher environment.  I am not 100% how the adaptive thresholds search would act if you try migrate that one this way so be a bit extra careful with those.  In general it's also good practice to maintain your KPI searches and Correlation searches in your separate application context. You can there after use a reference to a savedsearch in the ITSI KPI/KPI Base/Correlation searches with "| savedsearch my_saved_search_from_app_1" Note that the my_saved_search_from_app_1 needs to be shared globally.  This will enable you to promote KPI search changes to another Splunk environment from your custom app context without needing to restore a ITSI backup.   /Seb     ... View more

Okey @taparshi1.  Looking around in the documentation and in the UI this use case does not seem to be supported. Please add it to https://ideas.splunk.com.   However in you situation I would solve it slightly different. You can manage information fields of the entities to achieve the same result.  Service Template: my_database_service_template For example create a service using a service template where you defined a set of entity filters. At the time you run the service import job you will set these filters to match a value from your CSV.  Simple Example:  Service name: my_oracle_db  Service Template: my_oracle_db_service_template  Entity filter: my_entity_type:oracle_db, my_qualifier_info_field:true  With the above setup you can instead manipulate the info field "my_qualifier_info_field" of the entities to activate / deactivate entities from your service. The qualifier for true/false can in turn be managed in a lookup, that in turn are used doing scheduled updates / imports of entities. This decouples the service from fixed list of entity host names and makes it adaptable to entities available that passes the entity filter.    /Seb  ... View more

What kind of alerts? check what process search are generating them and what their trigger conditions are and importantly what the values was at the time the alert war triggered.  /Seb ... View more

Do you specialise exactly which entities you want in the entity filter? and then use scheduled service imports to change that? I have not tried that.  One option as a walkaround could be to have more generic entity filter and instead use info fields of entities to pass the filter or not. I.e. let a scheduled import/update of the entities define whether they qualify for being used by the services.  /Seb   ... View more

You could consider searching the raw data in the glass table instead of using the itsi kpi summary.  Remember that the KPI will have a time unit. For example number of logins per minute the past one minute. If you change this KPI to be calculate every X seconds either change the KPI to calculate the number of logins for the past  X seconds as well.  Also note that there will always be some ingestion delay, i.e. time from when an event/metrics was created to when splunk will be able to retrieve it from an index. Configuring ITSI to search to close to the current time can case KPIs to show up as "NA" since no data for that time was available at the time the search was executed.    /Seb  ... View more

Hi!  What is not working for you the "/servicesNS/nobody/SA-ITOA/itoa_interface/entity/bulk_update?is_partial_data=1" endpoint? What was the response?  This works for me on SE 9.0.3  curl -k -u user:password https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/entity/bulk_update?is_partial_data=1 -H "Content-Type: application json" -X POST -d '[{"_key":"entity_guid","cpu":["56"]}]' Note that the payload needs to be a JSON list.        ... View more

Hi!  Try something like this:  | tstats latest(_time) as _time latest(alert_level) as alert_level latest(itsi_group_severity) as itsi_group_severity latest(itsi_group_status) as itsi_group_status where index=itsi_grouped_alerts AND NOT itsi_group_status::5 earliest=-24h latest=now by itsi_group_id   /Seb  ... View more

To add to joys answer. If you have that meta data of the entities available in Splunk and or in a CSV. You can use the import entities with search or CSV method to append additional info fields (meta data) to the entities.  See doc at: https://docs.splunk.com/Documentation/ITSI/4.15.0/Entity/ImportSearch /Seb  ... View more

Hi!  You should not use stats in a KPI search, in most cases that will never be needed. Reasons are: 1. ITSI itself is doing the "stats" for you. 2. stats if done wrong removes _time which might be the problem in you case.  I would suggest changing your KPI search to something like:  index=my_index sourcetype=my_type transaction=* | eval has_transaction=1  Now you can extract 2 KPIs depending on you use case, either sum of has_transaction (Number of logged transactions in past X min) or do a dc by transaction (Number of unique transaction in the past X min) /Seb          ... View more

No deleting service in ITSI does not affect the entities. In your case I would:  * identify what filters are used to filter entities to the services you are to delete. * Use the entity management view to verify if these entities used in any other services? If yes, do you really want to delete them? If no select and delete them from the entity management UI. Preferably before deleting the service so you can verify in the service that the entities now are gone.  Also note that if these entities and or services are created by a scheduled job, they might reappear when those jobs are run again.  /Seb ... View more

Creating a specific role to manage deployment servers  serverclass  I experienced the same issue with  User 'ds_user' with roles { ds_role, ds_user, user } cannot write: /nobody/system/serverclass/serverClass:My_server_class:app:my_app/restartSplunkWeb { read : [ * ], write : [ admin ] }, removable: no The ds_role has the capabilities: edit_deployment_client, edit_deployment_server, list_deployment_client list_deployment_server To be able to add an app to a serverclass the only option was to give the capability admin_all_objects. Which effectively would make ds_role users admins.   To avoid this our workaround was to edit /opt/splunk/etc/system/metadata/local.meta to grant write privilege for ds_role to serverclass objects #On Deployment Server #/opt/splunk/etc/system/metadata/local.meta [serverclass] access = write : [ admin, ds_role ] export = system       ... View more

To list the Glass Table Keys and Title you can use:    | rest /servicesNS/-/-/itoa_interface/glass_table report_as=text fields="_key,title" | eval value=spath(value,"{}") | mvexpand value | eval glass_table_id=spath(value, "_key"), glass_table_title=spath(value, "title") | fields - value   ... View more

I had a situation where Splunk web refused to start after upgrading to SE 8.0.1, with UnicodeDecodeError. In our situation if was due to the system locale being set to sv_SE. Changing it to en_GB helped. ... View more

To answer my own question, our problem was that the user running the splunk process had the locale sv_SE. This cased python3 to fail when importing formecode. Workaround is to set the system and user locale to en_GB or en_US [splunk@ip-10-202-18-65 ~]$ LANG=en_US.utf8 [splunk@ip-10-202-18-65 ~]$ /opt/splunk/bin/splunk start ... View more