This is confusing because your search specifically sets the values you want to remove.  Simple solution is to remove the last pipe and eval of the additional field.  Assuming you need that for some alternate reason then I would recommend. 1) Create base search "ds_base" don't include the pipe and eval of the overlay 2) Create the viz and map the data source to the base search 3) Create a chain search which has the pipe and eval of the overlay field and map it to the base 4) Map the alternate need to the chain search as the source ... View more

Currently the only event is an onClick type trigger as far as I can see in the documentation. https://docs.splunk.com/Documentation/Splunk/9.3.1/DashStudio/WhatNew Since it appears what you want to do is trigger a search and then wait you might want to look into the recently added Submit button options which should allow you to trigger a data source research on demand. ... View more

Yes it is possible. Dropdown configuration: Rectangle configuration: My test run visual results: The left side is just a markup window which shows me the value of the token.  The dropdown is initially populated by a local system index list. ... View more

Since nginx is forwarding some logs you know the connection is functional.  So then when you mention not all logs like WAF and DoS do you mean none of those message types are ingested at Splunk or just some messages of those types are not ingested. If all messages like WAF and DoS then perhaps a filter update is required, what happens to messages that do no have a matching filter is there a catch all index setup? Any packet captures to demonstrate the WAF and DoS messages are forwarded from nginx to sc4s.  ... View more

What is your search string behind the viz?  It could be as simple as appending the search with... | fields - overlay_field_name ... View more

I think I understand what you are asking about but without sample ingested data and the new output sample it is harder to decipher what is going wrong. ... View more

No option to wrap it - honestly you may want to replace your viz with Tables which can treat all of these as text and can wrap.   Showing a lot of information like you are there are options that a table will provide that may fit better. ... View more

https://docs.splunk.com/Documentation/Splunk/9.3.1/Admin/MigrateKVstore#Migrate_the_KV_store_in_a_single-instance_deployment Seems your first install of Splunk contained MongoDB for KVStores.  Very early in the release of 9 was supposed to contain an out of band upgrade from mmap to wiredTiger.  The link above is how the DB upgrade was to be handled and may still work for you.  I would recommend reaching out to Splunk support if you have a contract with them for support while you process this now. ... View more

Developer guide does not provide any evidence of loading different icons based on selected theme.  The documentation states the order the Icons are loaded which are triggered based on quality of display and location in the app. https://dev.splunk.com/enterprise/docs/developapps/createapps#Add-icons-to-your-app File name Image size (pixels) Placement Example image appIcon_2x.png 72 x 72 App list on the Splunk Web home page for high-resolution displays   appIcon.png 36 x 36 App list on the Splunk Web home page for standard-resolution displays   appIconAlt_2x.png 72 x 72 App menus for Splunk bar and search bar for high-resolution displays   appIconAlt.png 36 x 36 App menus for Splunk bar and search bar for standard-resolution displays   appLogo.png 160 x 40 (maximum size) App bar on standard-resolution displays — appLogo_2x.png 320 x 80 (maximum size) App bar on high-resolution displays — ... View more

Load the Monitoring Console Indexing -> Performance -> Indexing Performance: Instance Select various Indexers in your cluster to compare - If various Indexers have massively different queue values then you may have a data imbalance, since UF's by default stick to an ingestion queue for 30 seconds you should observe this over time. - If all queues left to the right are full then this is a disk write issue, the indexer can't write to disk fast enough. - You can via .conf settings override default indexer queue and pipeline settings to increase available size, but you should be very confident in your admin abilities and I don't recommend this for novice administrators.  Working with Splunk support is recommended regardless of your experience novice or advanced.  ... View more

Doing that to the Search Heads can cause more troubles than it's worth.  Best to backtrack that change. Then opt for a transforms.conf option to rewrite the host field value. [hostname-override] SOURCE_KEY = MetaData:Host REGEX = . FORMAT = host::$HOSTNAME ... View more

Since both of these are data source content issues it's difficult to determine from the Splunk side.  I would start with more research at a specific machine side.  What traffic is being generated and from what application.  Who if anyone is logged in live. ... View more

You really need to investigate your internal logs for bucket replication messages to get an idea of what is happening or not happening.  There are so many contributing factors to what could be occurring it would be difficult to provide an answer at this point. ... View more

Checking history in answers and Dell/EMC websites this has been an issue for a few years, no obvious solutions were ever provided. ... View more

Checking in other Answers it doesn't appear that "extracted_eventtype" is specific to DUO logs or app extractions.  Leads me to believe this is automagically generated at search time via Splunk default behavior. ... View more

Copy and Paste in a formatted or code window the exact command you are using.  The web browser accepts some special characters that something like wget can not interpret. ... View more

| tstats count where index=* index!=_* by host This will only give you a count of events, there is no insight to size of storage requirements.  You can add sourcetype and/or source after the host field if you need more detailed information. It all depends upon what your specific goals are.  ... View more

Since you have performed a tcpdump on the incoming packet you know that the format is based upon source configuration or by application design. 1) Can you correlate the format to specific hosts leading me to believe a configuration at the source as a possible solution 2) Do the specific hosts have different versions of fortinet installed, perhaps the vendor has modified the syslog message format between releases (unlikely but not impossible) 3) Are the message formats correlated to specific message types or services within the fortinet services, this would be less likely fixable via configuration.  The RFC for syslog was more of a suggestion than a hard rule and this is why vendors and applications don't often have a standard implementation of the recommended fields.  You have the potential through rsyslog destination configurations the opportunity to manipulate the output to add fields or clone fields into a specific order but most security rules indicate you should not modify logs in transit.  That might not be an issue for your implementation so you can look into it. The duplicate ingestion is interesting and much harder to pinpoint.  Depending upon frequency can you get a tcpdump to indicate if the message was generated at the source twice or did the UF monitoring the file have a hiccup. ... View more

Assuming you have co-located the syslog-ng app install on the same server where you have located your Splunk HF then there are a few options available to you.  You can continue to create the stand alone app to provide some experience and learning opportunities. Or... Most people when running syslog-ng will set the destination of syslog events to a file and separated by some sort of host details.  Since the file is local to the disk then you can leverage the HF web interface to set a Data Input monitor file option and it will guide you through common event breaking, line breaking, time extraction options, and some field extraction options.  Essentially all the things a stand alone app would do but easier to manage down the road by continuing to use the web interface. In a previous life I set the destination of the syslog-ng as a HEC receiver.  Which in your situation can be the local host HF or the IDX cluster, but that takes a bit of work and you already have a lot of development to write to file so maybe not the right idea for you. ... View more

Let Splunk manage the buckets for you. https://docs.splunk.com/Documentation/Splunk/9.3.0/Indexer/Automatearchiving Currently your script doesn't seem to have any filter on cold and would appear to copy all files so is this a one time execution as this on a cron would seem to copy over and over again creating a storage issue. Benefits of Splunk Management 1) Frozen reduces storage as the raw data in compressed format is stored, the IDX files are stripped away 2) Easy methods to reintroduce Frozen data back to thawed 3) Expands automatically if you have IDX clustering 4) Your not duplicating storage of Cold qualified time spans in a manual frozen folder 5) Folder management and creation is automatic ... View more

Leverage your monitoring console as the easy method to check volume sizes on each of your indexers.  Ideally the total space should be absolutely identical across all indexers, ie 100MB x 8idx/site x 2 sites (completely made up numbers). | rest splunk_servers=* /services/data/index-volumes Run that SPL on your search head and it will return for all servers in your search cluster and indexing cluster.  You can add more search terms to get down to the indexer level and then transform the results for "/dataHot", "/dataCold", and "_splunk_summaries".  Look at the per server results for used vs available/total and create a calculated field for %used.  Anything above 85% for /dataCold is typically a strong indication you need to expand your storage capabilities.  Note that "/dataHot" by design runs full before it will roll a bucket over to the /dataCold volume. ... View more

Do you have Cold Volume storage restrictions?  The offline may have cold buckets that want to replicate back to the always on site which that site may have removed due to volume utilization restrictions.  Do you have any details in your internal logs which indicate which buckets are not replicating, anything special about those specific buckets? ... View more