There is a much easier solution for your requirement. You can use the collect command to route the data of your search to your desired index. You can even specify a sourcetype of your choice, but that will result into the data being counted against your license. By default, collect command uses sourcetype as stash. Please consider that while doing this.   your search | collect index=<your_index>   More about the command: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Collect +++ Please consider accepting as an answer if this helps +++ ... View more

You're welcome. If you need more assistance in this, please feel free to reach out or put a new question in the community. Happy Splunking and best wishes. ... View more

Will this is be of some interest for your needs? https://splunkbase.splunk.com/app/2648 ... View more

Are you using Victoria experience or classic experience in Splunkcloud. If you are using classic experience, please file a Splunk support case to check and get SOAR's IP address whitelisted. If you are using Victoria experience, you can go to Settings -> Server Settings -> IP allow list for doing the same. Please note that this requires a user with sc_admin role assigned to it. ... View more

Well, let's gather a bit more information for an effective solution. How are you sending logs from the VM to Splunk UF? Is Splunk UF installed on a linux or windows server? A little more detail will help to identify the set up and then get a good solution. Regarding syslog-ng or rsyslogd, these are the services which come by default in Linux and can be installed in Windows. They work separately like any service and their job is to listen on ports that you specify in their configuration for incoming logs and then write them to the path in the server which you also specify in their configuration files.  For your current issue, multiple servers are trying to connect to Splunk on a single port. Splunk is attempting to listen to them but it can only do one at a time, which makes the others wait for their turn and they get the error "Requested server/endpoint cannot be reached". You need to redistribute this work load either by giving each server a different port to connect to Splunk UFs (Ex: 1514 for server A, 1515 for server B and so on) or use a lot of other servers with Splunk on them, neither which is an appropriate solution. So, how do we address this? Effective solution:  Get a server (Preferably linux), configure either syslog-ng or rsyslogd to listen to the incoming logs from your VMs and have them write down on the disk. Then make Splunk UF monitor those directories using [monitor:///] stanza in the inputs.conf. Or install Splunk UF on each VM and point them directly to Splunk indexers. ... View more

Updating the thread: HTML dashboards have been deprecated and must be converted using Splunk dashboard studio as Splunk has moved away from them. They cannot be fixed or used. ... View more

You need to ensure that the FortiSOAR's IP address is whitelisted in Splunkcloud  as it is mostly geofenced for initiating connections via the API. Also, the user that you have created should have enough access to do what you are trying to accomplish by having appropriate role assigned to it in Splunk cloud. Hope this helps. ... View more

Try the following: index=_internal sourcetype=scheduler status=skipped | stats values(reason) as reason, count by savedsearch_name When you run the search, let it execute, then click on "Save As" on the top right hand corner, then click on save as alert, fill in the details in the dialogue box which is pretty straight forward (If you want all results in one email, select in the dialogue box Once, if you want an individual email for each search, then select for each results) and then select the alert action as per your requirement. Ex: Send email alert action to send an email to you and others. Hope this helps. ... View more

Requires more context but by the looks of it your UF is getting overwhelmed with the amount of data it is receiving. It is listening to a server sending logs on the port 1514, while other servers are trying to communicate on the same port as well, leading for them to wait for the port to get free and logs to queue up, just like you said. You can either add more servers to outputs.conf and then send the logs there for further ingestion or try sending logs to different ports in Splunk for different servers by configuring inputs.conf accordingly, which IMO is not an effective architecture. Quick question: Why don't you use a service like Syslog-NG or rsyslogd to listen to the incoming logs via Syslog and write them on the disk of the server and then have the Splunk UF on it monitor those directories and ingest the logs to Splunk. You can configure logrotate to clear out the accumulating logs older than a day or even zip them for higher retention. Easy to maintain and less chances of data getting queued up. Also, are you using one UF or multiple? ... View more

Are you using a syslog service like syslog-ng/rsyslogd or Splunk TCP/UDP? If its syslog-ng/rsyslogd, you may want to look in /var/log/messages and see if there's any error that the syslog service is encountering. If its Splunk TCP/UDP using a HF, then it can be a bunch of reasons behind the drop. A usual quick fix is restarting the splunk service on the HF. You can also try restarting the syslog-ng/rsyslogd service and see if that fixes the issue. If you are using a linux server as a syslog server and forwarding via a UF, then you may want to consider the queues utilization, especially the tcpout queue. One more thing to consider is UDP. If there's a network latency/issues, the packet may get lost during the transit. More can be commented on this by me/other folks once you clarify a bit more about how are you onboarding the data. ... View more

Tried changing the start_date_time and restarted Splunk, didn't help. It still tries to pull the data from 8 days ago. ... View more

Hello @jconger , Could you please help me to find as to where is the checkpoint containing the timestamp for the last log being pulled stored? We ran into an issue with the credentials, took a week to get them resolved and now it won't pull the logs older than 8 days, so we need to change the date of the last log being pulled to something new. Thank you, ... View more

Okay. This seems interesting though. We've eliminated the most common reasons for this issue. The only ones that remain are: Checking if there are events with timestamp extraction issues by searching the string "is suspiciously far away from the previous event's time" and checking if it is happening for your affected log source. Network connectivity issue between HF and Indexers (Which is highly unlikely)   ... View more

Ah. Was the instance recently restarted as well? If there's no problem with the log source, you should not face the issue again anytime soon hopefully. Restarting the indexers rolls all the hot buckets to warm, so that would have done the trick for now. ... View more

Could you please run the following search for last last 7 days and see if it returns the name of the affected indexer. If it doesn't returns a result, please try to take the "Received shutdown signal." string and run it in the splunkd.log of the indexer (If you have access to its box). index=_internal "Received shutdown signal." sourcetype=splunkd component!="SearchParser" | dedup host | stats max(_time) as _time by host   ... View more

Interesting. Could you kindly share as to which version of Splunk are you using and what's the % of hot buckets that Splunk is giving in the error message. Please try running the search from this post and see if the index that you got the error message for gets identified.  https://community.splunk.com/t5/Getting-Data-In/The-percentage-of-small-of-buckets-is-very-high-and-exceeded-the/td-p/435706?_ga=2.190394606.491538500.1654181553-1264184537.1654181553&_gac=1.15517442.1652887736.EAIaIQobChMI9cytzq7p9wIVChvUAR3_OQNbEAAYASAAEgKeLvD_BwE&_gl=1*1bpvpx5*_ga*MTI2NDE4NDUzNy4xNjU0MTgxNTUz*_gid*NDkxNTM4NTAwLjE2NTQxODE1NTM.*_gac*RUFJYUlRb2JDaE1JOWN5dHpxN3A5d0lWQ2h2VUFSM19PUU5iRUFBWUFTQUFFZ0tlTHZEX0J3RS4xNjUyODg3NzM2#answer-766370   Could you also check if: The issue is present in all of your indexers. Were the affected indexers restarted recently and had any issues accepting the data (Can be found by looking into splunkd.log) ... View more

My bad. I wrote the timeranges incorrectly for the search. Have updated the answer above. ... View more

"A few hours offset" can be a big contributing factor here. How big of an offset are we talking here? If you haven't changed the value of maxHotBuckets for the index, it defaults to auto, which the indexers use to the set the value to 3. If the timestamp of the data is all over the place (Some in the future, some really old data), Splunk ingests it but force roll a hot bucket to create a new one for the data with unusual timestamp and if it happens frequently, you find a lot of small hot buckets being created in a short span of time. Please check if the data coming from AWS is being accepted and stored "in the future" by running searches with index=yourindex sourcetype=aws:* earliest=+5m latest=+7d. If the volume is considerably large, this could be a big contributor to the error. Please look for errors like "Accepted time is suspiciously far away from the previous event's time". This can tell you if the events that got ingested have timestamps far back in the past and Splunk ended creating hot buckets for them. Creating a custom props.conf to define TIME_PREFIX, MAX_TIMESTAMP_LOOKAHEAD and TIME_FORMAT along with line breaking to ensure that Splunk reads the timestamp properly from your data. Doing a sanity check of the data itself. I've seen some log sources in  some environments, where timestamp within the log source was all over the place all the time. Ended up with DATETIME_CONFIG = current to resolve the problem.  Hope this helps, ###If it helps, kindly consider accepting as an answer/upvoting### ... View more

It's a "not supported" add-on, so not sure if anyone would have an ETA for this. @jconger , @abalogh_splunk might be able to shed more light on it as they contributed immensely to the development of it. ... View more

Yes it should be able to. You can configure multiple multisite indexers to be searchable to a SH cluster. https://docs.splunk.com/Documentation/Splunk/8.2.6/Indexer/Configuremulti-clustersearch ###If it help, kindly consider accepting as an answer/upvote### ... View more

I've updated the regex above. Please try that. Should extract the sentences that you listed as examples. If it needs more modification, kindly share some sample data to create an accurate regex. ###If it helps, kindly consider mark as accepted answer###     ... View more