Any reason why you don’t use inputs.conf with blacklist on source side? ... View more

You cannot set like 1st example with GUI. It always set the default output which must be some other splunk instance. When you want to set default as devNull you must do it with conf files. ... View more

I don’t propose to do above commands as those have several really bad side effects! ... View more

Hi can you show how those files are in file system (e.g. find /…. -type f)? Of course mask real IPs, FQDNs etc. Couple of lines is enough.  You could check if your splunk user could see & read those by trying ls and cat for those as splunk user. If it cannot see those or content of those then you should use setfacl to give access to only splunk user. Never use any chmod which gives access to all users! This is actually security breach… One thing which you could try as splunk user splunk list inputstatus which shows is splunk read those files and if how much it has already reads. r. Ismo ... View more

Hi i think that this is the correct instruction for this case https://splunk.my.site.com/customer/s/article/KV-Store-Error-after-upgrading-Splunk-Enterprise The issue is with server name, not with CA. So try to disable sslVerifyServerName attribute like above instruction is guided. r. Ismo ... View more

I see that you are running splunk on windows? I haven’t so much experience how window’s internals works in current versions, but are you sure that splunk can use all that added memory without additional configuration? E.g. in Linux you must run at least disable boot-start and re-enable it again. Otherwise systemd didn’t know that splunk is allowed to use that additional memory. ... View more

Hi I’m thinking this is just shortcut to test your written search and it works just like copy&paste your query into another window. If you want to fulfill your “tokens” etc into that query then you must run it from dashboard not from this edit mode. I haven’t DS on my hand to test it now, but you could do it easily by yourself. r. Ismo ... View more

You can add base forwarding (forward all events) to target host with gui. But if/when you need so send only some events in that target and other to some another target then you must do it with conf files.  Anyhow I strongly recommended you to do this kind of base configurations by apps! In that ay those are much easier to admin, especially in larger environments. Also your auditors etc. are happier when you are fulfilling their requirements. ... View more

When your hf isn’t indexing events, it doesn’t take care anything what you have defined in indexes.conf. Only place where it use that conf file is give you list of index names from you could choose index where events are stored on your indexers. Personally I didn’t configure any index definitions on HFs unless there are any modular inputs, which wants those in GUI and I cannot configure those by .conf files. ... View more

Usually there is only one outputs.conf file. There is defined where those events are sent. If/when the next step is your indexer cluster then the easiest way is use indexer discovery feature (see docs.splunk.com). If the next step is another intermediate forwarder or individual indexer(s) then just add those there. See more from outputs.conf definition. I propose that you will do a separate app, which contains just this outputs.conf and other recommended conf files. If you are packing those apps into zip files, then you must extract those back into correct directory hierarchy in …/etc/apps folder. ... View more

When you have set in outputs.conf as described earlier in this thread all events goes only into indexers. Of course you can log in into HF and look if there are anything under /opt/splunk/var/lib/splunk where all indexes are. Just check if in your own indexes have events or not. ... View more

Hi You should start with dev.splunk.com where is described how to create an app. That documentation is base for understanding app development for Splunk. Then there are https://docs.splunk.com/Documentation/Splunk/9.4.1/AdvancedDev/Whatsinthismanual where you can found more information. Also conf.splunk.com contains lot of good presentations. Personally I haven't use YouTube.com for splunk, but there should be also something about this. For details about props.conf and transforms.conf you can found https://docs.splunk.com/Documentation/Splunk/9.4.1/Admin/Propsconf With google you will found lot of other sources too. r. Ismo ... View more

If that name which is in event's metadata is exactly sama than in indexes.conf file on that indexer where events comes. Basically you can change that name in props+transforms.conf on first full splunk instance (in your case that HF which collect that data). If indexers haven't have correctly named index in their index list then there should be index called lastChangeIndex see below lastChanceIndex = <index name> * An index that receives events that are otherwise not associated with a valid index. * If you do not specify a valid index with this setting, such events are dropped entirely. * Routes the following kinds of events to the specified index: * events with a non-existent index specified at an input layer, like an invalid "index" setting in inputs.conf * events with a non-existent index computed at index-time, like an invalid _MetaData:Index value set from a "FORMAT" setting in transforms.conf * You must set 'lastChanceIndex' to an existing, enabled index. Splunk software cannot start otherwise. * If set to "default", then the default index specified by the 'defaultDatabase' setting is used as a last chance index. * Default: empty string where all those events ,which haven't have name which exists in indexes.conf files, will send. As you see there is no default for that and this leads that indexer drops those events. For that reason you should always have this lastChangeIndex defined and restricted access to it. Also you should monitor it to react when someone try to send events into wrong index. ... View more

Here is more about it IndexAndForward Processor----- and indexAndForward attribute  #----TCP Output Global Configuration ----- # You can overwrite the global configurations specified here in the # [tcpout] stanza in stanzas for specific target groups, as described later. # You can only set the 'defaultGroup' and 'indexAndForward' settings # here, at the global level. # # Starting with version 4.2, the [tcpout] stanza is no longer required. [tcpout] indexAndForward = <boolean> * Set to "true" to index all data locally, in addition to forwarding it. * This is known as an "index-and-forward" configuration. * This setting is only available for heavy forwarders. * This setting is only available at the top level [tcpout] stanza. It cannot be overridden in a target group. * Default: false   ... View more

Except newer DS which must have it like this [indexAndForward] index = true selectiveIndexing = true   On HF side it's actually depends on TA are those indexes needed on that server or not. At least some e.g. DBX have this field as text where you could just write those names instead of select those from popup list. ... View more

You @livehybrid are correct. It should be 1st send all to nullQueue and then select those events which you want to keep. ... View more

Hi usually this should do when you are migrating to new server. There are couple of old posts how this can do e.g. https://community.splunk.com/t5/Installation/Upgrading-and-migrating-to-a-new-host-how-to-migrate-large/td-p/601048 I’m afraid that your new server have all same (named) indexes as old node have? Also they have starting index numbering from start (0 or 1). This leads that you could have equally named buckets in both nodes (at least hot buckets). For that reason you couldn’t do rsync without losing some events or you are in situation when splunk cannot start. https://docs.splunk.com/Documentation/Splunk/9.4.1/Indexer/HowSplunkstoresindexes#What_the_index_directories_look_like see bucket names. In theory there are probably two or three way to do it.  Create a new clustered indexers add/migrate those both nodes there, then transfer all buckets into one node and then go back to single node or keep it as one node cluster. Transfer old indexes into new node but change index names and later use both indexes when you are make queries and/or create eventtypes for those. Transfer old indexes to new node and ensure that there haven’t been any bucket name collisions. You must rename those directories before transfer. You also must update <index name>.dat file to highest <local id>+1 before starting nod again. It’s mandatory to test these scenarios before you do it with your production nodes! Also take offline backups before transfer, so you could rollback if/when needed. I haven’t try any of these as I have always migrate the data into new node before switchover. Also ensure that you have same splunk version on both node! If I must do this I probably chose 1st option? r. Ismo   ... View more

This shows  /opt/log/ type = directory /opt/log/cisco_ironport_web.log file position = 207575 file size = 207575 parent = /opt/log/ percent = 100.00 type = finished reading that splunk has read this one log file 100%. This means that it had sent it to indexers (I suppose that this has defined in your inputs.conf).  Why you don’t see those? There could be several reasons for that wrong timestamp recognition  wrong index definition  you have some transformations for drop those something else To tell the real reason you should try to query those e.g.  index=* earliest=1 latest=+5y that shows if those have wrong time or those have gone to wrong index. You should also check all conf files from UF to indexers and SH to see if there is something weird. ... View more

It good to know that. Then this (on UF) splunk list inputstatus Shows to you what inputs your UF sees and what it has read. ... View more

Hi I’m expecting that you have Splunk trial not free license? Free license doesn’t contain most of those features which you are trying to use! The easiest way to check why those files are not accessible is just sudo/su to your Splunk UF user and check if it can access those or not. If not the add permissions as @livehybrid already told. If it can access those, then start to debug with logs and e.g. with  splunk list inputstatus etc. You could find quite many posts here where this issue is already discussed and solved. r. Ismo  ... View more

This is depending from those apps. You must first check which are working in which splunk versions. It’s quite probable that you need to update those also step by step as it’s quite possible that same version doesn’t work on 7.x and 9.3. Also it’s possible that some apps don’t work anymore in 9.3. Also some may need OS level updates like OS version, Java or python updates etc. Depending of your data and integrations you should even think and plan if it’s possible to setup totally new node up with fresh install and newest apps. That could be much easier way to do version update? Of course it probably needs that you could leave the old node up and running until its data have expired. Also you must transfer license to new server and add old as a license client for it. ... View more

As already said, please define what you are meaning with word integrate! Here is one conf presentation about splunk and power bi https://conf.splunk.com/files/2022/slides/PLA1122B.pdf ... View more

I haven’t use slack alert action, so I just give general hints. Usually alert actions are written some log what happened into _internal index you should try to found something which is related to it. ... View more

Please use code block </> editor button to add your code. Otherwise it could meshed by editor. ... View more

Hi Have you read what has changed when splunk is updated from 8.1 to 9.1? There is read me first document(s) which told those. Especially there could be some removed features which have worked on old but not in a new version! Also if/when there are versions with higher patch level x.y.Z then you usually should select those instead of lower. https://docs.splunk.com/Documentation/Splunk/9.1.1/Installation/AboutupgradingREADTHISFIRST For example you found this from it “Splunk supports a direct upgrade to Splunk Enterprise 9.1 from versions 8.2.x and higher only”! If you have updated directly from 8.1.1 to 9.1.1 this is not supported and now you have missed some important migration steps which modified needed component between versions. Currently splunk support upgrades over only one minor version like 8.1 to 9.0 or 8.2 to 9.1. Also you should always train/test with test environment first and after you see that everything is ok then do those same steps with production. Your best and only supported solution is use your backup and do your upgrade again with supported path. Also you must start splunk in each versions which you are using on path from source to destination version! It didn’t do those migration steps with this. If you haven’t a backup then probably best option is create support ticket and ask if they have any instructions how you could try to fix the situation. r. Ismo ... View more