Hi fellows,  It's time to migrate our good old Standalone Splunk Enterprise to 'Small Enterprise deployment'. I read through tons of docs, unfortunately, I didn't find any step-by-step guide, so I have many questions. May Some of you can help. - The existing server is CentOs7, the new servers will be Ubuntu 22.04. Just before the migration, I plan to upgrade Splunk on it from 9.1.5 to the latest 9.3.1.  (it wasn't updated because Centos 7 is not supported by 9.3.1)  OR do I set up the new servers with 9.1.5 and upgrade them after the migration? - Our daily volume is 3-400 GB/ day. It will not grow drastically in the medium term. What are your recommended hardware specs for the indexers? Can we use the "Mid-range indexer specification" or go for the "High-performance indexer specification" (as described at https://docs.splunk.com/Documentation/Splunk/9.3.1/Capacity/Referencehardware ) - If I understand correctly I can copy the /etc/apps/ from the old server to the new Search Head, so I will have all the apps, saved searches, etc. But what config files must be modified to get the data for the new indexers? (For forwarders this is clear, but we are using a lot of other inputs (Rest-API, HEC, scripted) - Do I configure our existing server as part of the indexer cluster (3rd member), then when all the replications are done on the new servers remove it from the cluster, or copy the index data to one of the new indexers - rename the buckets ( adding the indexer's unique id) and let the cluster manager do the job? (Do I need a separate Cluster manager or the SH could do the job?) And here comes the big twist... - Currently, we using S3 storage via NAS Bridge for the cold buckets. This solution is not recommended and we are already experiencing fallbacks. So, we planned to change the configuration to use SmartStore. How I can move the current cold buckets there? (a lot of data and because of the NAS Bridge this is very very slow to copy...)  Thanks in advance Norbert   ... View more

Hello, Recently we replaced our Syslog server from rsyslog to syslog-ng.  We are collecting the network device's log - every source logged its own <IPaddress.log> file. Universal forwarder pushing them to the indexer.  Inputs, outputs are ok the data flowing, sourcetype is standard syslog. Everything is working as expected... Except for some sources... I spotted this because the log volume has dropped since the migration. For those, I do not have all of the events in Splunk.  I can see the file on the syslog server, let's say there are 5 events per minute. The events are the same - for example, XY port is down - but not identical; the timestamp in the header and the timestamp in the event's message are different. (events are still the same length). So in the log file, there are 5 events/min, but in Splunk, I can see only one event per 5 minutes. The rest are missing... Splunk randomly picks ~10% of the events from the log file (all the extractions are ok for those, there is no special character or something in the "dropped" events...) I feel it is because of similar events - Splunk thinks they are duplicated - but other hand it cannot be, because they are different. Any advice? Should I try to add some crc salt or try to change the sourcetype? BR. Norbert   ... View more

Hi, I have a json field where multiple values listed separated by backslash in raw (space in list view) like this: "value": "audit_retention_configure\nos_airdrop_disable......\nsystem_settings_wifi_menu_enable\n" In list view the extraction looks ok, but the whole list shown as a single value. I would like to split it. I did this: Mysearch   | rename "extensionAttribute.value" AS value | search value="*" AND NOT value="No Base*" | eval values=split(value,"X") | mvexpand values | table values   If i set X="\" (unbalanced quotes), or "\\", or " " (space), there is no change in the result, if I set forexample "_", it will split the field by _ like a charm... Please advise what should I do for  audit_retention_configure nos_airdrop_disable . . . nsystem_settings_wifi_menu_enable result. ... View more