I would do in this order HF, LM, CM, IDX ... View more

Yes you can do it, as suggested by others check DNS resolution, FW rules, routing, etc... If you have multiple HFs in HA, start from one of them so you can intercept the issues ... View more

Hello, It seems it requires a connection to Splunk Cloud: https://help.splunk.com/en/splunk-cloud-platform/search/splunk-ai-assistant/2.0.0/install-and-configure-splunk-ai-assistant/install-splunk-ai-assistant-for-splunk-enterprise-customers-with-cloud-connected You do not need to be a Splunk Cloud customer in the sense of hosting your Splunk environment in Splunk Cloud, but you do need Splunk’s cloud-connected AI service and activation. You should check this with your Splunk Account Sales/Engineer. Best Regards, Edoardo ... View more

See my reply here if it can help https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013 ... View more

See my reply here if it can help https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013 ... View more

See my reply here if it can help https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013 ... View more

See my reply here if it can help https://community.splunk.com/t5/Deployment-Architecture/Splunk-Storage-Sizing-Guidelines-and-calculations/m-p/708258/highlight/true#M29013 ... View more

Things have improved a lot thanks to tsidxWritingLevel enhancements. If you set tsidxWritingLevel=4, the maximum available today, and all your buckets have been already written with this level you can achieve a compress ratio of 5.35:1 This means 55 TB of raw logs will occupy around 10 TB (tsidx + raw) on disk. At least this is what we have in our deployment. This number can vary depending on the type of data you are ingesting. Here the query I used, running All Time, starting from the one present in the Monitoring Console >> Indexing >> Index and Volumes >> Index Detail: Instance   | rest splunk_server=<oneOfYourIndexers> /services/data/indexes datatype=all | join type=outer title [ | rest splunk_server=<oneOfYourIndexers> /services/data/indexes-extended datatype=all ] | `dmc_exclude_indexes` | eval warm_bucket_size = coalesce('bucket_dirs.home.warm_bucket_size', 'bucket_dirs.home.size') | eval cold_bucket_size = coalesce('bucket_dirs.cold.bucket_size', 'bucket_dirs.cold.size') | eval hot_bucket_size = if(isnotnull(cold_bucket_size), total_size - cold_bucket_size - warm_bucket_size, total_size - warm_bucket_size) | eval thawed_bucket_size = coalesce('bucket_dirs.thawed.bucket_size', 'bucket_dirs.thawed.size') | eval warm_bucket_size_gb = coalesce(round(warm_bucket_size / 1024, 2), 0.00) | eval hot_bucket_size_gb = coalesce(round(hot_bucket_size / 1024, 2), 0.00) | eval cold_bucket_size_gb = coalesce(round(cold_bucket_size / 1024, 2), 0.00) | eval thawed_bucket_size_gb = coalesce(round(thawed_bucket_size / 1024, 2), 0.00) | eval warm_bucket_count = coalesce('bucket_dirs.home.warm_bucket_count', 0) | eval hot_bucket_count = coalesce('bucket_dirs.home.hot_bucket_count', 0) | eval cold_bucket_count = coalesce('bucket_dirs.cold.bucket_count', 0) | eval thawed_bucket_count = coalesce('bucket_dirs.thawed.bucket_count', 0) | eval home_event_count = coalesce('bucket_dirs.home.event_count', 0) | eval cold_event_count = coalesce('bucket_dirs.cold.event_count', 0) | eval thawed_event_count = coalesce('bucket_dirs.thawed.event_count', 0) | eval home_bucket_size_gb = coalesce(round((warm_bucket_size + hot_bucket_size) / 1024, 2), 0.00) | eval homeBucketMaxSizeGB = coalesce(round('homePath.maxDataSizeMB' / 1024, 2), 0.00) | eval home_bucket_capacity_gb = if(homeBucketMaxSizeGB > 0, homeBucketMaxSizeGB, "unlimited") | eval home_bucket_usage_gb = home_bucket_size_gb." / ".home_bucket_capacity_gb | eval cold_bucket_capacity_gb = coalesce(round('coldPath.maxDataSizeMB' / 1024, 2), 0.00) | eval cold_bucket_capacity_gb = if(cold_bucket_capacity_gb > 0, cold_bucket_capacity_gb, "unlimited") | eval cold_bucket_usage_gb = cold_bucket_size_gb." / ".cold_bucket_capacity_gb | eval currentDBSizeGB = round(currentDBSizeMB / 1024, 2) | eval maxTotalDataSizeGB = if(maxTotalDataSizeMB > 0, round(maxTotalDataSizeMB / 1024, 2), "unlimited") | eval disk_usage_gb = currentDBSizeGB." / ".maxTotalDataSizeGB | eval currentTimePeriodDay = coalesce(round((now() - strptime(minTime,"%Y-%m-%dT%H:%M:%S%z")) / 86400, 0), 0) | eval frozenTimePeriodDay = coalesce(round(frozenTimePeriodInSecs / 86400, 0), 0) | eval frozenTimePeriodDay = if(frozenTimePeriodDay > 0, frozenTimePeriodDay, "unlimited") | eval freeze_period_viz_day = currentTimePeriodDay." / ".frozenTimePeriodDay | eval total_bucket_count = toString(coalesce(total_bucket_count, 0), "commas") | eval totalEventCount = toString(coalesce(totalEventCount, 0), "commas") | eval total_raw_size_gb = round(total_raw_size / 1024, 2) | eval avg_bucket_size_gb = round(currentDBSizeGB / total_bucket_count, 2) | eval compress_ratio = round(total_raw_size_gb / currentDBSizeGB, 2)." : 1" | fields title, datatype currentDBSizeGB, totalEventCount, total_bucket_count, avg_bucket_size_gb, total_raw_size_gb, compress_ratio, minTime, maxTime freeze_period_viz_day, disk_usage_gb, home_bucket_usage_gb, cold_bucket_usage_gb, hot_bucket_size_gb, warm_bucket_size_gb, cold_bucket_size_gb, thawed_bucket_size_gb, hot_bucket_count, warm_bucket_count, cold_bucket_count, thawed_bucket_count, home_event_count, cold_event_count, thawed_event_count, homePath, homePath_expanded, coldPath, coldPath_expanded, thawedPath, thawedPath_expanded, summaryHomePath_expanded, tstatsHomePath, tstatsHomePath_expanded, maxTotalDataSizeMB, frozenTimePeriodInSecs, homePath.maxDataSizeMB, coldPath.maxDataSizeMB, maxDataSize, maxHotBuckets, maxWarmDBCount | search title=* | table title currentDBSizeGB total_raw_size_gb compress_ratio | where isnotnull(total_raw_size_gb) | where isnotnull(compress_ratio) | stats sum(currentDBSizeGB) as currentDBSizeGB, sum(total_raw_size_gb) as total_raw_size_gb | eval compress_ratio = round(total_raw_size_gb / currentDBSizeGB, 2)." : 1"         ... View more

I had a reply from the Splunk Support, it seems that since a while init.d is not supported anymore as mentioned here: https://docs.splunk.com/Documentation/Splunk/9.2.1/Admin/ConfigureSplunktostartatboottime   "The init.d boot-start script is not compatible with RHEL 8 and higher. You can instead configure systemd to manage boot start and run splunkd as a service. For more information, see Enable boot start on machines that run systemd."   In fact I have this issue on a Oracle Linux 8 machine.   ... View more

Some additional details why it happens here:   https://community.splunk.com/t5/Getting-Data-In/splunk-winprintmon-exe-splunk-winPrintMon-monitorHost/m-p/680673 ... View more

You are welcome. I would try checking based on what is written here: https://docs.splunk.com/Documentation/Splunk/latest/Data/TroubleshootHTTPEventCollector In particular: 1- Check if HEC token is enabled (I guess so :-)) 2- Verify if ACK is enabled 3- Look at the log file directly in the machine $SPLUNK_HOME/var/log/introspection/splunk/http_event_collector_metrics.log 4- Run a more general query index="_introspection" token 5- Enable logs in DEBUG ... View more

Hello @nunoaragao , unfortunately I don't have access anymore to the Splunk UF to perform a check. Never had access to the third party Splunk where we were sending the data. By the way I didn't really get which is the issue you are facing. Please remember that in outputs.conf you don't have to explicit the HEC endpoint (/services/collector/s2s) but just the URI (https://yourdomain.com) uri=https://yourdomain.com/services/collector/s2s ... View more

Hello, Yes I have been able to find a good way to do it. I wanted to write a solution post for this topic but I never had chance. I’ll do it providing all the steps and config. To summarize the way I found is: 1- in Azure AKS in diagnostic settings (if I remember well) you can decide to spool the logs you need into a Storage Account or a Streaming Service. If you don’t need real time go with Storage Account that is cheaper.  2- you then read with Microsoft TA from that Storage Account every 5 minutes 3- you set-up a policy to cancel data older than 7 days from your Storage Account. Retention policy can be adjusted as per your preference, but here act mostly like a buffer. In this way the cost will be under control. Also, about the REST API billing I didn’t see much of a difference honestly. 4- the Microsoft TA modular input seems having a bug. Basically scheduling it every 5 minutes after several hours it stopped working. As a workaround I downloaded an app with an SPL command that allows you to reload the endpoint you want. I embedded it into a scheduled search that run every 5 minutes, keeping the modular input every hour. In this way it is the scheduled report that trigger the data download. Schedule frequency need to be higher than the time it takes to download your data from the Storage Account and then parse them 5- once you download the data you then have to parse removing the unwanted data. Unfortunately it is a JSON into another JSON, and you need the nested one. I did this for AKS audit but probably can be easily adjusted for other typology of logs As soon as I have some time I will provide the config as well. Best Regards, Edoardo ... View more

@splunkreal ideally if you create the app you should put the configs in the default folder. You should see you as the author. About your question, yes when you deploy an app the entire app folder on the client is replaced by the new one. Therefore if you first manually created an app on a client (for test), and later you want to move the management of that app on the deployment server because you have several client, then is the deployment server that will drive. Since then any change need to be made on the DS. Best Regards, Edoardo ... View more

Thank you! Same issue here on Splunk 9.2.1 Splunk was NOT starting at boot-start (with init.d) but manually was starting correctly. After having commented the mentioned line is now properly booting with the VM (Oracle Linux). I am going to open a case to the support to inform them about it. ... View more

@dnavara please have a look on my explanation here: https://community.splunk.com/t5/Getting-Data-In/Why-has-the-index-process-paused-data-flow-How-to-handle-too/m-p/631226/highlight/true#M108187   ... View more

Thank you all, here are updated configurations adapted from the previous ones. NSG logs are written into an Azure Storage Account. Then a Splunk HF reads the logs from the Azure Storage account with "Splunk Add-on for Microsoft Cloud Services" and send back to the Indexers.   Configuration applied on the Splunk Heavy Forwarder (can be applied in an Indexer if you don't have an HF) hf_in_azure_nsg_app/default inputs.conf #Inputs is defined directly in the Splunk HF via WEB-UI with "Splunk Add-on for Microsoft Cloud Services" and can be found here /opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local/inputs.conf props.conf #NOTE: Following set-up allow to extract only the flowTuples from the payload and set _time based on flowTuples epoch #First LINE_BREAKER apply, then SEDCMD-remove_not_epoch that keeps only flowTuples, then TRANSFORMS with INGEST_EVAL that overwrite _time #flowTuples data parsing is done at search time in the Search Head with separate app #The "source" field already contains the resourceId informations (subscriptionID, resourceGroupName, nsgName, macAddress) that can be extracted on the Search Head at search time #NOTE 2: LINE_BREAKER has been enhanced to avoid extracting events with macAddress containing first 10 numeric digits #TO BE DONE: Understand if SEDCMD- has some limit on very huge payload #TO BE DONE 2: In the INGEST_EVAL with a case statement if length is lower than 10 digits valorize now() as _time #References: #https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview #https://community.splunk.com/t5/Splunk-Search/How-do-I-import-Azure-NSG-LOGs/td-p/396018 #https://community.splunk.com/t5/Getting-Data-In/How-to-extract-an-event-timestamp-where-seconds-and-milliseconds/m-p/428837 [mscs:nsg:flow] MAX_TIMESTAMP_LOOKAHEAD = 10 LINE_BREAKER = (\")\d{10}\,|(\"\,\")\d{10}\, SHOULD_LINEMERGE = false SEDCMD-remove_not_epoch = s/\"\D.*$|\{|\}|\[|\]//g TRUNCATE = 50000000 TRANSFORMS-evalingest = nsg_eval_substr_time transforms.conf [nsg_eval_substr_time] INGEST_EVAL = _time=substr(_raw,0,10)   Configuration applied on the Splunk Search Head sh_azure_nsg_app/default props.conf #References: #https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview #https://community.splunk.com/t5/Splunk-Search/How-do-I-import-Azure-NSG-LOGs/td-p/396018 #https://community.splunk.com/t5/Getting-Data-In/How-to-extract-an-event-timestamp-where-seconds-and-milliseconds/m-p/428837 [mscs:nsg:flow] REPORT-tuples = extract_tuple_v1, extract_tuple_v2 REPORT-nsg = sub_res_nsg FIELDALIAS-mscs_nsg_flow = dest_ip AS dest src_ip AS src host AS dvc EVAL-action = case(traffic_result == "A", "allowed", traffic_result == "D", "blocked") EVAL-protocol = if(match(src_ip, "^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$"), "ip", "unknown") EVAL-direction = case(traffic_flow == "I", "inbound", traffic_flow == "O", "outbound") EVAL-transport = case(transport == "T", "tcp", transport == "U", "udp") EVAL-bytes = (coalesce(bytes_in,0)) + (coalesce(bytes_out,0)) EVAL-packets = (coalesce(packets_in,0)) + (coalesce(packets_out,0)) EVAL-flow_state_desc = case(flow_state == "B", "begin", flow_state == "C", "continuing ", flow_state == "E", "end") transforms.conf [extract_tuple_v1] DELIMS = "," FIELDS = time,src_ip,dest_ip,src_port,dest_port,transport,traffic_flow,traffic_result [extract_tuple_v2] DELIMS = "," FIELDS = time,src_ip,dest_ip,src_port,dest_port,transport,traffic_flow,traffic_result,flow_state,packets_in,bytes_in,packets_out,bytes_out [sub_res_nsg] SOURCE_KEY = source REGEX = SUBSCRIPTIONS\/(\S+)\/RESOURCEGROUPS\/(\S+)\/PROVIDERS\/MICROSOFT.NETWORK\/NETWORKSECURITYGROUPS\/(\S+)\/y=\d+\/m=\d+\/d=\d+\/h=\d+\/m=\d+\/macAddress=(\S+)\/ FORMAT = subscriptionID::$1 resourceGroupName::$2 nsgName::$3 macAddress::$4 eventtypes.conf [mscs_nsg_flow] search = sourcetype=mscs:nsg:flow src_ip=* [mscs_nsg_flow_start] search = sourcetype=mscs:nsg:flow flow_state=B [mscs_nsg_flow_end] search = sourcetype=mscs:nsg:flow flow_state=E tags.conf [eventtype=mscs_nsg_flow] network = enabled communicate = enabled [eventtype=mscs_nsg_flow_start] network = enabled session = enabled start = enabled [eventtype=mscs_nsg_flow_end] network = enabled session = enabled end = enabled   Best Regards, Edoardo   ... View more

Hello, Do you mean the 200GB/day is for an 12vCPU/12GB RAM/900 IOPS Heavy Forwarder that is indexing locally and also forwarding to Indexers but not performing local searches? In this 200GB/day are you also including logs from internal indexes ( index=_* ) ? If so, what about an Heavy Forwarder with same specs that is not locally indexing? How many GB/day can process (internal and non internal logs)? Thanks a lot, Edoardo ... View more