Thank you. I would set this as the solution, but i can only do one solution 😞   Thank you for your time.  ... View more

Thank you both for the prompt response.    This is the impression I was under as well when speaking with our Splunk rep. Now that we are upgrading, the mis match of OS will only be for a week or two at most, but was wanting to confirm that the data availability/searchability wouldn't be affected. Thank you both for the resources and in depth answers given.  ... View more

Hey,  Thank you for assisting. 1PB is incredible. I though 1.7 TB was a lot. I am not too sure about instance type, I am reaching out to see. As far as Smart Store, we aren't using anything of the sort i believe. We just have retention policy to roll over cold/frozen data. Ill look over the docs regarding the smart store.    In regard to min requirements, a lot of our administration servers, Deployment Server, some of the servers handling syslog data, DMZ heavy forwarders, Cluster Managers, etc, all have around 6-8 Cores, and roughly around 6 CPUs.  The Server pictured below/above, is of our Azure  Cluster Manager. It manages a cluster, and may index itself, not sure, but it only has 8 CPU and 4 cores. Should all servers at least meet the minimum requirements? Especially with our ingestion load? I would imagine so. I can work with Support to answer any specific questions as I already have an ODS case open to handle this Splunk version upgrade. This RHEL upgrade is being pushed due to RHEL 7's support expiring.  12 physical CPU cores, or 24 vCPU at 2 GHz or greater speed per core. 12 GB RAM. A 1 Gb Ethernet NIC, optional second NIC for a management network.   ... View more

I am ,    I am working alongside the unix team, as they have a better understanding of storage and resource requirements than I do. But this upgrade is long overdue.  Thanks for the assistance. I think I have enough for the required requests I am filling out.  ... View more

Yes it is. I wasn't around when these were spun up, but now that it's up to me to fix it, I want to make sure we wont run into these issues for a few years.  If we already have 10 Physical indexers that handle most of the data, would I need the 96 vCPU for the other 6 in Azure, I have to consider costs with this as well.    Thanks  ... View more

Hey,  Thanks for your response. With this azure cluster, we receive latency alerts all of the time. Comparing the installed and available resources to the recommended, we are under resourced. We have a clustered environment, two clusters at a different location for both the Search Heads as well as the Indexers. We have 10 Physical indexers, each with about 60TB of storage and 48 cores and 96 CPUs. Compare these indexers to the Azure ones, which have 4 Cores and 8 CPUs.   Our Azure cluster has been giving us latency warning for a few years now, even after a few upgrades. Now that I am more comfortable with our environment, I want to finally upgrade the cpu and memory to the recommended values.    We have 6 Azure Indexers, all of which have latency issues at some point or another. We have about 100 indexes, our top 3 sources ingest 600GB daily, and we average about 1.7 TB a day.    To sum up, these are under resourced, and they need more CPU.    Thanks ... View more

I am working on implementing this query, but I need to rename and standardize the output to the C_Label values so i can stats count on those. I need a count per source. I dont think i cram my eval statements with the OR statements. Ill try to incorporate this in my query   Thanks ... View more

Hey, thanks for your answer. After i posted this, i went to investigate the source of the data and any props or transforms set up for it.  I ran the following from our forwarder, the server that has the netskope TA app installed on it.   ./splunk btool props list --debug | grep "netskope:application"   I dont have any transforms with that tag.  Here is the output of the default netskope application inputs: [source::...netskope_file_hash_modalert.log*] SHOULD_LINEMERGE = true sourcetype = tanetskopeappforsplunk:log TZ = UTC [source::...netskope_url_modalert.log*] SHOULD_LINEMERGE = true sourcetype = tanetskopeappforsplunk:log TZ = UTC [source::...ta-netskopeappforsplunk*.log*] SHOULD_LINEMERGE = true sourcetype = tanetskopeappforsplunk:log TZ = UTC [source::...ta_netskopeappforsplunk*.log*] SHOULD_LINEMERGE = true sourcetype = tanetskopeappforsplunk:log TZ = UTC [netskope:event:v2] SHOULD_LINEMERGE = 0 category = Splunk App Add-on Builder pulldown_type = 1 [netskope:alert:v2] SHOULD_LINEMERGE = 0 category = Splunk App Add-on Builder pulldown_type = 1 [netskope:web_transaction] INDEXED_EXTRACTIONS = W3C TIME_FORMAT = %Y-%m-%d %H:%M:%S TZ = Etc/GMT SHOULD_LINEMERGE = 0 TRUNCATE = 999999 EXTRACT-from_source = .*[\\\/](?<input_name>.*)_(?<bucket_name>\d{8})_(?<bucket_file_name>.*) in source EVAL-vendor_product = "Netskope" FIELDALIAS-app = x_cs_app AS app FIELDALIAS-timestamp = _time as timestamp FIELDALIAS-bytes_in = cs_bytes AS bytes_in FIELDALIAS-bytes_out = sc_bytes AS bytes_out FIELDALIAS-category = x_category AS category FIELDALIAS-dest = s_ip AS dest EVAL-http_content_type = coalesce(cs_content_type, sc_content_type) FIELDALIAS-http_method = cs_method AS http_method FIELDALIAS-http_referrer = cs_referer AS http_referrer FIELDALIAS-http_user_agent = cs_user_agent AS http_user_agent FIELDALIAS-response_time = time_taken AS response_time FIELDALIAS-src=c_ip AS src FIELDALIAS-status = sc_status AS status FIELDALIAS-uri_path = cs_uri AS uri_path FIELDALIAS-uri_query = cs_uri_query AS uri_query FIELDALIAS-user = cs_username AS user EVAL-fullurl = cs_uri_scheme . "://" . cs_dns . cs_uri . if(isnull(cs_uri_query), "", "?") . coalesce(cs_uri_query,"") EVAL-x_c_browser=if(isnull(x_c_browser),"N/A",x_c_browser) EVAL-x_c_device=if(isnull(x_c_device),"N/A",x_c_device) FIELDALIAS-dest_port = cs_uri_port AS dest_port EVAL-url = cs_uri_scheme . "://" . cs_dns . cs_uri . if(isnull(cs_uri_query), "", "?") . coalesce(cs_uri_query,"") FIELDALIAS-duration = time_taken AS duration FIELDALIAS-http_referrer_domain = cs_referer AS http_referrer_domain EVAL-site = replace(cs_host, "^([^\.]+).*", "\1") [source::netskope_events_v2_connection] KV_MODE = json sourcetype = netskope:connection TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s SHOULD_LINEMERGE = false TRUNCATE = 999999 [source::...*events_iterator_page*.csv] INDEXED_EXTRACTIONS = CSV sourcetype = netskope:connection TIMESTAMP_FIELDS=timestamp TIME_FORMAT = %s SHOULD_LINEMERGE = false TRUNCATE = 999999 [netskope:connection] FIELDALIAS-src_ip = srcip AS src_ip FIELDALIAS-src=srcip AS src FIELDALIAS-dest_ip = dstip AS dest_ip FIELDALIAS-dest = dstip AS dest EVAL-dvc = coalesce(hostname, device) EVAL-app_session_key = app_session_id . "::" . host EVAL-vendor_product = "Netskope" FIELDALIAS-page_duration = page_duration AS duration FIELDALIAS-bytes = numbytes AS bytes FIELDALIAS-in_bytes = client_bytes AS bytes_in FIELDALIAS-category = appcategory AS category FIELDALIAS-out_bytes = server_bytes AS bytes_out FIELDALIAS-http_referrer = useragent AS http_user_agent EVAL-http_user_agent_length = len(useragent) FIELDALIAS-page = page AS url FIELDALIAS-src_location = src_location AS src_zone FIELDALIAS-dest_location = dst_location AS dest_zone EVAL-url_length = len(page) # from netskope:web EVAL-action = if(isnotnull(action),action,"isolate") FIELDALIAS-oc = object_type AS object_category FIELDALIAS-fu = from_user AS src_user [netskope:audit] SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s TRUNCATE = 999999 KV_MODE = json EVAL-vendor_product = "Netskope" # acl_modified, cleared, created, deleted, modified, read, stopped, updated EVAL-action = case(match(audit_log_event,"create|Create"),"created", match(audit_log_event,"granted"), "acl_modified", match(audit_log_event, "ack|Ack"), "cleared", match(audit_log_event, "delete|Delete"), "deleted", match(audit_log_event,"edit|Edit|Add"),"modified",match(audit_log_event,"Push|push|Reorder|update|Update"),"updated",match(audit_log_event,"Disable|disable"), "stopped",1=1,"unknown") EVAL-status = case(match(audit_log_event,"success|Success"),"success",match(audit_log_event,"fail|Fail"),"failure",1=1,"unknown") FIELDALIAS-severity_id = severity_level AS severity_id FIELDALIAS-data_type = supporting_data.data_type AS object FIELDALIAS-date_type_attr = supporting_data.data_values{} AS object_attrs FIELDALIAS-object_cat = category AS object_category FIELDALIAS-result = audit_log_event AS result [source::netskope_events_v2_application] KV_MODE = json TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s sourcetype = netskope:application SHOULD_LINEMERGE = false TRUNCATE = 999999 [source::...*events_iterator_application*.csv] INDEXED_EXTRACTIONS = CSV sourcetype = netskope:application TIMESTAMP_FIELDS=timestamp TIME_FORMAT = %s SHOULD_LINEMERGE = false TRUNCATE = 999999 [netskope:application] FIELDALIAS-src_ip = srcip AS src_ip FIELDALIAS-src=srcip AS src FIELDALIAS-dest_ip = dstip AS dest_ip FIELDALIAS-dest = dstip AS dest EVAL-dvc = coalesce(hostname, device) FIELDALIAS-src_location = src_location AS src_zone FIELDALIAS-dest_location = dst_location AS dest_zone FIELDALIAS-signature = policy AS signature EVAL-file_hash = coalesce(local_sha256, local_md5) FIELDALIAS-file_name = filename AS file_name EVAL-app_session_key = app_session_id . "::" . host EVAL-vendor_product = "Netskope" FIELDALIAS-oc = object_type AS object_category FIELDALIAS-fu = from_user AS src_user [source::netskope_events_v2_network] KV_MODE = json TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s sourcetype = netskope:network SHOULD_LINEMERGE = false TRUNCATE = 999999 [source::...*events_iterator_network*.csv] INDEXED_EXTRACTIONS = CSV sourcetype = netskope:network TIMESTAMP_FIELDS=timestamp TIME_FORMAT = %s SHOULD_LINEMERGE = false TRUNCATE = 999999 [netskope:network] FIELDALIAS-src_ip = srcip AS src_ip FIELDALIAS-src=srcip AS src FIELDALIAS-dest_ip = dstip AS dest_ip FIELDALIAS-dest = dstip AS dest EVAL-dvc = coalesce(hostname, device) EVAL-vendor_product = "Netskope" FIELDALIAS-bytes = numbytes AS bytes FIELDALIAS-in_bytes = client_bytes AS bytes_in FIELDALIAS-out_bytes = server_bytes AS bytes_out FIELDALIAS-packets_in = client_packets AS packets_in FIELDALIAS-packets_out = server_packets AS packets_out FIELDALIAS-src_port = srcport AS src_port FIELDALIAS-dest_port = dstport AS dest_port FIELDALIAS-session_id = network_session_id AS session_id FIELDALIAS-duration = session_duration AS duration [netskope:incident] SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s TRUNCATE = 999999 KV_MODE = json FIELDALIAS-signature_id = internal_id AS signature_id FIELDALIAS-action = dlp_match_info{}.dlp_action AS action FIELDALIAS-object_path = url AS object_path FIELDALIAS-object_category = true_obj_category AS object_category FIELDALIAS-signature = title AS signature FIELDALIAS-src=src_location AS src FIELDALIAS-src_user = from_user AS src_user FIELDALIAS-dest = dst_location AS dest # FIELDALIAS-user = to_user AS user EVAL-user = coalesce(user, to_user) EVAL-vendor_product = "Netskope" [source::netskope_alerts_v2] KV_MODE = json TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s sourcetype = netskope:alert SHOULD_LINEMERGE = false TRUNCATE = 999999 [source::...*alerts_iterator*.csv] INDEXED_EXTRACTIONS = CSV SHOULD_LINEMERGE = false TIMESTAMP_FIELDS=timestamp TIME_FORMAT = %s sourcetype = netskope:alert TRUNCATE = 999999 [netskope:alert] EVAL-dvc = coalesce(hostname, device) EVAL-vendor_product = "Netskope" EVAL-severity_id = coalesce(severity_id, severity_level_id) EVAL-severity = coalesce(severity_level, dlp_rule_severity, dlp_severity, mal_sev, malware_severity, severity, severity_level) EVAL-object_path = if(file_path="NA", object, coalesce(file_path, object)) FIELDALIAS-id = internal_id AS id FIELDALIAS-srcip = srcip AS src FIELDALIAS-dstip = dstip AS dest EVAL-file_hash = coalesce(local_sha256, local_md5) FIELDALIAS-signature = alert_name AS signature FIELDALIAS-oc = object_type AS object_category FIELDALIAS-fu = from_user AS src_user FIELDALIAS-src_location = src_location AS src_zone FIELDALIAS-dest_location = dst_location AS dest_zone FIELDALIAS-file_name = filename AS file_name [netskope:infrastructure] SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s TRUNCATE = 999999 KV_MODE = json FIELDALIAS-device = device_name AS device EVAL-app = "Netskope" EVAL-vendor_product = "Netskope" [netskope:endpoint] SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s TRUNCATE = 999999 KV_MODE = json EVAL-vendor_product = "Netskope" [netskope:clients] KV_MODE = json FIELDALIAS-make = attributes.host_info.device_make AS make FIELDALIAS-model = attributes.host_info.device_model AS model FIELDALIAS-os = attributes.host_info.os AS os FIELDALIAS-ver = attributes.host_info.os_version AS version FIELDALIAS-name = attributes.host_info.hostname AS dest FIELDALIAS-user = attributes.users{}.username AS user EVAL-vendor_product = "Netskope" SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 35 TIME_FORMAT = %s TRUNCATE = 999999 [netskope:api] KV_MODE = json EVAL-vendor_product = "Netskope" [netskope:alertaction:file_hash] FIELDALIAS-action_status = status AS action_status FIELDALIAS-action_name = orig_action_name AS action_name [netskope:alertaction:url] FIELDALIAS-action_status = status AS action_status FIELDALIAS-action_name = orig_action_name AS action_name # For proper ingestion of Alert action events used in Splunk ES App [source::...stash_common_action_model] sourcetype=stash_common_action_model [stash_common_action_model] TRUNCATE = 0 # only look for ***SPLUNK*** on the first line HEADER_MODE = firstline # we can summary index past data, but rarely future data MAX_DAYS_HENCE = 2 MAX_DAYS_AGO = 10000 # 5 years difference between two events MAX_DIFF_SECS_AGO = 155520000 MAX_DIFF_SECS_HENCE = 155520000 TIME_PREFIX = (?m)^\*{3}Common\sAction\sModel\*{3}.*$ MAX_TIMESTAMP_LOOKAHEAD = 25 LEARN_MODEL = false # break .stash_new custom format into events SHOULD_LINEMERGE = false BREAK_ONLY_BEFORE_DATE = false LINE_BREAKER = (\r?\n==##~~##~~ 1E8N3D4E6V5E7N2T9 ~~##~~##==\r?\n) TRANSFORMS-0parse_cam_header = orig_action_name_for_stash_cam,orig_sid_for_stash_cam,orig_rid_for_stash_cam,sourcetype_for_stash_cam TRANSFORMS-1sinkhole_cam_header = sinkhole_cam_header     Looking and running your suggested command (Good command btw), i get the following output: I don't see any evidence of us modifying or creating a dlp_rule value. I had specifically mapped the dlp_rule to these values below: These are the values I was seeing. I was using this mapping and values in every other query as well, so i must have seen them.  This is the default netskope app. I also looked at any possible sourcetypes or transforms via the gui, and I didn't see any. I am working on this data with a coworker that has insight into the Netskope portal, and he said that the dlp_role field is blank there as well. If the data had changed, the old data shouldn't have changed. I haven't updated the netskope app.    There are too many fields to paste in here for the logs themselves, but here are the fields we are looking at: dlp_fail_reason: dlp_file: dlp_incident_id: 0 dlp_is_unique_count: dlp_mail_parent_id: dlp_parent_id: 0 dlp_profile: dlp_rule: dlp_rule_count: 0 dlp_rule_severity: dlp_scan_failed: dlp_unique_count: 0 dst_country: US dst_geoip_src: 0 dst_latitude: 7.40594 dst_location: Mow dst_longitude: -1.1551 dst_region: C dst_timezone: America/ dst_zipcode: N/A dsthost: dstip: 1.5.5.5 dstport: 455   With this specific dashboard and use case, I am searching for all time. And the field in general is blank. We only get 3 dlp_rule values, and the rest, 99% are blank.  Not sure how to track down if the data set changed due to me searching for all time right now.    Thanks for any guidance  ... View more

Ingest actions are exactly what I am looking for. While not as intuitive as an entire tool dedicated to modifying data, I think this would do the trick, as long as i can trim out field values before forwarding them to an indexer for ingestion.    I am looking for docs explaining how to do just that, but I am struggling to find step by step instructions. Can you send me some good docs that show how to use expressions to do what I want.    Thank you. This may be my ticket to saving us hundreds of thousands of dollars in licensing costs.  ... View more

Thanks for the response. I was able to get everything sorted. We are trying to reduce our license, so if we can trim up data and remove unwanted fields and then ingest it, that would be ideal. Where in the pipeline does data count towards the splunk license? Can we apply props.conf and transforms.conf to modify and trim the data? If i wanted to remove 5 fields from a log being ingested, would the above approach apply? And if so, if I trim it up before ingesting, would that save on our license?    Thanks ... View more

This is exactly what I was looking for. Really nice doc linked. Thanks ... View more

I signed in just to say I had this exact problem, and your question was exactly what I was looking for. Thank you. This forum post helped answer my issue.  jowenssi Reply was what I was looking for.      ... View more

That makes sense. I was able to find some errors in Splunk _internal index   DO i just need to salt every file? How would i re-ingest those, or why are those not ingesting, but the other ones are?    ... View more

For a pickle, that was a very fast response.  But running those commands looks like it outputs the internal logs. All of the logs monitored at /export/opt/splunk.  Doesn't really show anything other than those directories.    ... View more

Just to follow up with what my problem was, I had a license set for an individual instance. I thought distributed meant multiple instances of each type of Splunk Server, ie, multiple indexers, SH, forwarders, etc. I didnt realize one SH, one Indexer, and one Forwarder counted as a distributed. Either way, putting the 10 GB/day distributed license did the trick.   Now dev works 🙂  ... View more