I am working on implementing this query, but I need to rename and standardize the output to the C_Label values so i can stats count on those. I need a count per source. I dont think i cram my eval statements with the OR statements. Ill try to incorporate this in my query   Thanks ... View more

Hey, thanks for your answer. After i posted this, i went to investigate the source of the data and any props or transforms set up for it.  I ran the following from our forwarder, the server that has the netskope TA app installed on it.   ./splunk btool props list --debug | grep "netskope:application"   I dont have any transforms with that tag.  Here is the output of the default netskope application inputs: [source::...netskope_file_hash_modalert.log*] SHOULD_LINEMERGE = true sourcetype = tanetskopeappforsplunk:log TZ = UTC [source::...netskope_url_modalert.log*] SHOULD_LINEMERGE = true sourcetype = tanetskopeappforsplunk:log TZ = UTC [source::...ta-netskopeappforsplunk*.log*] SHOULD_LINEMERGE = true sourcetype = tanetskopeappforsplunk:log TZ = UTC [source::...ta_netskopeappforsplunk*.log*] SHOULD_LINEMERGE = true sourcetype = tanetskopeappforsplunk:log TZ = UTC [netskope:event:v2] SHOULD_LINEMERGE = 0 category = Splunk App Add-on Builder pulldown_type = 1 [netskope:alert:v2] SHOULD_LINEMERGE = 0 category = Splunk App Add-on Builder pulldown_type = 1 [netskope:web_transaction] INDEXED_EXTRACTIONS = W3C TIME_FORMAT = %Y-%m-%d %H:%M:%S TZ = Etc/GMT SHOULD_LINEMERGE = 0 TRUNCATE = 999999 EXTRACT-from_source = .*[\\\/](?<input_name>.*)_(?<bucket_name>\d{8})_(?<bucket_file_name>.*) in source EVAL-vendor_product = "Netskope" FIELDALIAS-app = x_cs_app AS app FIELDALIAS-timestamp = _time as timestamp FIELDALIAS-bytes_in = cs_bytes AS bytes_in FIELDALIAS-bytes_out = sc_bytes AS bytes_out FIELDALIAS-category = x_category AS category FIELDALIAS-dest = s_ip AS dest EVAL-http_content_type = coalesce(cs_content_type, sc_content_type) FIELDALIAS-http_method = cs_method AS http_method FIELDALIAS-http_referrer = cs_referer AS http_referrer FIELDALIAS-http_user_agent = cs_user_agent AS http_user_agent FIELDALIAS-response_time = time_taken AS response_time FIELDALIAS-src=c_ip AS src FIELDALIAS-status = sc_status AS status FIELDALIAS-uri_path = cs_uri AS uri_path FIELDALIAS-uri_query = cs_uri_query AS uri_query FIELDALIAS-user = cs_username AS user EVAL-fullurl = cs_uri_scheme . "://" . cs_dns . cs_uri . if(isnull(cs_uri_query), "", "?") . coalesce(cs_uri_query,"") EVAL-x_c_browser=if(isnull(x_c_browser),"N/A",x_c_browser) EVAL-x_c_device=if(isnull(x_c_device),"N/A",x_c_device) FIELDALIAS-dest_port = cs_uri_port AS dest_port EVAL-url = cs_uri_scheme . "://" . cs_dns . cs_uri . if(isnull(cs_uri_query), "", "?") . coalesce(cs_uri_query,"") FIELDALIAS-duration = time_taken AS duration FIELDALIAS-http_referrer_domain = cs_referer AS http_referrer_domain EVAL-site = replace(cs_host, "^([^\.]+).*", "\1") [source::netskope_events_v2_connection] KV_MODE = json sourcetype = netskope:connection TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s SHOULD_LINEMERGE = false TRUNCATE = 999999 [source::...*events_iterator_page*.csv] INDEXED_EXTRACTIONS = CSV sourcetype = netskope:connection TIMESTAMP_FIELDS=timestamp TIME_FORMAT = %s SHOULD_LINEMERGE = false TRUNCATE = 999999 [netskope:connection] FIELDALIAS-src_ip = srcip AS src_ip FIELDALIAS-src=srcip AS src FIELDALIAS-dest_ip = dstip AS dest_ip FIELDALIAS-dest = dstip AS dest EVAL-dvc = coalesce(hostname, device) EVAL-app_session_key = app_session_id . "::" . host EVAL-vendor_product = "Netskope" FIELDALIAS-page_duration = page_duration AS duration FIELDALIAS-bytes = numbytes AS bytes FIELDALIAS-in_bytes = client_bytes AS bytes_in FIELDALIAS-category = appcategory AS category FIELDALIAS-out_bytes = server_bytes AS bytes_out FIELDALIAS-http_referrer = useragent AS http_user_agent EVAL-http_user_agent_length = len(useragent) FIELDALIAS-page = page AS url FIELDALIAS-src_location = src_location AS src_zone FIELDALIAS-dest_location = dst_location AS dest_zone EVAL-url_length = len(page) # from netskope:web EVAL-action = if(isnotnull(action),action,"isolate") FIELDALIAS-oc = object_type AS object_category FIELDALIAS-fu = from_user AS src_user [netskope:audit] SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s TRUNCATE = 999999 KV_MODE = json EVAL-vendor_product = "Netskope" # acl_modified, cleared, created, deleted, modified, read, stopped, updated EVAL-action = case(match(audit_log_event,"create|Create"),"created", match(audit_log_event,"granted"), "acl_modified", match(audit_log_event, "ack|Ack"), "cleared", match(audit_log_event, "delete|Delete"), "deleted", match(audit_log_event,"edit|Edit|Add"),"modified",match(audit_log_event,"Push|push|Reorder|update|Update"),"updated",match(audit_log_event,"Disable|disable"), "stopped",1=1,"unknown") EVAL-status = case(match(audit_log_event,"success|Success"),"success",match(audit_log_event,"fail|Fail"),"failure",1=1,"unknown") FIELDALIAS-severity_id = severity_level AS severity_id FIELDALIAS-data_type = supporting_data.data_type AS object FIELDALIAS-date_type_attr = supporting_data.data_values{} AS object_attrs FIELDALIAS-object_cat = category AS object_category FIELDALIAS-result = audit_log_event AS result [source::netskope_events_v2_application] KV_MODE = json TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s sourcetype = netskope:application SHOULD_LINEMERGE = false TRUNCATE = 999999 [source::...*events_iterator_application*.csv] INDEXED_EXTRACTIONS = CSV sourcetype = netskope:application TIMESTAMP_FIELDS=timestamp TIME_FORMAT = %s SHOULD_LINEMERGE = false TRUNCATE = 999999 [netskope:application] FIELDALIAS-src_ip = srcip AS src_ip FIELDALIAS-src=srcip AS src FIELDALIAS-dest_ip = dstip AS dest_ip FIELDALIAS-dest = dstip AS dest EVAL-dvc = coalesce(hostname, device) FIELDALIAS-src_location = src_location AS src_zone FIELDALIAS-dest_location = dst_location AS dest_zone FIELDALIAS-signature = policy AS signature EVAL-file_hash = coalesce(local_sha256, local_md5) FIELDALIAS-file_name = filename AS file_name EVAL-app_session_key = app_session_id . "::" . host EVAL-vendor_product = "Netskope" FIELDALIAS-oc = object_type AS object_category FIELDALIAS-fu = from_user AS src_user [source::netskope_events_v2_network] KV_MODE = json TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s sourcetype = netskope:network SHOULD_LINEMERGE = false TRUNCATE = 999999 [source::...*events_iterator_network*.csv] INDEXED_EXTRACTIONS = CSV sourcetype = netskope:network TIMESTAMP_FIELDS=timestamp TIME_FORMAT = %s SHOULD_LINEMERGE = false TRUNCATE = 999999 [netskope:network] FIELDALIAS-src_ip = srcip AS src_ip FIELDALIAS-src=srcip AS src FIELDALIAS-dest_ip = dstip AS dest_ip FIELDALIAS-dest = dstip AS dest EVAL-dvc = coalesce(hostname, device) EVAL-vendor_product = "Netskope" FIELDALIAS-bytes = numbytes AS bytes FIELDALIAS-in_bytes = client_bytes AS bytes_in FIELDALIAS-out_bytes = server_bytes AS bytes_out FIELDALIAS-packets_in = client_packets AS packets_in FIELDALIAS-packets_out = server_packets AS packets_out FIELDALIAS-src_port = srcport AS src_port FIELDALIAS-dest_port = dstport AS dest_port FIELDALIAS-session_id = network_session_id AS session_id FIELDALIAS-duration = session_duration AS duration [netskope:incident] SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s TRUNCATE = 999999 KV_MODE = json FIELDALIAS-signature_id = internal_id AS signature_id FIELDALIAS-action = dlp_match_info{}.dlp_action AS action FIELDALIAS-object_path = url AS object_path FIELDALIAS-object_category = true_obj_category AS object_category FIELDALIAS-signature = title AS signature FIELDALIAS-src=src_location AS src FIELDALIAS-src_user = from_user AS src_user FIELDALIAS-dest = dst_location AS dest # FIELDALIAS-user = to_user AS user EVAL-user = coalesce(user, to_user) EVAL-vendor_product = "Netskope" [source::netskope_alerts_v2] KV_MODE = json TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s sourcetype = netskope:alert SHOULD_LINEMERGE = false TRUNCATE = 999999 [source::...*alerts_iterator*.csv] INDEXED_EXTRACTIONS = CSV SHOULD_LINEMERGE = false TIMESTAMP_FIELDS=timestamp TIME_FORMAT = %s sourcetype = netskope:alert TRUNCATE = 999999 [netskope:alert] EVAL-dvc = coalesce(hostname, device) EVAL-vendor_product = "Netskope" EVAL-severity_id = coalesce(severity_id, severity_level_id) EVAL-severity = coalesce(severity_level, dlp_rule_severity, dlp_severity, mal_sev, malware_severity, severity, severity_level) EVAL-object_path = if(file_path="NA", object, coalesce(file_path, object)) FIELDALIAS-id = internal_id AS id FIELDALIAS-srcip = srcip AS src FIELDALIAS-dstip = dstip AS dest EVAL-file_hash = coalesce(local_sha256, local_md5) FIELDALIAS-signature = alert_name AS signature FIELDALIAS-oc = object_type AS object_category FIELDALIAS-fu = from_user AS src_user FIELDALIAS-src_location = src_location AS src_zone FIELDALIAS-dest_location = dst_location AS dest_zone FIELDALIAS-file_name = filename AS file_name [netskope:infrastructure] SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s TRUNCATE = 999999 KV_MODE = json FIELDALIAS-device = device_name AS device EVAL-app = "Netskope" EVAL-vendor_product = "Netskope" [netskope:endpoint] SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %s TRUNCATE = 999999 KV_MODE = json EVAL-vendor_product = "Netskope" [netskope:clients] KV_MODE = json FIELDALIAS-make = attributes.host_info.device_make AS make FIELDALIAS-model = attributes.host_info.device_model AS model FIELDALIAS-os = attributes.host_info.os AS os FIELDALIAS-ver = attributes.host_info.os_version AS version FIELDALIAS-name = attributes.host_info.hostname AS dest FIELDALIAS-user = attributes.users{}.username AS user EVAL-vendor_product = "Netskope" SHOULD_LINEMERGE = false TIME_PREFIX = "timestamp": MAX_TIMESTAMP_LOOKAHEAD = 35 TIME_FORMAT = %s TRUNCATE = 999999 [netskope:api] KV_MODE = json EVAL-vendor_product = "Netskope" [netskope:alertaction:file_hash] FIELDALIAS-action_status = status AS action_status FIELDALIAS-action_name = orig_action_name AS action_name [netskope:alertaction:url] FIELDALIAS-action_status = status AS action_status FIELDALIAS-action_name = orig_action_name AS action_name # For proper ingestion of Alert action events used in Splunk ES App [source::...stash_common_action_model] sourcetype=stash_common_action_model [stash_common_action_model] TRUNCATE = 0 # only look for ***SPLUNK*** on the first line HEADER_MODE = firstline # we can summary index past data, but rarely future data MAX_DAYS_HENCE = 2 MAX_DAYS_AGO = 10000 # 5 years difference between two events MAX_DIFF_SECS_AGO = 155520000 MAX_DIFF_SECS_HENCE = 155520000 TIME_PREFIX = (?m)^\*{3}Common\sAction\sModel\*{3}.*$ MAX_TIMESTAMP_LOOKAHEAD = 25 LEARN_MODEL = false # break .stash_new custom format into events SHOULD_LINEMERGE = false BREAK_ONLY_BEFORE_DATE = false LINE_BREAKER = (\r?\n==##~~##~~ 1E8N3D4E6V5E7N2T9 ~~##~~##==\r?\n) TRANSFORMS-0parse_cam_header = orig_action_name_for_stash_cam,orig_sid_for_stash_cam,orig_rid_for_stash_cam,sourcetype_for_stash_cam TRANSFORMS-1sinkhole_cam_header = sinkhole_cam_header     Looking and running your suggested command (Good command btw), i get the following output: I don't see any evidence of us modifying or creating a dlp_rule value. I had specifically mapped the dlp_rule to these values below: These are the values I was seeing. I was using this mapping and values in every other query as well, so i must have seen them.  This is the default netskope app. I also looked at any possible sourcetypes or transforms via the gui, and I didn't see any. I am working on this data with a coworker that has insight into the Netskope portal, and he said that the dlp_role field is blank there as well. If the data had changed, the old data shouldn't have changed. I haven't updated the netskope app.    There are too many fields to paste in here for the logs themselves, but here are the fields we are looking at: dlp_fail_reason: dlp_file: dlp_incident_id: 0 dlp_is_unique_count: dlp_mail_parent_id: dlp_parent_id: 0 dlp_profile: dlp_rule: dlp_rule_count: 0 dlp_rule_severity: dlp_scan_failed: dlp_unique_count: 0 dst_country: US dst_geoip_src: 0 dst_latitude: 7.40594 dst_location: Mow dst_longitude: -1.1551 dst_region: C dst_timezone: America/ dst_zipcode: N/A dsthost: dstip: 1.5.5.5 dstport: 455   With this specific dashboard and use case, I am searching for all time. And the field in general is blank. We only get 3 dlp_rule values, and the rest, 99% are blank.  Not sure how to track down if the data set changed due to me searching for all time right now.    Thanks for any guidance  ... View more

Ingest actions are exactly what I am looking for. While not as intuitive as an entire tool dedicated to modifying data, I think this would do the trick, as long as i can trim out field values before forwarding them to an indexer for ingestion.    I am looking for docs explaining how to do just that, but I am struggling to find step by step instructions. Can you send me some good docs that show how to use expressions to do what I want.    Thank you. This may be my ticket to saving us hundreds of thousands of dollars in licensing costs.  ... View more

Thanks for the response. I was able to get everything sorted. We are trying to reduce our license, so if we can trim up data and remove unwanted fields and then ingest it, that would be ideal. Where in the pipeline does data count towards the splunk license? Can we apply props.conf and transforms.conf to modify and trim the data? If i wanted to remove 5 fields from a log being ingested, would the above approach apply? And if so, if I trim it up before ingesting, would that save on our license?    Thanks ... View more

This is exactly what I was looking for. Really nice doc linked. Thanks ... View more

I signed in just to say I had this exact problem, and your question was exactly what I was looking for. Thank you. This forum post helped answer my issue.  jowenssi Reply was what I was looking for.      ... View more

That makes sense. I was able to find some errors in Splunk _internal index   DO i just need to salt every file? How would i re-ingest those, or why are those not ingesting, but the other ones are?    ... View more

For a pickle, that was a very fast response.  But running those commands looks like it outputs the internal logs. All of the logs monitored at /export/opt/splunk.  Doesn't really show anything other than those directories.    ... View more

Just to follow up with what my problem was, I had a license set for an individual instance. I thought distributed meant multiple instances of each type of Splunk Server, ie, multiple indexers, SH, forwarders, etc. I didnt realize one SH, one Indexer, and one Forwarder counted as a distributed. Either way, putting the 10 GB/day distributed license did the trick.   Now dev works 🙂  ... View more

I was testing out a lot of different things. I know I def did edit the distsearch manually. I did most everything from the CLI. Redoing and moving the License Manager through the GUI fixed some of the issues, as i can now search the data.    Thanks ... View more

After a bit of work, I made the indexer the License Master. I already wanted the SH to also server as the DMC, and im not sure what was happening, but i made the indexer the License master, confirmed a few settings, and I was able to add a new search peer. That window now allows me to see search peers under Distributed peers:   Not entirely sure what the problem was.  But in this instance, I am trying to get one indexer, one SH, and one forwarder working. i made the indexer the License master, the forwarder just a forwarder, and hopefully the SH as a SH and DMC.   Thanks for the help.  ... View more