I've been wanting to build some integrity checking and other functionality based on knowing the fields in a sourcetype for a while now. At my company we've built a data dictionary of indexes and sourcetype of interest to the SOC. They can search the dictionary to help them remember the important data sources. I'd like to augment/use this info in a couple of new ways: 1) give them a field list for all of these sourcetypes so they could search for which sourcetypes have a relevant field (like src_ip) 2) I'd like to note the fields that appear in 100% of records for a sourcetype and then every day find out if is missing any of those fields. This would quickly clue me into data issues related to the event sent, parsing, or knowledge objects. I know how to get a list of fields for 1 sourcetype and store that info. And I know how to compare a sourcetype to a past set of fields to a current set. My challenge now is how do I get the list of fields for the 100 sourcetypes of interest so far my best idea is to create 100 jobs to handle each sourcetype. Something like ```1-get the sourcetypes of interest and pull back data for them``` [| inputlookup dataDictionary.csv where imf_critical=true | eval yesterday=relative_time(now(),"-1d@d") | where evalTS>yesterday | dedup sourcetype | sort sourcetype | head 5 | tail 1 | table sourcetype] earliest=-2d@d latest=-1@d ```2-get samples for all indexes in which the sourcetype appears``` | dedup 10 index sourcetype | fieldsummary ```3-determine field coverage so we can pick the hallmark fields``` | eventstats max(count) as maxCount | eval pctCov=round(count/maxCount,2)*100 | table field pctCov ```4-add back in the sourcetype name``` | append [| inputlookup dataDictionary.csv where imf_critical=true | eval yesterday=relative_time(now(),"-1d@d") | where evalTS>yesterday | dedup sourcetype | sort sourcetype | head 5 | tail 1 | table sourcetype] | eventstats first(sourcetype) as sourcetype | eval evalTS=now() | table sourcetype evalTS field pctCov ```5-collect the fields to a summary index daily``` | collect index=soc_summary marker="sumType=dataInfo, sumSubtype=stFields" If I ran 100 jobs like this, the number after head would increment to give me the next sourcetype. But I feel like there has to be a better way to do fieldsummary on a lot of sourcetypes. Any ideas?

Hi Splunkers ! Is it a way to automatically retrieve Entity information like OS, IP address, OS version, ... and add it as Dimensions ? All my Entities are retrieved via Splunk Addon For *nix and via Splunk Addon For Windows. All my Entities are correctly imported to ITEW but no one has other information like his IP or his OS in Entities Info Fields. I have about 1200 entities so, I'm looking for a way to add those information to all my entities automatically. All needed data are correctly indexed. My save search ITSI Import Objects - TA *Nix and ITSI Import Objects - Perfmon get those info correctly. Can somebody help me with these issue ? Happy Splunking !

I am trying to use Splunk Dashboard Studio, I have a search for a single value viz: | makeresults | eval Date=strftime(now(),"%Y-%m-%d %H:%M:%S") | table Date | rename Date AS UTC-DateTime The single value viz always returns the "time" in this format "2022-12-02T20:39:21", ignoring the format strftime in my search. I can apply a format to a table column no problem. How can I format the value in the single value viz as "2022-12-02 20:39:21" and... how can I modify or refresh the query that gets the time every second? I saw a youtube tutorial on this, but the author did not explain the query or the process to refresh or how to apply a different format to the value. Please advise, thanks, eholz1

We are looking to see the size of all the fields in a particular index. We have come up with this search to see the size of a particular field but we would like to see the size of all the fields in the index in order to understand where the bulk of the data is sitting. index=index_name | eval raw_len=(len(_raw)/1024/1024/1024) | stats sum(raw_len) as GB by field_name | sort -GB

Dear all, I have the use case that my splunk universal forwarder does not continuously monitor my logs. Because of this nature, I am using batch mode to have the files deleted after ingestion. Now, I occasionally receive log files which I have already received at an earlier point in time. Problem is: The features crcSalt, initCrcLength etc. are only available in monitor mode. This means that I am not able to benefit from splunks features to prevent duplicate ingestion of the same data. Any help on a solution for this is greatly appreciated.

I have two Splunk Enterprise environments, both at 9.0.2. For users in one environment, search history goes back only two days. For users in the other environment, search history goes back more than 8 months. Any clue about what could cause that? Both environments are using a single search head. Users are set up the same in each environment. The limits.conf on both search heads is identical. I verified that the user's search history .csv file goes back two days on one and 8 months on the other.

Hello Splunkers!! We have a dashboard which works on the loadjob. When users try accessing the dashboard, they are getting "No results found" message. First I thought problem with permissions, but out of 4 colleagues with same admin access as mine, 3 members are able to see the dashboard results. So it seems it is not problem with permissions. To figure out the problem in query, we back traced the logic line by line and found the line from where user is not getting 0 results. Search Query: |loadjob reportname .....some evals & lookups.... |eval valid=if(match(backlog_dates,e_time),"yes","no") | search valid=yes --->no results from this line replaced 'match' with 'like' but still no results tried the below line but same issue. | where backlog_dates like e_time Checked the logs for both users who are able to get results and who are not able to get results. But nothing to suspect and no errors in log. It is very strange that it is working for some users. Please help me on figuring out the issue. Below is the sample data

Hi I am sending windows system and security data to splunk cloud. Data is collected using UF and forwarded to cloud through HF. I want to get rid of extra text in windows data (example:4624). I saw SED command stanzas are there in the documentation. I tried to place them in the sourcetype on splunk cloud, but it is not working. But same is working for my onprem indexer. Not sure what is wrong. Any suggestions would be appreciated.

index="*dockerlogs*" source="*gps-request-processor-test*" OR source="*gps-external-processor-test*" OR source="*gps-artifact-processor-test*" event="*Request" | eval LabelType=coalesce(labelType, documentType) | eval event = case (like(event,"%Sync%"),"Sync",like(event,"%Async%"),"Async") | stats count(eval(status="Received")) as received count(eval(status="Failed")) as failed by sourceNodeCode geoCode LabelType event where as the source : - is my application name event :- Type of request whether synchronous request or Asynchronous request labeltype : - Different type of label sourcenodecode and geocode :- is the shopcode and shopregion from where the label is requested received - no of label request received failed - no of label request failed Now i want to find the received and failed request count based on sourceNodeCode, geoCode, LabelType, event But for failed request count i want to add condition - in case of synchronous request or event the failed count should fetch from '*gps-request-processor-test*' application in case of asynchronous request or event the failed count should fetch from "*gps-external-processor-test*" OR "*gps-artifact-processor-test*" application The output should look something similar to this attached o/p.