There are lots of unknowns about your requirement such as data format or size, but here are some suggestions that may help. They are based on the assumption that the file is small and you can read the entire file contents into Splunk on a regular basis. Once the data is in Splunk, you can use standard tools to compare lines, but it would only really be a good use case for Splunk if the data structure is simple and changes easy to identify. Firstly the simplest option for a small file, is set up a script to read the contents on a regular basis - for example every hour, then once you have more than one copy of the file in Splunk, you can run a query to compare both versions. This will tell you what is different, but not what time it changed (only between the two collections) . Another script could just run a dir / ls command to show the file timestamp and collect that data if required. If you want to detect the exact time the file content changed and trigger something at that specific time, initCrcLength & crcSalt may help to read the contents on change, but it can be hit and miss depending on the type of change. One other option may be to use indexed_extrations. if the file is a structured format (xml / json / csv etc) - you can  monitor the whole file and use the INDEXED_EXTRACTIONS and "CHECK_METHOD = modtime" options in a props.conf file on the collecting system. Every time the file mod time changes, it will re-read the whole file You now potentially have two copies in Splunk that are indexed with the timestamp of the file change (if the contents have timestamps, you may need to disable this on indexing). In addtion, you would want to separate each line of the file into a separate event in Splunk. You then search for all events from both versions of the file, you tag events from each version so you can determine which version it was from - For example we create a variable called "version" and set this to "old" if it is from the previous version and set the value to "new" if from the latest version. Then you merge all the events (i.e. lines from the file) into one table using stats and include all the values of "version" for each line. If the value of "version" for a line contains both "old" and "new", then you know that line is present in both versions and so has not changed. If it only contains "old" the line has been removed and if it only contains "new", you know it has been added. For "who" changed the file - you need to look at OS level auditing for that level of detail.   ... View more

By default the metrics will only provide details on the top 10 items in a series - this applies to almost all the metrics that are collected. https://docs.splunk.com/Documentation/Splunk/8.1.3/Troubleshooting/Aboutmetricslog The way to address this, is to change the limit applied using the limits.conf file - the following stanza added to a indexer / HF will increase the number of items which are tracked [metrics] maxseries = 50 interval = 60 This will ensure the top 50 items are included - so you can adjust this to reflect the likely number of sourcetypes if that is the main metric to be tracked The "interval" reference allows you to change the frequency that this data is collected to control the volume of data generated in the log. In the same, the frequency is 60seconds to allow minute by minute tracking - but that may be excessive for some purposes. ... View more

It appears that Splunk detects the "_time" fields and makes a decision that this should be first - I assume as _time is the only true law in the universe. So if you rename the field to a custom name like "timefield" - is it no longer considered a special case and will now follow the order specified by the fields or table command. Another consideration is that when a field is called "_time" and contains an epoch time value - Splunk will automatically convert it to a readable format. However when you rename to something else, Splunk will just show the epoch value. So you also need to add a eval with strftime to convert the value to your preferred readable time. ... View more

Did you manage to fix the issue? or is the issue still occuring with the ipv6 events? ... View more

I would not rule out a regex issue. I am not saying there is a problem with the props or transforms, but the source data- we have seen issues with firewall logs where local configurations on each firewall meant the structure of each event was slightly different from different devices. I am also assuming that you have checked the full list of fields that are extracted - i.e many fields will be hidden if Splunk thinks they do not contain useful information. I just want to cover everything and without additional information, I have no way of knowing how familiar you are with Splunk. You also mention that you have multi searchheads - and an index cluster. When searching, the field extractions on each indexer are controlled by the search head configuration, but this has to be replicated to each indexer through the knowlegde bundle (https://docs.splunk.com/Documentation/Splunk/8.0.3/DistSearch/Knowledgebundlereplication). I have seen issues with problems on individual indexers where replication is delayed or corrupted and this is not immediately obvious when you get your search results. ... View more

Is this on a single indexer? or through multiple indexers? It is also not clear if you are referring to index time field extraction or search time - each will have completely different root causes and ways to diagnose the issue. In most cases this can be down to even a single character or extra space in the raw event which makes the regular expression not work. Maybe some examples of events that work and some that do not work may help. For example if occurring at specific times in the day, maybe the timestamp in the raw event changes from single digit hour to double digit hour ... View more

I would check the version of snmp_ta you are using and the version of Splunk you are trying to get working on. I suspect that you need specific versions of each to work together. There are also some other answers which suggest some possible issues in Splunk v8 - https://answers.splunk.com/answers/144720/snmp-input-type-not-displaying-in-splunk-web.html Please also note that the SNMP modular input (I presume you are using this - https://splunkbase.splunk.com/app/1537) now requires an activation key and so may not work otherwise. Personally I could not get the SNMP modular input to work reliably - it would often fail on complex data and getting the MIB files converted to work is a big pain. Depending on the type of device you are collecting from, MIB's are essential to get any sort of readable data. My preferred approach is to use http://www.net-snmp.org/. Just create a standard batch/shell script using snmpwalk to collect the data. Not only can you easily test outside of Splunk, but it allows you to collect large number of different targets by just expanding your script. You then create a scripted input in Splunk to run the script on your required frequency. If can get a script working outside of Splunk, it is not too difficult to then get it working inside Splunk. if you want to stick with the modular input, here is some info - https://www.splunk.com/en_us/blog/tips-and-tricks/making-snmp-simpler.html These two articles are also very helpful, especially around transforming SNMP results when they are received which is useful whether you are using the modular input or another script. https://www.splunk.com/en_us/blog/tips-and-tricks/adventures-with-snmp-and-cisco-nexus-pt1.html https://www.splunk.com/en_us/blog/tips-and-tricks/adventures-with-snmp-and-cisco-nexus-pt2.html ... View more

Not out of the box - serverclasses can only use a small number of variables to control membership - such as name, IP address, OS. There are two general ways this could be achieved, but both require a bit of programming to get working Create a script to generate the serverclasses.conf content which has the required mapping. This would use data from Azure to identify which hosts have which tags and link them to the relevant serverclass. Then run the splunk reload deploy-server command to get the config active in Splunk. You script will need to pull the list of tags and hosts from Azure or maybe even you can use an existing Azure add-in to retrieve this data from Splunk. Use the "clientname" option in deploymentclient.conf to append a tag name to the computer name and then you can use the built-in wildcard options in the serverclasses.conf (or web interface). You will need a script running on each host to get the tags associated with that host and write that data to a local deploymentclient.conf file. This option is only workable for one or two tags per host. ... View more

ok, understood. I have also used CentOs7 for a lot of Splunk work without any issues, but not tested on CentOS8 yet. We do not deploy in production on CentOS, but use RHEL instead - but this is more around support and minimal environment from a security perspective. In terms of OS, we have a mix of Windows and Linux servers we run Splunk on. In some situations we find Windows easier to manage, but other cases we use Linux. It also depends on the functions we are trying to perform on each. ... View more

Are you looking for a recommendation for which OS to host Splunk on? i.e. indexers / deployment servers? Or is there a specific issue with CentOS you are concerned with? ... View more

From my interpretation of your base search - the issue is that the base search is not transforming. This is a requirement for post-processed searches https://docs.splunk.com/Documentation/Splunk/8.0.3/Viz/Savedsearches#Post-process_searches_2 If you take a simple search with stats/chart command and then run it in the standard search window, you will get the results you want. However if you split this in a form/dashboard and only have the initial search in the base search, you will not get any results from your post processing. You will need to add a stats command or similar to the base search to generate a table of results before this will work. If I take one of your examples, the full query is as follows index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" | search host = DMOPWMD1PDDB0* | eval FreeSpace =100-( Value ) | stats min(FreeSpace) as hostavg by host,instance | table host,instance,hostavg | chart min(hostavg) by host,instance It looks like you have created the following base search - however this only returns raw events and not an table index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" I would split this up as follows Base search index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" | eval FreeSpace =100-( Value ) | stats min(FreeSpace) as hostavg by host,instance Post-processing search |search host = DMOPWMD1PDDB0* |chart min(hostavg) by host,instance In many cases, you need to create a temporary stats table in the base search, just to get this to work, even if you would not normally do this in an interactive search. If you base search cannot be easily combined into a single stats table, then you can create multiple base searches. I don't see the code you are using for the search ID's so just in case, it needs to be in this general format <search id="BaseSearchName1"> <query>index=........</query> <earliest>-24h</earliest> <latest>now</latest> </search> ... View more

The name in the stanza is just my own name to identify the data collection. The reference to perfmon counters is defined in the "counters" and "object" fields under the stanza. I am only using standard counters that are available on Windows - to get a list on a particular server, run the following command in a command window - "typeperf -q". You will see if you run this on a Citrix server, you get more counters than would exist on a plain Windows servers So if I take two example [perfmon://citrix-userCPU] counters = CPU Entitlement; CPU Reservation; CPU Shares; CPU Usage; Long-term CPU Usage instances = * interval = 60 object = Citrix CPU Utilization Mgmt User useEnglishOnly=true mode = multikv showZeroValue = 1 [perfmon://citrix-tssessions] counters = Active Sessions;Inactive Sessions;Total Sessions instances = * interval = 60 object = Terminal Services useEnglishOnly=true mode = multikv showZeroValue = 1 Each perfmon stanza must be in the following format - [perfmon://] Where is your own personal name. In the first stanza, the specific perfmon counter is "Citrix CPU Utilization Mgmt User" - this comes from Xenapp (https://blog.citrix24.com/xenapp-6-5-performance-counters/) In the second stanza, the specific perfmon counter is "Terminal Services" - This is a standard terminal services counter - available on any Windows server using terminal services. I got all of these names by running the typeperf command or looking at the perfmon tool on a server. Any perfmon counter you can collect using native perfmon, you can collect via the Splunk. For example if I run the typeperf command and got the following output for the terminal services object - \Terminal Services\Total Sessions \Terminal Services\Inactive Sessions \Terminal Services\Active Sessions You can see in my second staza above, I am listing these counters under the "Terminal Services" object, I have listed all three, but you can decide which specific counters you actually want to collect. counters = Active Sessions;Inactive Sessions;Total Sessions Finally there is the "instances" field - this just defines whether you collect from a named instance or wildcard (i.e. all instances). In the context of Citrix/TS, we want to collect data from all sessions, so just use the wildcard instance. If you were collecting CPU data, you might just specify the "_total" instance rather than collect CPU data for all individual cores. ... View more

Just to clarify - Splunk forwarders / heavy forwarders can only push data to another Splunk instance - this is explicitly defined in the name "forwarder". They cannot pull data from another Splunk instance. You cannot "pull" data from a heavy forwarder. The only time you use Splunk to "pull" data is in the following scenarios A Splunk instance (forwarder/indexer etc) has an input defined to remotely collect data. For example to monitor a file on a remote share You send a remote command to a Splunk instance to read/write/apply configuration settings. - for example run CLI command or rest API call. However this generally relates to configuration only and not to perform regular data pulls. A splunk search head can point to a remote indexer to perform a remote search and effectively pull the data back to the search head - but the target needs to be an indexer which stores data. So if you want to have data remaining in the DMZ that you can "pull" - you would have to deploy an indexer to store this data in the DMZ, then you use a remote search instance to "pull" results as and when required. ... View more

There are two options for the frozen data - either writing the Splunk formatted buckets to disk (coldToFrozenDir) or run a script (coldToFrozenScript), both are configured on the index, in the indexes.conf file see - https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf. The coldToFrozenScript will run any defined script before the data is deleted - so it depends on what your script does to determine whether the raw data is in its original form or not. There is a sample script provided ($SPLUNK_HOME/bin/coldToFrozenExample.py) which can be modified for your requirements. The script contains a sample function called "handleOldFlatfileExport". This is not run unless you modify the code to use this function. There are also some articles about using this script - https://answers.splunk.com/answers/338594/does-anyone-have-a-working-example-of-coldtofrozen.html You would need to modify the script and test to see if you can get it working with your data. I would set-up a test index and set the "frozenTimePeriodInSecs" to a low number and then use a test file to import data into that index. You can then test the script in a short timeframe ... View more

Assuming you mean that you want to send the data before it is deleted from Splunk - then the answer is yes. See this link. Please note that the sending to the third party is not managed by Splunk you will need to manage that part yourself. https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Automatearchiving if you set-up a frozen path for an index, then Splunk will write the data to that path before removing/deleting from the index. You can then use another tool to send the data. For example you could set-up syslog to write the contents of the frozen folder to a third party system. Another option is to define a frozen script which will run when data is being deleted - which would allow a more complex data processing function to be implemented. ... View more

It is difficult to know exactly how to address this without an example event. Assuming that the "application" field is the one that filled with one of the possible application names, then you could use the fillnull command to assign a specific value and then filter on that - for example | fillnull value="no app" application | search application="no app" |stats values (application) by site computer This will assign the application field with the value "no app" when this field does not exist in a raw event. The search command then filters out any event which does not have an application assigned. Assuming there are computers with no applications assigned or you initial search is just for a single application, is missing that one application. ... View more