About wyfwa4

wyfwa4 · ‎12-14-2021

We have just upgraded to v8.1 and because we have a small license, we are subject to the license enforcement. The document states that enforcement will occur if you receive 45 warnings over a rolling 60-day window. What is unclear is what counts as a "warning". For example I have 9 indexers all sharing a single license pool, and when we went over the daily limit, we appears to receive 9 warnings - one per indexers. Is this expected? - for example This pool has exceeded its configured poolsize=xxx bytes. A CLE warning has been recorded for all members So does the 45 warning limit apply to these pool warnings?, hard warnings or license master warnings? I.e. going over the daily limit = 1 warning?

wyfwa4 · ‎05-21-2021

By default the metrics will only provide details on the top 10 items in a series - this applies to almost all the metrics that are collected. https://docs.splunk.com/Documentation/Splunk/8.1.3/Troubleshooting/Aboutmetricslog The way to address this, is to change the limit applied using the limits.conf file - the following stanza added to a indexer / HF will increase the number of items which are tracked [metrics] maxseries = 50 interval = 60 This will ensure the top 50 items are included - so you can adjust this to reflect the likely number of sourcetypes if that is the main metric to be tracked The "interval" reference allows you to change the frequency that this data is collected to control the volume of data generated in the log. In the same, the frequency is 60seconds to allow minute by minute tracking - but that may be excessive for some purposes.

wyfwa4 · ‎09-11-2020

It appears that Splunk detects the "_time" fields and makes a decision that this should be first - I assume as _time is the only true law in the universe. So if you rename the field to a custom name like "timefield" - is it no longer considered a special case and will now follow the order specified by the fields or table command. Another consideration is that when a field is called "_time" and contains an epoch time value - Splunk will automatically convert it to a readable format. However when you rename to something else, Splunk will just show the epoch value. So you also need to add a eval with strftime to convert the value to your preferred readable time.

wyfwa4 · ‎05-21-2020

Did you manage to fix the issue? or is the issue still occuring with the ipv6 events?

wyfwa4 · ‎05-15-2020

I would not rule out a regex issue. I am not saying there is a problem with the props or transforms, but the source data- we have seen issues with firewall logs where local configurations on each firewall meant the structure of each event was slightly different from different devices. I am also assuming that you have checked the full list of fields that are extracted - i.e many fields will be hidden if Splunk thinks they do not contain useful information. I just want to cover everything and without additional information, I have no way of knowing how familiar you are with Splunk. You also mention that you have multi searchheads - and an index cluster. When searching, the field extractions on each indexer are controlled by the search head configuration, but this has to be replicated to each indexer through the knowlegde bundle (https://docs.splunk.com/Documentation/Splunk/8.0.3/DistSearch/Knowledgebundlereplication). I have seen issues with problems on individual indexers where replication is delayed or corrupted and this is not immediately obvious when you get your search results.

wyfwa4 · ‎05-15-2020

Is this on a single indexer? or through multiple indexers? It is also not clear if you are referring to index time field extraction or search time - each will have completely different root causes and ways to diagnose the issue. In most cases this can be down to even a single character or extra space in the raw event which makes the regular expression not work. Maybe some examples of events that work and some that do not work may help. For example if occurring at specific times in the day, maybe the timestamp in the raw event changes from single digit hour to double digit hour

wyfwa4 · ‎05-06-2020

I would check the version of snmp_ta you are using and the version of Splunk you are trying to get working on. I suspect that you need specific versions of each to work together. There are also some other answers which suggest some possible issues in Splunk v8 - https://answers.splunk.com/answers/144720/snmp-input-type-not-displaying-in-splunk-web.html Please also note that the SNMP modular input (I presume you are using this - https://splunkbase.splunk.com/app/1537) now requires an activation key and so may not work otherwise. Personally I could not get the SNMP modular input to work reliably - it would often fail on complex data and getting the MIB files converted to work is a big pain. Depending on the type of device you are collecting from, MIB's are essential to get any sort of readable data. My preferred approach is to use http://www.net-snmp.org/. Just create a standard batch/shell script using snmpwalk to collect the data. Not only can you easily test outside of Splunk, but it allows you to collect large number of different targets by just expanding your script. You then create a scripted input in Splunk to run the script on your required frequency. If can get a script working outside of Splunk, it is not too difficult to then get it working inside Splunk. if you want to stick with the modular input, here is some info - https://www.splunk.com/en_us/blog/tips-and-tricks/making-snmp-simpler.html These two articles are also very helpful, especially around transforming SNMP results when they are received which is useful whether you are using the modular input or another script. https://www.splunk.com/en_us/blog/tips-and-tricks/adventures-with-snmp-and-cisco-nexus-pt1.html https://www.splunk.com/en_us/blog/tips-and-tricks/adventures-with-snmp-and-cisco-nexus-pt2.html

wyfwa4 · ‎05-06-2020

Not out of the box - serverclasses can only use a small number of variables to control membership - such as name, IP address, OS. There are two general ways this could be achieved, but both require a bit of programming to get working Create a script to generate the serverclasses.conf content which has the required mapping. This would use data from Azure to identify which hosts have which tags and link them to the relevant serverclass. Then run the splunk reload deploy-server command to get the config active in Splunk. You script will need to pull the list of tags and hosts from Azure or maybe even you can use an existing Azure add-in to retrieve this data from Splunk. Use the "clientname" option in deploymentclient.conf to append a tag name to the computer name and then you can use the built-in wildcard options in the serverclasses.conf (or web interface). You will need a script running on each host to get the tags associated with that host and write that data to a local deploymentclient.conf file. This option is only workable for one or two tags per host.

wyfwa4 · ‎05-04-2020

I presume you are seeing the known issue with perfmon process CPU collection. There is a specific page which covers this in the docs - https://docs.splunk.com/Documentation/Splunk/7.3.0/ReleaseNotes/WorkaroundforPerformanceDataHelperAPIWindowswindowsmulticore It looks like a workaround was added to v7.3 and above by using the useWinApiProcStats option. However this does not look like a permanent fix, just a workaround for some people. You would need to test if this works for your environment. The other options are to use other sources for perfmon data rather than using Splunk modular inputs - for example use a powershell script to collect the data and read the output, or run perfmon in CSV collection mode and read the output files. None of ideal and will all need work to get up and running.

wyfwa4 · ‎04-30-2020

ok, understood. I have also used CentOs7 for a lot of Splunk work without any issues, but not tested on CentOS8 yet. We do not deploy in production on CentOS, but use RHEL instead - but this is more around support and minimal environment from a security perspective. In terms of OS, we have a mix of Windows and Linux servers we run Splunk on. In some situations we find Windows easier to manage, but other cases we use Linux. It also depends on the functions we are trying to perform on each.

wyfwa4 · ‎04-30-2020

Yes, it is possible to use the app "Splunk App for Windows Infrastructure" without AD access. The app covers a wide range of data collection of which AD is just one type of data. The dashboards will just be empty for those items you do not collect data from. The app itself does not collect data, for that you need the Splunk Add-on for Windows (https://splunkbase.splunk.com/app/742/). This app contains all the data collection options and you need to determine which are enabled or disabled. I believe these are all disabled by default - so you need to specifically decide which to enable. The Add-One for windows would be installed on all the servers that you need to collect data from (deployed within a Splunk forwarder if collecting from hosts other than the Splunk server) and the Splunk App for Windows Infrastructure is installed on the Splunk server only. The app provides the data processing logic and dashboards, while the add-on simply collects the data. You can think of these apps as a starter-pack to show what can be collected and how the data can be presented in Shell - but can be quite daunting with such a wide range of possible data sources. I tend to use my own data collection apps based to keep the collection configurations simple and easier to maintain. For example if you want to collect Windows event logs - the process is covered here - https://docs.splunk.com/Documentation/Splunk/8.0.3/Data/MonitorWindowseventlogdata

wyfwa4 · ‎04-30-2020

Are you looking for a recommendation for which OS to host Splunk on? i.e. indexers / deployment servers? Or is there a specific issue with CentOS you are concerned with?

wyfwa4 · ‎04-30-2020

The issue is that files in .evtx format are not readable - they are a custom binary format used by Microsoft. So even if you tries to read them on a Windows based Splunk server, it would not work. If they are sitting on a disk folder, then somebody has exported them and they are no longer Windows event logs, but just files containing data extracted from a windows event log. When using the standard Splunk Windows logs collection process - [WinEventLog://Application] - this is using API calls to read each event, rather than trying to read a file directly on disk. You will need to either convert the files to readable text, or switch to reading the events within the eventlog before being exported. There seems to be some details on using the tool WEVTUTIL to perform this conversion. https://techcommunity.microsoft.com/t5/ask-the-performance-team/windows-vista-and-exported-event-log-files/ba-p/372550

wyfwa4 · ‎04-30-2020

From my interpretation of your base search - the issue is that the base search is not transforming. This is a requirement for post-processed searches https://docs.splunk.com/Documentation/Splunk/8.0.3/Viz/Savedsearches#Post-process_searches_2 If you take a simple search with stats/chart command and then run it in the standard search window, you will get the results you want. However if you split this in a form/dashboard and only have the initial search in the base search, you will not get any results from your post processing. You will need to add a stats command or similar to the base search to generate a table of results before this will work. If I take one of your examples, the full query is as follows index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" | search host = DMOPWMD1PDDB0* | eval FreeSpace =100-( Value ) | stats min(FreeSpace) as hostavg by host,instance | table host,instance,hostavg | chart min(hostavg) by host,instance It looks like you have created the following base search - however this only returns raw events and not an table index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" I would split this up as follows Base search index=perfmon source="Perfmon:LogicalDisk" counter="% Free Space" | eval FreeSpace =100-( Value ) | stats min(FreeSpace) as hostavg by host,instance Post-processing search |search host = DMOPWMD1PDDB0* |chart min(hostavg) by host,instance In many cases, you need to create a temporary stats table in the base search, just to get this to work, even if you would not normally do this in an interactive search. If you base search cannot be easily combined into a single stats table, then you can create multiple base searches. I don't see the code you are using for the search ID's so just in case, it needs to be in this general format <search id="BaseSearchName1"> <query>index=........</query> <earliest>-24h</earliest> <latest>now</latest> </search>

wyfwa4 · ‎04-20-2020

Since we upgrades our UF to v7.2.9, we are seeing lots of application crash errors in the application event log on our hosts. This is happening on large volumes of hosts. Initially I thought it may be a specific counter, but it occurs when the Splunk-Perfmon.exe process is running, even if no perfmon collection is occurring. I don't see any errors in Splunk itself and the Splunk-Perfmon process itself keeps running and sending data. Looking into these errors, there seems to be some suggestion this is related to "data execution prevention" which is blocking Splunk trying to run code in data memory (error include code c0000005 which is an access denied error) , but I have not been able to confirm this. servers previously running v6 did not show this error, only when upgraded did the error start to appear. example error below SourceName=Windows Error Reporting EventCode=1001 EventType=4 Type=Information ComputerName=xxxxxxxxxxxx TaskCategory=The operation completed successfully. OpCode=Info RecordNumber=230239 Keywords=Classic Message=Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0 Problem signature: P1: splunk-perfmon.exe P2: 1794.2305.24028.63924 P3: 5ddcfc22 P4: splunk-perfmon.exe P5: 1794.2305.24028.63924 P6: 5ddcfc22 P7: c0000005 P8: 00000000005bc5d8 P9: P10: Attached files: These files may be available here: C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_splunk-perfmon.e_2f9ed6fb118b57ac0e734f67ff573c73ad1654a_64da0b14_48835327

wyfwa4 · ‎04-06-2020

The name in the stanza is just my own name to identify the data collection. The reference to perfmon counters is defined in the "counters" and "object" fields under the stanza. I am only using standard counters that are available on Windows - to get a list on a particular server, run the following command in a command window - "typeperf -q". You will see if you run this on a Citrix server, you get more counters than would exist on a plain Windows servers So if I take two example [perfmon://citrix-userCPU] counters = CPU Entitlement; CPU Reservation; CPU Shares; CPU Usage; Long-term CPU Usage instances = * interval = 60 object = Citrix CPU Utilization Mgmt User useEnglishOnly=true mode = multikv showZeroValue = 1 [perfmon://citrix-tssessions] counters = Active Sessions;Inactive Sessions;Total Sessions instances = * interval = 60 object = Terminal Services useEnglishOnly=true mode = multikv showZeroValue = 1 Each perfmon stanza must be in the following format - [perfmon://] Where is your own personal name. In the first stanza, the specific perfmon counter is "Citrix CPU Utilization Mgmt User" - this comes from Xenapp (https://blog.citrix24.com/xenapp-6-5-performance-counters/) In the second stanza, the specific perfmon counter is "Terminal Services" - This is a standard terminal services counter - available on any Windows server using terminal services. I got all of these names by running the typeperf command or looking at the perfmon tool on a server. Any perfmon counter you can collect using native perfmon, you can collect via the Splunk. For example if I run the typeperf command and got the following output for the terminal services object - \Terminal Services\Total Sessions \Terminal Services\Inactive Sessions \Terminal Services\Active Sessions You can see in my second staza above, I am listing these counters under the "Terminal Services" object, I have listed all three, but you can decide which specific counters you actually want to collect. counters = Active Sessions;Inactive Sessions;Total Sessions Finally there is the "instances" field - this just defines whether you collect from a named instance or wildcard (i.e. all instances). In the context of Citrix/TS, we want to collect data from all sessions, so just use the wildcard instance. If you were collecting CPU data, you might just specify the "_total" instance rather than collect CPU data for all individual cores.

wyfwa4 · ‎03-31-2020

I suspect there is more going on in your logs than a simple Citrix logon. On my Citrix servers, I see three basic events per logon 4648 A logon was attempted using explicit credentials 4624 An account was successfully logged on 4627 Group Membership information The logon type is 10 and all three events are linked by the the same user name in the field "TargetUserName". Logon Type 8 is never used for a direct RDP/ICA logon, but can often relate to IIS basic authentication. So it is possible (just a guess) that you are seeing events from the Citrix web interface. So users will logon to the CWI first and then enumerate apps and only when they have selected an app, be directed to their target Citrix server. If you can filter out these possible CWI logons, it may make the process simpler to understand. Personally I find tracking Citrix sessions via windows logs to be too complex and CPU intensive - you generally have to use the transaction command to combine all of these events which can be very slow and sessions can last for hours, meaning the transaction cover unlimited time spans. So I user two other methods to track this data. You don't mention if you are running a forwarder on the server or you have the ability to collect additional data, but if you are, these may help. 1) User perfmon to collect Citrix specific counters - there is one counter you can use to get the overall number of active/disconnected sessions. This only gives the overall numbers, not per user info. [perfmon://citrix-tssessions] counters = Active Sessions;Inactive Sessions;Total Sessions object = Terminal Services 2) Use a scripted input to run the command "qwinsta" on the end server. This will give you a list of all active sessions including user names. I run this on a scheduled basis (for example every 5 mins) and then I can track sessions for each user very easily.

wyfwa4 · ‎03-31-2020

Just to clarify - Splunk forwarders / heavy forwarders can only push data to another Splunk instance - this is explicitly defined in the name "forwarder". They cannot pull data from another Splunk instance. You cannot "pull" data from a heavy forwarder. The only time you use Splunk to "pull" data is in the following scenarios A Splunk instance (forwarder/indexer etc) has an input defined to remotely collect data. For example to monitor a file on a remote share You send a remote command to a Splunk instance to read/write/apply configuration settings. - for example run CLI command or rest API call. However this generally relates to configuration only and not to perform regular data pulls. A splunk search head can point to a remote indexer to perform a remote search and effectively pull the data back to the search head - but the target needs to be an indexer which stores data. So if you want to have data remaining in the DMZ that you can "pull" - you would have to deploy an indexer to store this data in the DMZ, then you use a remote search instance to "pull" results as and when required.

wyfwa4 · ‎07-03-2019

I currently use the TA_NMON add-on to collect data from various linux hosts. This works ok, but generates lots of data. The problem I have is that I want to have a high frequency for very dynamic data (such as CPU usage) which only generates a few events per collection, while reducing the frequency of fairly static data such as DF_Table DF_Inodes which generate large numbers of events per collection where large numbers of disks are installed. However for disk information, the data is fairly static and we can track the trends by only collecting this data once every 5 or 10 minutes. While for CPU usage, we would like to see every 30 or 60 seconds. One option I have looked at, is to create two copies of the same app, but with different data collection frequencies, but this is challenging to set-up, due to the use of fixed var folders to store the collected data. Any suggestions on how to set-up different schedules within TA_NMON?

wyfwa4 · ‎05-31-2019

There are two options for the frozen data - either writing the Splunk formatted buckets to disk (coldToFrozenDir) or run a script (coldToFrozenScript), both are configured on the index, in the indexes.conf file see - https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf. The coldToFrozenScript will run any defined script before the data is deleted - so it depends on what your script does to determine whether the raw data is in its original form or not. There is a sample script provided ($SPLUNK_HOME/bin/coldToFrozenExample.py) which can be modified for your requirements. The script contains a sample function called "handleOldFlatfileExport". This is not run unless you modify the code to use this function. There are also some articles about using this script - https://answers.splunk.com/answers/338594/does-anyone-have-a-working-example-of-coldtofrozen.html You would need to modify the script and test to see if you can get it working with your data. I would set-up a test index and set the "frozenTimePeriodInSecs" to a low number and then use a test file to import data into that index. You can then test the script in a short timeframe

Posts	43
Solutions	7
Karma Given	2
Karma Received	12
Member Since	‎02-24-2016

Online Status	Offline
Date Last Visited	‎12-14-2021 03:29 PM

licence enforcement for multiple indexers - warnin...

Splunk-Perfmon crash errors in forwarder 7.2.9

ta_nmon data collection frequency for different ty...

how to change 5 minute timeout on script run from ...

Why am I getting duplicate saved search results af...

NMON Performance Monitor for Unix and Linux System...

Why am I getting incorrect results from btool duri...

licence enforcement for multiple indexers - warnin...

Re: Daily index volume by sourcetype

Re: Why does Email Report change column order?

Re: Juniper firewall search time field extraction ...

Re: Juniper firewall search time field extraction ...

Re: Juniper firewall search time field extraction ...

Re: snmp modular input

Re: Using Azure Tags for Server Classes

Re: perfmon issue with %ProcessorTime scaling/max ...

Re: Splunk on CentOS 8

Re: Can I use Splunk App for Windows Infrastructur...

Re: Splunk on CentOS 8

Re: windows evtx logs to splunk linux deployment u...

Re: Issue with the search base

Splunk-Perfmon crash errors in forwarder 7.2.9

Re: windows citrix

Re: windows citrix

Re: Heavy Forwarder Server vs Clients connecting d...

ta_nmon data collection frequency for different ty...

Re: How to send indexed logs from Splunk to third ...