Is there a need for keeping the _internal index logs past a certain time period? My _internaldb is pretty large at 218GB total, db - 31, cold - 112, frozen - 75. You can see my current settings below. We have about 140 forwarders reporting to this indexer. Should I just remove the path to frozen and let them get deleted? Does anyone ever thaw internal logs? If so, what for? [_internal] homePath = $SPLUNK_DB\_internaldb\db coldPath = $SPLUNK_DB\_internaldb\colddb thawedPath = $SPLUNK_DB\_internaldb\thaweddb coldToFrozenDir = $SPLUNK_DB\_internaldb\frozendb frozenTimePeriodInSecs = 5184000 tstatsHomePath = volume:_splunk_summaries\_internaldb\datamodel_summary maxConcurrentOptimizes = 6 maxWarmDBCount = 60 maxHotSpanSecs = 86400 maxHotBuckets = 8 maxDataSize = auto ... View more

Not a question. I struggled with working getting the regex syntax correct for a while to blacklist some noisy event code items and wanted to post my successful strings. Just in case someone else searches and finds this useful.  blacklist1 = EventCode="4688" Message="New Process Name:\s+(?:[C-F]:\\(?:Program Files\\SplunkUniversalForwarder|Splunk)\\bin\\(?:splunk|splunkd|splunk-optimize|splunk-powershell|splunk-admon|splunk-netmon|splunk-MonitorNoHandle|python3|btool)\.exe) blacklist2 = EventCode="(4663|4660|4907)" Message="Process Name:\s+(?:[C-F]:\\(?:Program Files\\Microsoft Configuration Manager\\bin\\X64|Program Files \W\w{3}\W\\Symantec\\Symantec Endpoint Protection\\\d{1,5}\.\d{1,5}\.\d{1,5}\.\d{1,5}\.\d{1,5}\\Bin64|Program Files\\SMS_CCM|Windows\\System32|Windows\\System32\\(?:inetsrv|wbem)|.WINDOWS.~BT\\Sources|Windows\\WinSxS\\amd64_microsoft-windows-servicingstack_\w{16,20}\.+\d\.+\d{1,6}\.+\w{22,30})\\(?:smsexec|sitecomp|TiWorker|SetupHost|WmiPrvSE|w3wp|poqexec|CcmExec|ccSvcHst)\.exe) I'm a novice with regex so there might be some ways to clean this up and make it shorter, but it works. If anyone has simplification recommendations, feel free to share. ... View more

Every month when software updates go out, my Enterprise deployment exceeds the license. I get overloaded with Event Code 4663. After the first time, I just added it to the blacklist in inputs.conf and problem solved. I'd like to leave that EventCode active and only disable it when the majority of systems are updating. I know I can do this manually but am trying to find if there is a way to automatically enable the blacklist based on date? Or to set a trigger based on a specific series of event codes that indicate software updates? If anyone has tried this before I'm very curious if there's a solution? ... View more

I admin a Splunk Enterprise instance for an isolated LAN that has 3 workstations and two DCs (1 is file server, 2 is Splunk indexer and search head | both 32 GB RAM). Every 7-10 days, the Splunk server starts to slow down and then I'm completely unable to sign-in. Have to do a hard restart and it works fine again. Rinse and repeat. I was glancing through the performance metrics on the "Monitoring Console" and saw a build up and spike of CPU and memory usage right before the freeze.  The issue is that I cannot find any reason from Splunk as to why this is happening. The daily indexing rate over the last 30 days had a high of 1.3 GB with a max allowable of 5GB. I don't have any custom scheduled searches so maybe there's a default setting that is getting replicated? If anyone has dealt with a similar issue I'd appreciate a suggestion. Not sure if related: About a year ago we had big issues with indexing rates and event log duplication which caused us to exceed the license several times. We resolved that but this issue started around the same time. ... View more

I'm trying to differentiate between cd burns and cd read codes from Window Event Viewer using WinZipBurn. From what I've seen, a cd burn will generate event codes 1001, 1003, 1004, and then 1001 again in WinEventLog:Application. Reading from a cd will just generate code '1001.'   I'm trying to create a report that will determine if the sequence 1001, 1003, 1004, 1001 occurs within a 10 minute span. I'm not great with transactions so I'm sure there is an error. The following is what I tried: sourcetype=WinEventLog:Application WinZipBurn | transaction Account_Name maxspan=10m maxpause=3m startswith=eval(EventCode="1001") endswith=eval(EventCode="1001") I don't think it can tell the difference between the first one and the last. In EventViewer, there is more info that says the cd is blank vs finalized but Splunk isn't pulling it over. 1003 and 1004 are error codes that I'm not sure what for but they only occur in the middle of a burn. ... View more

I'm trying to configure my indexes.conf to roll all db files based on time. Hot -> Warm (1 day) -> Cold (2 weeks) -> Frozen (6 months).  Now I know how to do the cold to frozen and frozen to thawed but I'm trying to figure out if I can do Hot to Warm to Cold based on time and not size. I found references to a work around with the following set up [main] maxHotBuckets = 3 maxHotSpanSecs = 86400 (1day) maxHotIdleSecs = 86400 maxWarmDBCount = 14 frozenTimePeriodinSecs = 15724800 (6 months) coldToFrozenDir = <path> thawedPath = <path> Will this work to roll buckets from hot to warm in 24 hours, then from warm to cold in 2 weeks? Does anyone see an issue with this? ... View more

I recently took over as an admin for Splunk on one of my company's networks. We have 4 Forwarders and one enterprise instance. We recently updated our workstations and started getting large increases in events and exceeded our index by 8x everyday. I recently monitored the data at different points in the day and realized every event is getting re-indexed every minute. I watched one time period grow from 2500 events to 250,000 by the end of the day. If i refreshed the search it would have an additional 1200 events every minute (roughly). What could be causing Splunk to re-index the same events everytime a new one gets logged? ... View more

I'm dealing with a lot of duplicate event logs at the exact same millisecond. From what I can tell, everytime this happens, the events in my search results have all of the same data. There are some events that follow with slightly different fields including a ascending "Record Number" count. Should the record number be different for every log?  I'm looking for a better way to identify duplicate data and stop logging it. In the past 16 hours, I've logged 550 events with different record numbers but 390,000 different events. Some record numbers repeating up to 600 times. If you have a solution for this I'd appreciate it, but also just looking for a Unique ID to confirm this is an issue. ... View more

I don't have much experience with Splunk but am starting to use it in a new role and have done a lot of research before asking this question. There are two parts and I cannot provide screenshots. I'm running Splunk Enterprise with 3 workstations and 1 DC forwarding to the backup DC which holds the Splunk Server. We recently did a hardware update and began exceeding our license by 3-4x per day. The configuration didn't change and I cannot find what is causing this. I blacklisted the 10 event codes that were generating 80% of the logs and while they are no longer showing in my search, the server appears to continue to index them and by 8am today my index capacity was at 17500MB/5000MB for the day. I've also noticed anywhere from 50-1500 event logs for a single "Record Number." It's my understanding that a record number is unique to a single event and this means one event is getting logged several times. The time stamp is the same down to the millisecond. This I would argue is the bigger issue. WinEventLog://Security disabled = 0 start_from = newest blacklist = 4648,4701,.... <-- ... is not literal, just have 8 more ... View more