About HathMH

HathMH · ‎11-02-2022

Received error this morning on one of our non-distributed search head: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch. Nothing works, cannot search, dashboards are non-functional. Searching produces this error: Search not executed: The minimum free disk space (5000MB) reached for /opt/splunk/var/run/splunk/dispatch. user=admin., concurrency_category="historical", concurrency_context="user_instance-wide", current_concurrency=0, concurrency_limit=5000 I did quite a bit of digging in the community and found the following on my instances, non-distributed: Dispatch Tried the clean-dispatch command on our bloated 8873 count in /opt/splunk/var/run/splunk/dispatch Shut down splunk even run in sudo, results in error of Permission denied Ran command: ./splunk cmd splunkd clean-dispatch /temp -1day bundle files distsearches.conf has no maxbundlesize addressing the large .bundle files in /opt/splunk/var/run If I delete out the bundle files above, I can search for alittle bit on the search head, but then it craps out. Now, I am at a loss after reading so many articles, how-tos and docs. I'm not a splunk guy, but I am trying to get this stable.

HathMH · ‎10-20-2022

How did you troubleshoot this? I have 3 splunk instances down with this same issue

HathMH · ‎09-16-2022

I had a somewhat similar question a bit ago. Taking a search, sort by index, then further sort by values within the index, etc... Can I do all that in one search string, or do i ha... - Splunk Community The solution I received has put me on a good path to the intended result.

HathMH · ‎09-09-2022

I am not sure how to word this so I'm going to bring it as an example. We have 3 firewalls that send logs for ingestion. Each FW is for a separate purpose so they are configured slightly differently. Each appliance has their logs ingested into Splunk to go into separate indexes (due to their purposes and location in the logical topology). Within each firewall, there are of course field values that are helpful to sort and do stats on. Now my question: I am still learning spl, reading through Exploring Splunk by Carasso, so I don't have a full understanding in all the nuances. In one search string, can I reference each index, create a table for each index, which further divides and displays that index into categories like firewall action as one field, type of request as another field, and then provide stat counts on each of those categories (how many of field 1, field 2, etc) and then also provide a total bandwidth displayed (bytes)....all this within the same table. Index FW1 stat count ------ FW Action ---- (nested sort) Type of Request ---- bytes total Index FW2 stat count ------ FW Action ---- (nested sort) Type of Request ---- bytes total Index FW3 stat count ------ FW Action ---- (nested sort) Type of Request ---- bytes total Can I do all that in one search string, or do i have to create a search for each index?

HathMH · ‎09-04-2022

When setting up a index cluster, is there an option to make it multisite or something? I ask because our index cluster has 8 indexers on our 192.x main network, 1 deployment/master node on the 192.x network, and then 1 indexer on the 10.x network (our DR disaster recovery site).

HathMH · ‎09-02-2022

I inherited a splunk mesh of search-heads, deployment server, index cluster, etc. I am trying to figure out all this splunk stuff, but ran into an issue that I am not sure if it ignores best practice, poor judgement, or as intended. We have 8 main indexers that do what indexers do, all clustered as peer nodes. The deployment server is the master node and the search head for the cluster (which I don't understand that since we also have 5 main separate search heads). We also have a disaster recovery DR site that has an indexer as a peer node part of the aforementioned cluster. The cluster has a Replication factor 3, for # of copies of raw data. The cluster has a Search factor of 2, for # of searchable copies. Newer to cyber so forgive me if I don't understand right away or if I am missing something glaringly obvious. But does it makes sense to have the DR indexer be part of the cluster? If it does makes sense, then how do i ensure that the other 8 indexers send a copy of all their stuff to the DR indexer? I thought the master node just kind of juggles the incoming streams from the forwarders and balances the data across all the indexers. Also; - should the deployment server double as a master node and search head for the index cluster? - what is the difference between the 5 main separate search heads and the search head in the index cluster? - (last one, i swear) would it make sense to have a search head cluster, or keep the search heads separate as they the 5 are accessed and used by different groups (networking, development, QA/testing, cybersecurity, and UBA (which we dont even have UBA servers active right now cuz I cannot get them to work or web ui to launch))

HathMH · ‎08-10-2022

We are trying to standardize our nomenclature on indexes. Is it possible to rename an index along with moving data from the old index to new index name? Example: index "fit_azure" need to change to "top-azure" Are there things to consider that I'm probably not keeping in mind. My concerns doing this are: Would renaming the index require re-ingesting of data? If so, what about when ingesting data into an index can sometimes delete the log on the on the system inputting into splunk? How would this impact storage, hot/warm/cold and all that? - would previous storage be inaccessible to the new index naming? Would permissions on viewing the index change?

HathMH · ‎08-10-2022

Ran into an issue whereas an index is not viewable by a particular user. So I tried to see what permission the user has, and find that even though the user (steveng) can log in via the AD credentials, the user is not actually listed in the users page. Also, did a search with various search strings like |rest /services/authentication/users | search realname=* roles!=app* roles!=index* | dedup title type realname email tz roles | table title type realname email tz roles | rename title as Username realname as "Full name" tz AS "Time zone" email AS "Email address" type AS "Authentication system" No result still on the ghost account for steveng. Any thoughts?

HathMH · ‎08-08-2022

Thank you so much. I learned a lot from your reply. It appears we are not following some best practices such as the Master Node combined with the Deployment Server among others.

HathMH · ‎08-05-2022

Long post, newish to splunk, search strings are still a foreign language to me. So I am tasked with incorporating azure gov into splunk. Splunk support recommended to use a particular app for microsoft cloud services. The app is easy enough to configure and whatnot. But having issue with creating an index for the app and ingesting into splunk. We have the master node/deployment server, 8 indexers, 5 search heads, 2 heavy forwarders. How do i create an index in a index cluster? I ask because the directions seem easy enough, however there are some hiccups. When I look at our indexes listed in splunk web, it does not match what is shown in the indexes.conf files. Which is in itself an issue. These are the locations that I have found indexes.conf $SPLUNK_HOME/var/lib/splunk lists all my indexes and their dat files $SPLUNK_HOME/etc/system/default/ the default files $SPLUNK_HOME/etc/system/local/ has a listing of almost 80 indexes, but not all that are in the web portal search head, missing some of the sensitive indexes with naming conventions for systems like our txs and usr like txs_systemlog, usr-firewall, etc. I went to our master node and the location $SPLUNK_HOME/etc/master-apps/_cluster/local/ to look at what the indexes.conf file says there...but its not present. Yet we obviously have indexes across our cluster. So here are the issues: 1 - This prevents me from creating the needed index "usr-azure" as I do not where to put it. 2 - why are some indexes, like the sensitive ones, not listed in the conf files but are listed in the /var/lib/splunk/ ? 3 - Why is my master node web showing 48 indexes yet my indexers separately show 99 indexes? Additionally, another issue. I know we need to use CLI and edit the indexes.conf file for a indexer cluster, but I tried to do it via the web on indexer1, Settings > Indexes (under Data), and I can click the New Index button. All is good, but when I get to the the App selection, it only lists all the apps. Whereas all the indexes listed show TWC_all_indexes Q4 - how do i get that for this app setting "TWC_all_indexes" for new index I am creating? I assume it has something to do with the index clustering and a setting on the master node. But I don't even see that option in the indexes.conf file.

HathMH · ‎07-28-2022

I have been asked to check with Splunk Support on whether we can run 2 different Splunk add-ins for "Splunk Add-on for Microsoft Cloud Services". Can we have one connect to Azure Commercial while the other connects to Azure Government event hubs? Or is this a case in which we would need 2 separate splunk servers to support that? What else could we do? IE. could we set it up on the heavy forwarder in the FTI subscription for Government for server 1 and use the existing server for commercial?

HathMH · ‎07-25-2022

First, new to splunk, learning as I go. Oh, BTW, I'm the splunk 'person' now in my org. Trying to figure out how to get MS Azure Gov into splunk securely. Yay! In preparation for this, I have decided to see about the MS Security Graph app and where it should be installed. Ok, so today I learned that "Splunk recommends installing Splunk-supported add-ons across your entire Splunk platform deployment, then enabling and configuring inputs only where they are required" and " you can install any add-on to all tiers of your Splunk platform architecture – search tier, indexer tier, forwarder tier – without any negative impact." Which I got those from the splunk doc site: Where to install Splunk add-ons - Splunk Documentation Our architecture looks ad-hoc more than anything. We have apps on a search head but not on others, we have apps on 1 or 2 indexers but not the rest. That's just the apps from splunkbase. So now I have this task to create a spreadsheet of which instances have what apps so that I may streamline and make it so across the board. Question 1: Is it truly best to install an app on all instances, across all tiers? Example, a forensic investigator tool that really only interacts with the splunk portal for (a search head), does it really need to be on forwarders and the indexers? Question 2: Is there a way to export the list of apps installed on a splunk instance. This is so i can make an easy spreadsheet of what server has what app and then start the task of ensuring that app is spread across the board. Question 3a: Do I really need all of the MS add-ons? Microsoft Graph Security API add-on for Splunk, Microsoft Sysmon Add-on, Splunk Add-on for Microsoft Windows, Splunk Add-on for PowerShell, TA-microsoft-sysmon_inputs? Question 3b: I don't really see others that I would have thought would be good like Splunk Add-on for Microsoft Security (by Splunk), Splunk Add-on for Microsoft Office 365 (by splunk), and others. Would it be beneficial to have those? Question 4: Anyone have experience, do's and don'ts for the Microsoft Graph Security API add-on for Splunk? I have been told this is the app to install and configure to ensure Azure Gov data is brought into splunk securely.

HathMH · ‎07-08-2022

New to cybersecurity, been in my first entry level job for 6 months. New to splunk, took some classes but they were quick and didn't detail a whole lot, splunk instructor read the slides basically. Ran into issue, red warning on 8.2.4 The percentage of small buckets (100%) created over the last hour is high and exceeded the red thresholds (50%) for index=mail, and possibly more indexes, on this indexer. At the time this alert fired, total buckets created=5, small buckets=1 and then it would list that last 50 related messages, early this morning it did....but now it says 'None' It happens on indexer #7 of the 8 we have. Crawling the web to gain some understanding. Found this link: Solved: The percentage of small of buckets is very high an... - Splunk Community The OP had the same issues but talked about timeparsing and they were able to fix. - what is and how to fix timeparsing? I am not great with search strings, regex, etc...splunk just kind of fell in my lap. I tried to follow the search string that @jacobpevans wrote up in reply to the post above, not sure if I follow it well. basically its serching for each hot bucket for index _internal sourcetype splunkd, listing those hot buckets that are moving to warm, renaming to index to join, join command then joins each intance to a rollover. I run his string as is, and get a lot of readout listing many indexes but not index mail as indicated in the warning. Also the readout shows 4 rows (2 indexes on 2 different indexers) with Violation and 100% small buckets. I would like to resolve this issue but i am seriously lost, haha. I think splunk may be the death of my career even before i get started.

HathMH · ‎06-23-2022

Thank you, I tried that and it errors out. I may play around with that index=* [] and see what happens

HathMH · ‎06-22-2022

First of all I am new to cyber, and got splunk dumped in my lap. I am really trying to get knowledgeable on it but 1) I am horrible with coding and apparently that includes Regex 2) Long lines of code or search strings is like sensory overload to me That being said, I am trying to clean up our alerts as we are getting bogged down daily with well over 3k alerts that could most likely be expunged. Many of our alerts are based on tstat search strings. It shows a great report but I am unable to get into the nitty gritty. For example, the brute force string below, it brings up a Statistics table with various elements (src, dest, user, app, failure, success, locked) showing failure vs success counts for particular users who meet the criteria in the string. My issue, I try to click on a user, choose view events, brings up new search with a modified string (of course) but still only shows tstats table, but with different headers (action, src, det, user, app, count, failure, success). What I would like to do is when I click to choose view event for a particular user, it actually shows me that even and correlating log input. Is this possible? Why would I want an brute force alert if I cannot narrow down to the events, especially the failure logins? Again, please have mercy, I am entry level and still learning splunk. I love the apps and abilities it has but using the search box is like i lost all my intelligence. The brute force search | tstats `summariesonly` values(Authentication.app) as app,count from datamodel=Authentication.Authentication by Authentication.action, Authentication.src, Authentication.dest, Authentication.user | `drop_dm_object_name("Authentication")` | search user!=unknown user!=SYSTEM app!=splunkd_remote_searches src!=MWG* user!=TWC-* | eval success=if(action="success",count,0),failure=if(action="failure",count,0) | stats values(dest) as dest, values(user) as user, values(app) as app, sum(failure) as failure, sum(success) as success by src | join user [search index=top_wineventlog EventCode=4740 | eventstats count(user) as locked_count by user | dedup user, host | table user, locked_count] | search failure>30 success>0 | where failure>success

Posts	16
Solutions	0
Karma Given	12
Karma Received	0
Member Since	‎05-09-2022

Online Status	Offline
Date Last Visited	‎11-02-2022 04:07 PM

How can I resolve clean-dispatch issues- Can't sea...

Can I do all that in one search string, or do i ha...

Does it make sense to have a DR indexer be part of...

Is it possible to rename an index along with movin...

User accounts not listed- Why is index not viewabl...

How to create new index on index cluster?

Can we have one connect to Azure Commercial while ...

Splunk addons, listing

Why is the percentage of small of buckets are very...

Why can I not see events from tstats?

How can I resolve clean-dispatch issues- Can't sea...

Re: Splunk not running on web interface- How shoul...

Re: How to make a new row in a table based on one ...