About jessieb_83

jessieb_83

You are correct, @gcusello , I wouldn't normally. The only reason I was listing it was because the error I was getting at start up about it being incorrect, so I was trying to override it. As it turns out, the error came from when I fixed it on the distributed indexes.conf file, it didn't actually SAVE the correction, so that peer-app kept overriding everything else I did with the wrong configuration. and it took me several hours of staring at the error before I actually saw it. So it was a carbon based error.

jessieb_83

In Indexes.conf from the CM, I tried to set thawedHomePath to a volume, which I have since learned does not work. I set the path from volume:cold back to $SPLUNK_DB, but no matter what I do the indexer will not acknowledge that I changed it back. It still thinks it's set to the volume. I modified it, commented it out, deleted the whole indexes.conf file and loaded a manual one in the `/etc/system/local/indxes.conf and nothing will un-stick it. Every time I start the indexer, the logs show it won't start because thawedHomePath is mapped to a volume still. When I run ~\splunk btool indexes list --debug it shows the thawedHomePath in question is configured correctly. Has anyone ever experienced this before? Any suggestions on how to get it to accept the change? Running Splunk 9.2 on RHEL 8 with 1 CM and 2 IDXs clustered together. Fairly new deployment, still working the bugs out.

jessieb_83

I don't recall exactly what fixed this, but I would hazard the guess it's to do with the sslPassword in server.conf

jessieb_83

I don't want to get too specific, because it may work different for different environments, but it they keys were: mount.cifs to mount the windows drive on athe destination linux machine rsync -avhipP copied from the windows drive to the linux drive and adjusted from a windows file structure to a linux file structure.

jessieb_83 · ‎05-31-2024

I've been fighting this for a week and just spinning in circles. I'm building a new distributed environment in a lab to prep for live deployment. All is RHEL 8, using Splunk 9.2. 2 indexers, 3 SH's, cluster manager, deployment manager, 2 forwarders. Everything is "working" I just need to tune it now. The indexers are cranking out 700,000 logs per hour, and it's 90% coming off audit.log; the indexers processing the logs in and out of buckets. We have a requirement to monitor audit.log at large, but do not have a requirement for it to index what the buckets are doing. I've been looking at different approaches to this, but I would imagine I'm not the first person to encounter this. Would it be better to tune audit.rules from the linux side? Black list some keywords in the indexers inputs.conf? Tuning through props.conf? Would really appreciate some advice on this one. Thanks!

jessieb_83 · ‎05-09-2024

I'm seeing this same error on a new build. Did you ever find an answer?

jessieb_83 · ‎05-09-2024

Running 9.2 and getting the same error

jessieb_83 · ‎04-16-2024

Finally figured out it was a permission issue. I didn't give splunk ownership over the index locations.

jessieb_83 · ‎04-11-2024

I left the Frozen drive to point to $SPLUNK_DB on the indexer's drive, but I'm not trying to employ frozen buckets at all. I'm trying to use the volumes on external drives for hot and cold, that's how our current instance is set up. The difference being the current is on Windows, and this new one is going to be on RHEL8.

jessieb_83 · ‎04-10-2024

I'm setting up a lab instance of Splunk Ent in prep to replace our legacy instance in a live environment and getting this error message: "homePath='/mnt/splunk_hot/abc/db' of index=abc on unusable filesystem" I'm running RHEL 8 VM's, running Splunk 9.1, 2 indexers clustered together and a cluster manager. I've attached external drives for hot and cold to each indexer. The external drives have been formatted in ext4 and set in fdisk to mount at boot every time as /mnt/splunk_hot and /mnt/splunk_cold and pointed indexes.conf by volume to them. They come up at boot, I can navigate to them and write to them. They're currently owned by root. I couldn't find who should have permission over them so I left them as is to start. I tried to enable OPTIMISTIC_ABOUT_FILE_LOCKING=1 but that didn't do anything. That being said, i suspect I've missed a step in the actions taken mounting the external drives. I wasn't able to find specifics about the way I'm doing this, so I pose the question: Am I doing something wrong, or missing a step on mounting these external drives? Is that now a bad practice? I'm stumped. my indexes.conf: [volume:hot] path=/mnt/splunk_hot [volume:cold] path=/mnt/splunk_cold [abc] repFactor = auto homePath = volume:hot/abc/db coldPath = volume:cold/abc/db thawedPath = $SPLUNK_DB/abc/thaweddb ##We're not utilizing frozen storage at all so I left it default Any advice here would be greatly appreciated!

jessieb_83 · ‎08-15-2023

Reviving an older post, I'm in this exact same position and am curious; How did you go about switching the file structure over? I've seen some people create scripts. Is that how this is getting done?

jessieb_83 · ‎12-09-2022

I don't remember at this point exactly what I did, but this is the article I used to get through it. Resync the KV store - Splunk Documentation

jessieb_83 · ‎06-17-2022

Did you find the solution? Different version, but I'm having the exact same problem.

jessieb_83 · ‎05-20-2022

The metadata is gone, but the indexed data is still there. I'm contractually obligated to maintain the last 5 years worth. I was just wondering if there was a way to look at the old data and see if I can figure out how big it was back then. Apologies being confusing. I was having a hard time putting that request to words.

jessieb_83 · ‎05-20-2022

I had not seen this before. It will be very helpful in our future planning. Thank you!

jessieb_83 · ‎05-20-2022

For the normal indexes ( not including _internal or the Drive monitor index) our retention policy is contractually specified to 5 years. I just want to see if we can extrapolate from what exists, how big it was back then.

jessieb_83 · ‎05-19-2022

I've been asked to find historical index volume information, going back 5 years, to make projections for future infrastructure and license needs. _internal is of no use, because it's cleared after 30 days. We track disk space, and I can find the disk space info for the Cold bucket on the indexer, but it's set to roll off after 60 days so that's out as well. I understand that anything like that would be slightly lower than the actual, as there are several indexes that whose index info would have rolled off, but I'm just trying to find a rough base line to track program growth. This is pretty far beyond my SPL abilities, so I would be grateful for any help!

jessieb_83 · ‎05-18-2022

@prakash007Did you ever find the solution?

jessieb_83 · ‎04-06-2022

Thanks for the reply Mayurr! I thought the same thing. I did find an app being pushed to all the UF's [and verified it's getting to the workstations] to override default with the following entry from %splunkHome\etc\apps\Splunk_UF\default\outputs.conf: [tcpout] forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = .* forwardedindex.2.whitelist = (_audit | _introspection | _internal | _telemetry) forwardedindex.fileter.disable = false It seems like the 0.whitelist entry is unnecessary but I wonder if that is actually conflicting with gathering audit info.

jessieb_83 · ‎04-06-2022

With little to no Splunk experience, I inherited a 7.2.3 windows deployment (We're closed network and I'm not cleared to upgrade yet) I've been finding little things here and there. One of the bigger ones being I'm ONLY getting _Audit logs from the Splunk servers; I'm not getting any audit input from any work stations, or other production servers. I've been dredging the boards for 3 days now and haven't found anything that seems along this line. I've checked the %Splunk\var\log\audit.log on several and the host's audit logs are getting input, but they're not getting ingested. I've gone through the deployment_app input.conf and output.conf files and don't see any glaring indications. So, I'm asking for ideas on other things to check.

jessieb_83 · ‎11-23-2021

https://wiki.splunk.com/Community:DeployHardenedSplunk This walk through covers binding to LDAP.

jessieb_83 · ‎08-28-2020

Trying to figure out what gets added upon running OUTPUTCSV vs manually exporting to CSV and how to remove it. We run a weekly audit I'd like to automate and then run a script to zip the resulting file. I did some experimenting and found that using outputcsv or outputlookup instead of exporting manually is almost tripling the file size of the output, from 300 Mb to 1.11 Gb. Exact same search criteria, the exact same time range, returning the exact same number of results. I also checked https://docs.splunk.com/Documentation/Splunk/8.0.5/SearchReference/Outputcsv and removed the internal fields it listed as being added. Any insight here would be great!

Posts	22
Solutions	2
Karma Given	9
Karma Received	1
Member Since	‎11-15-2019

Online Status	Offline
Date Last Visited	a week ago

Indexes.conf config stuck

Indexer indexing too much

Unusable Filesystem

How to find historical index volumes?

Why is _audit index not capturing all hosts?

What does outputcsv add to

Re: Indexes.conf config stuck

Indexes.conf config stuck

Re: Receiving "Client is not authenticated" Error

Re: Migrate Splunk from Windows to Linux question

Indexer indexing too much

Re: Receiving "Client is not authenticated" Error

Re: Has anyone else run into this Product Version ...

Re: Unusable Filesystem

Re: Unusable Filesystem

Unusable Filesystem

Re: Migrate Splunk from Windows to Linux question

Re: KV Store status failed

Re: KV Store status failed

Re: How to find historical index volumes?

Re: How to find historical index volumes?

Re: How to find historical index volumes?

How to find historical index volumes?

Re: WHat does it mean this error regarding bundle ...

Re: Why is _audit index not capturing all hosts?

Why is _audit index not capturing all hosts?

Re: Update BindDNpassword on deployer in authentic...

What does outputcsv add to