About hectorvp

hectorvp · ‎10-15-2020

Hi @richgalloway , There would be other indexers who will be receiving actual logs from UF, those indexers are not going to be managed by our team. Our team has only responsibility to manage UFs and DS for UF. Although there are unknown factors, I atleast need to know whether eventcounts that I've estimated is in the range of prod server since we do not have approval to install UFs yet on servers. We are implementing this for the first time and hence have no clue of ideally how many internal events get generated when agents are on prod. Or shall I consider 500 events per minute??? To keep it at higher end or 300 events per minute are enough. I'm calculating based events per minute bcoz metrics.log will fire logs at an interval of 30s so....

hectorvp · ‎10-15-2020

Hi Splunkers, I've been working over capacity planning where for estimating indexer requirement. I'm stuck while calculating disk space. Our indexers are supposed to only store internal logs. And internal logs are only from UFs. Since no Splunk setup is present at prod servers currently, I'm not able to get how many internal logs are generated per day by UF. What I did is I spun a VM ,installed a UF and found that 288k internal events per day i.e 200 events per minute are generated. While calculating disk space I'm considering 300 events/min i.e 200 + adding 100 events/min as a buffer. Can anyone help me to give a that on average how many internal events are generated by 1 UF on prod server at every day or at every minute??? I understand there may be many factors like no.of addons, sources and all; just need to confirm that are prod servers event count around 300 events/minute,is the estimation in range?? In simple terms I need answer of this "query" for prod servers to get events per minute "tstats count where index=_internal and host ="ANY one host randomly" groupby index, _time span=60s | stats avg(count) PS: I dont have splunk set up to check this prior deciding disk space using search head.

hectorvp · ‎10-05-2020

Thanks @gcusello , Since I'm storing only internal logs in my standalone indexer, I may use raid0 or 15k rpm sas HDD....still would think over it.

hectorvp · ‎10-05-2020

Hello Splunkers, My sole intention is to only view events if they are fetched appropriately from UFs or not, as a part of testing after deploying any TA through DS. I don't need any clustering with indexers or distributed search head. I'm thinking to provision test server and install Splunk enterprise with personalized dev test license, and place it within prod servers network, for testing, debugging. Will this suffice my need, how reliable it would be to use personalized dev test license for such testing or development monitoring purpose? Will this decision be reliable for a longer tem, considering I renew license every 6 months? What can be other factors I need to understand. I'm trying to find alternative instead of extending our prod license, since we don't have large prod license. Any suggestions would be appreciated to help me to take my decision over this part

hectorvp · ‎10-04-2020

Are internal events compressed to 50% as it does for any normal events? For avg raw size of events in metrics.log is 150bytes So while storing it in indexer will it take approx 75 bytes??

hectorvp · ‎10-04-2020

Hi @isoutamo , I did the following to make rough estimate that is avgerae size of internal logs = 300 bytes average events per seconds by internal index with 1 UF = 10 total internal data at 1 day in MB by 1UF= 10 * 300 * 60 *60 *24 / (1000*1000) = 260 MB So for 1 day with 500UFs = 26 * 500 = 130000= 130GB in day Compression ratio of 50% so total 1 day data = 65GB so 60 days of retention capacity then = 65*60 = approx 4TB Am I going in right direction??? Or missing any factors?

hectorvp · ‎10-03-2020

thanks @richgalloway , Our technical add on would only be for testing purpose of simple configurations and won't ingest more than 50GB/day, Will it be ok if we use personal dev/test license??? We won't be having any full time development of add ons, basically it would be change requests to make simple conf changes, test it and push it in prod dev server. I'm having thought to keep test server on stand by mode and only use when such requests arrive. Does this dev license you mentioned in ur response, a paid license???

hectorvp · ‎10-03-2020

Hello Splunkers, Our current requirement is to only forward events from servers to customers indexers. We have decided to opt minimum of 2GB or 5GB of Splunk Enterprise license since we aren't storing any logs.i think this will suffice our need. Now we need to have a test server where we may have some testing when our technical add ons has been customizied, before deploying new add ons on prod deployment server. Which License do we need for test server??? Dev/Test license having 50GB capacity? But I read it is a personal dev/test... Or any other license? We are expecting to reside test server within the network where prod servers reside.This will help us not allwoing any events to travel out of network and security issues. What are general approachs by splunk professionals ? PS: We are setting up splunk env for the first time and new with splunk skillsets.

hectorvp · ‎10-03-2020

Hi Splunkers, We need to estimate the disk space required for our single box Splunk enterprise. We are planning to only ingest internal logs of splunkd and don't see any way, how can I estimate disk space for internal logs. Don't know how many events are generated from a UF and how much size a single event size would be. We would be having around 400 UFs running on servers and have expectancy of 60days of retention policy. I'm afraid if 500GB space will fill up before 60days and have wont have internal logs. Apart from this, Please suggest if I really need RAID levels 1+0 for internal logs, there would be few schedule search for health checkups and DMC. Or any other simple storage will suffice? Is there any way to estimate this part?

hectorvp · ‎10-01-2020

Hi @isoutamo , We are currently having 500UFs and no HFs and scope is to fetch only UFs internal logs. We are using single box for indexing,search head and as a DS, coz main purpose is to forward logs to 3rd party destn servers. So we are suppose to only store internal logs, need to plan how much disk space will be required, currently we opting with 1TB but I guess this capacity planning has to be revisited. We need atleast retention policy of 60days. Any suggestions on this, what could be the approach, we don't have actual access to 3rd party servers to check volume by using queries.

hectorvp · ‎10-01-2020

Just for a sake of knowledge, how much amount of _internal data is generated. Incase my daily indexing is of 6TB??? Will it 15% of 6TB? I know it doesn't consume my license...

hectorvp · ‎09-30-2020

Hi @richgalloway , We need to take a control over existing prod servers where UFs are installed by certain vendor, since vendor has been decommissioned, we are told to reinstall UF by uninstalling previous one(with new admin pwd and forwarder license), but since indexers are going to be same and nothing is going to be changed in indexers, there is possibility of getting events reindexed. Need to avoid it. Reusing same UF may cause forwarder license owning issue.

hectorvp · ‎09-30-2020

Hi Splunkers, I need to uninstall existing UF and re install new UF (sounds weird, but I have such case) without allowing to re index past data again? Is this possible? Need to run new UF as if it wasn't installed newly but is the same one, care of conf files can be taken.

hectorvp · ‎09-27-2020

Thanks @inventsekar I'm just trying to understand worst case; as mentioned in the blog link which is provided in your response it say if UF crashes then in-memory data is lost; so if UF crashes and queues were also completely full then 6MB (parsing queue default size)+ 500KB (output queue default size) = 6.5MB would be the maximum data lost right, that can ever happen? The blog but is only mentioning loss of 500KB in such situation

hectorvp · ‎09-27-2020

Thanks @gcusello , I've follow up question, in case you wish to answer. https://community.splunk.com/t5/Monitoring-Splunk/Use-of-persistent-queue-in-UF/m-p/521654#M4491

hectorvp · ‎09-27-2020

In which situation the persistent queue would be used in UF, only if indexer is slow in writing or is down for a long time? I mean it won't have to do in any cases to drop events in UF, if UF is crashing??? If UF is crashed and when we restart it would start resending events in where he had left off files reading and in-memory queue will be written to persistent queue or would it read again from events source?

hectorvp · ‎09-27-2020

Thanks @richgalloway , UF generally pushes logs using TCP connection, so it gives reliability transportation layer, adding indexer ACK would add reliability at application layer right? Since you haven't mentioned any such scenario, I'm considering there are no such known issues where events are dropped.

hectorvp · ‎09-27-2020

Hi @gcusello , I need to know if in any unwanted situations UF wasn't able to deliver events to indexers. For ex: If UF crashes , the data in memory queue i.e parsing queue and output queue is lost. Need to find if there is anyway to find number of events dropped. I saw somewhere that there is some internal log where it says "splunkd had begun drop event" (maybe this log would be coming up if syslog is used) Just thought if we can calculate something by using metrics.log, but didn't find anything concrete yet.

hectorvp · ‎09-27-2020

Can anyone enlist possible reasons when UF may drop events? There may be many situations, but some known reasons came from practical experiences...

hectorvp · ‎09-27-2020

Incase indexer is down or has slow speed for writing events in a disk, I guess in these cases UFs parsing queue and output queue would be full enough and considering dropEventsOnQueueFull = -1 Suppose indexer was up again upon next day, From where would UF start events, from where he had left off reading files or would drop events??? Can I consider that in metrics.log if group=queue and blocked=true, the IF is blocked and may drop events Or will it send all logs when queue has space or indexer is up and running, where he had left off. I don't have option of persistent queue bcoz I may monitor some logs files using monitor stanza

hectorvp · ‎09-27-2020

Is there any way to find how many events were dropped by UF in a day? Need a daily report to find how may events were dropped by UF. Can I know number of events dropped? OR xyz MB of events are dropped?

hectorvp · ‎09-26-2020

@renems , were you able to find any root cause with this???

hectorvp · ‎09-26-2020

@richgalloway Then if UF crashes and we restore it by some means and again UF is up and running ,although UF dropped events which was read but then from where he would start reading new events from the file?? Again would UF start from where he had left off??

hectorvp · ‎09-25-2020

If I gracefully shutdown the UF, it will send all logs from output queue and from internal parsing queue. Suppose I restart the UF after 1min, will it start sending logs from logs file where he had left before shutdown??? Or will it start sending new logs which are getting appended independent of where had left off. If in such scenarios logs are getting dropped, is there any way to detect how many such logs were dropped? What may happen if UF is crashed, obviously it will drop queue logs but from where he would start once he is up and running??

hectorvp · ‎09-25-2020

Thanks @gcusello,

Posts	80
Solutions	2
Karma Given	42
Karma Received	13
Member Since	‎07-11-2020

Online Status	Offline
Date Last Visited	‎09-23-2021 05:23 AM

Sufficiency of Hardware Specs at indexer layer

Indexer on RAID 0

Single Site or Multi Site Indexer Clustering

Filtering logs at HF to get a single event of each...

Events Source Validation

Does Splunk DB Connect need UF?

Splunk DB Connect: Logs forwarding

Splunk Add-On For MS SQL Server

Getting NetScaler Data In

Disk configuration for indexer

Re: What is average Internal Events count per day ...

What is average Internal Events count per day on a...

Re: Estimating Storage requirement when only inter...

Personalized dev/test license as a pre-production

Are internal events compressed ??

Re: Internal Index volume

Re: License for non prod server

License for non prod server

Estimating Storage requirement when only internal ...

Re: Internal Index volume

Internal Index volume

Re: Reinstall Splunk Universal Forwarder

Reinstall Splunk Universal Forwarder

Re: Use of persistent queue in UF

Re: How to find events dropped

Use of persistent queue in UF

Re: Dropping events possible reasons?

Re: How to find events dropped

Dropping events possible reasons?

UF queues when Indexer down.

How to find events dropped

Re: forwarder dropping events

Re: UFs new pointer after restart

UFs new pointer after restart

Re: Deployment Server

Join the Conversation