About hectorvp

hectorvp · ‎12-10-2020

Hello Splunkers, We may have around 400UFs forwarding 1GB of events each UF, with total estimated daily ingestion of 400GB each day. Our main aim is to forward these events to the customer's indexers(customer also has indexer cluster), however with the strong requirement from the customer to provide us a validation for logs we need to store it at our indexer as well. (ok with more license consumption) We have decided to use indexer cluster with 2 indexers for us as well. So our indexer cluster will be performing dual role (storing event + forwarding (with anonymizing names in events)) using indexAndForward configuration. We are not allowed to use HF in between as customer sees this in a way that different version of same event (edited version for us & different edit for them) may be received. Or wont allow to send same event directly from UF to their indexer + our indexer. Our indexer has specs as 32GB RAM, 24 vCPUs & xxTBs (RAID 10). Every thing is in a single data centre. We have mainframe logs requirement in future as well. Will this suffice our need? Some one said me that if I'm sending events from UFs directly indexers, it will open multiple queues at indexers and will hamper performance to a greater extent, is that true ???(I don't believe so) Can one let me know how to estimate how many indexers are required based on daily ingestion capacity considering RF2 & SF2 OR RF1 & SF1?? I've attached a diagram for better understanding.

hectorvp · ‎12-10-2020

Thanks @gcusello , we've decided to keep all on different boxes.

hectorvp · ‎12-01-2020

Thanks @gcusello , Understood with this part, I've another thought to save budget on infra would be clubbing DS and search head on single box, since we dont have much aggressive searching, hardly 2 users so this may work. And currently DS will support 400 UFs and future upto 1000.

hectorvp · ‎12-01-2020

Hi Splunkers, Our customer is demanding to reduce cost on infra with storage aspects on indexers. Initially we had provided RAID 10 configurations (as per Splunk recommendations), but ,now we are moving forward with RAID 0 configuration as it will halve our storage requirement and will provide good IOPS atleast. Is anyone using RAID 0 on prods for indexers? I managed few with reducing on data retention policy....we have estimated 400 GB of daily ingestion by 400UFs. Indexer cluster with 2 indexers. And 1 search head....we may not have more than 2 users on search head and limit amount of searching (mostly scheduled searches? Infra service provider are providing some integrated disk backup to the indexers, this where I will currently be investigating about what exactly are they using as disk backup. I doubt if we can request to provide RAID 10 for hot and warm and RAID0 of cold... this is what again worrying me right now. But can anyone let me know if they are using successfully RAID 0 for indexers on prod?

hectorvp · ‎11-26-2020

Thanks @richgalloway Yes, these are other complexities as well as we are going to route these events to other third party indexers

hectorvp · ‎11-26-2020

Thanks @richgalloway , So we would opt for multi site configuration. Where 1 DC will contain 1 indexer and master node and other DC will contain another 1 indexer. Well when each indexer should be capable of 100% ingestion means we need to size its disk space to capable of storing all events.Along with other specs recommended by Splunk. And having replication factor to 2, did I understood this in a right way? Currently we have estimated that total events generated by DCs would be around 400GB per day.

hectorvp · ‎11-26-2020

Hi Splunkers, We have servers running on 2 different on premises Data Centre. We are planning Splunk deployment architecture to fetch the events from server. We are planning to have 1 indexer at each Data Centre and making a clustering of this 2 indexers. Reason for clustering is suppose 1 indexer goes down then the other one can keep on fetching events from the servers of other DC with no downtime with these aspects. And clustering will also help us to use 1 search head for both DCs as we dont have many users for monitoring. Both Data centres are connected to each other with some dedicated LAN channels.... Both indexers would be under same virtual network. Will this be as simple as single site cluster??? If I configure them as single site cluster would there be any architectural issues or I'm going in right way? Or will their be a concept multi site needs to be implemented?

hectorvp · ‎11-04-2020

Thanks @inventsekar

hectorvp · ‎11-04-2020

Thanks @richgalloway , this will help But just need ur thought over this aspect, group=per_source_thruput will provide me top 10 busiest sources at every 30s. Now suppose I have more than 10 sources for example 22 sources per host then it won't show up thruput for all 22 sources. In the above case I need to increase maxseries in limits.conf file to 25 , I mean maxseries should be always greater than number of sources. In this way the validation of event sources may get more reliable. Am I right with this thought?

hectorvp · ‎11-04-2020

Hi @richgalloway , Yes we are fetching internal logs of UF. But suppose in windows app logs if multiple sources are logging for ex: app A, app B and app C. Can we validate that app A just sent the log to customer indexer from internal index? Need to validate this at per day level that is atleast a single event was send from appA in a day, if not then it fire an alert. Will this be possible from metrics.log? I read it send info only regarding top 10 source types.

hectorvp · ‎11-04-2020

Hello Splunkers, I need to filter logs at HF to send only single log from each source from every host once in a day to the indexer A. And all the logs will be forwarded to indexer B where indexer B is a customer indexers. And hence we won't have access to indexer B. Indexer A is what we own, we need to use it for logs validation whether any log of certain appn is showing up or not at every day. This is part of our logs generating validation that we are asked by our customers.

hectorvp · ‎11-03-2020

Hi @gcusello , First of all thanks for the suggested approaches. Apologies, but it would be a great help if you can explain the 2nd approach in detail, I mean how can I configure HF to send single event of a particular source per host to our indexer in a day + all the events to customer indexers and then we can have validation to logs we are sending. Question is raised on community with the link, https://community.splunk.com/t5/Monitoring-Splunk/Forward-single-event-from-HF-to-Indexer/td-p/527781

hectorvp · ‎11-03-2020

Hi @gcusello , Our main aim is to forward events from our servers to customers indexers. Since we don't have access to the customer's indexers, we are not able to validate that are we sending all appn logs using above approach. Is all source sending events to customer indexers can be identified with _internal index?? I saw metrics.log only shows top 10 results for source type. We are storing _internal events of UFs in our single indexer for health checkup of UFs. Can we validate within internal events? Or what if we add an HF in between, which will see the source log for the first time in a day and store it in our indexer and then stops sending logs to our indexer and will forward only to the customer indexers, so our indexer will have only 1 log per source per server and won't replicate whole logs and make unnecessarily 2x of cost for license. Further on search head will use above approach the ony mentioned by you to raise an alert?? Is this possible on HF?

hectorvp · ‎11-03-2020

Hello Splunkers, I've been in some weird requirement/situation, which is, we need to validate if events of particular source and sourcetype are getting forwarded by UF or not. For Ex: Our application XYZ is logging events in windows event logging system we need to make sure that events are getting forwarded by UF to the indexers for particular source that is "XYZ". & application registers the events in windows event logging system very rarely say once in 7 days. Can we validate this using only UFs internal logs? Yes we have this big restriction of validating above situation only with UFs internal logs, we simple cannot query for the source and check its earliest coz we dont have access to indexers containing actual logs. I checked with metrics.log but it will tell only about top 10 sources of events, in that case if we have 35 sources on single server, I guess we won't be able to do it. Can anyone help me with this?

hectorvp · ‎10-30-2020

Thanks @richgalloway , this would help us.

hectorvp · ‎10-30-2020

Thanks @gcusello , By the way we are not trying to fetch MS SQL audit logs to the DS. We need to forward these audit logs to the customers indexers from our MS SQL servers. Till now I came to know having Splunk DB connect on DS won't help us to redirect audit logs to the customer indexers, it will fetch logs into DS which we don't want. We don't have HF in our control as well. The 2nd approach you mentioned is where I'm looking forward currently. Only instead of dumping audit logs into a file with some store procedure we would dump it into standard windows event application logs and UF will forward it. Thus is what we are anticipating. However just was going through community and found a issue mentioned with this approach, which yet personally have not experienced by me so just need to confirm on same. Below is the link of an issue https://community.splunk.com/t5/Splunk-Enterprise-Security/Monitoring-MS-SQL-logs-from-Windows-Event-viewer/m-p/439031 If this is the case we may need to ask customer to either have CIM modelling for sent logs or then at last need to ask them to fetch logs using Splunk DB connect by themselves.

hectorvp · ‎10-30-2020

Thanks a lot @gcusello

hectorvp · ‎10-30-2020

Hi @gcusello , No worries, you always help us a lot. Just one aspect with my above issue, if you can check the community link I'd pasted above they say there is some parsing field issue for forwarding these logs by the approach which we are looking forward. Someone has mentioned about CIM data modelling, this is the new concept for me today. As per my quick reading I found this is used to help Splunk to understand and identify logs with its fields. But just this needs to know where this CIm modelling will be used ahead, I mean in UF or HF/indexer or both??? And how reliable it is ??

hectorvp · ‎10-30-2020

Hi Splunkers, I read a little and came to know Splunk DB Connect uses JDBC driver to fetch events. It means Indexer or HF would be requesting these events from table at certain interval. Does Splunk DB Connect need UF??

hectorvp · ‎10-29-2020

Hi Splunkers, Can we install Splunk DB connect on deployment server to forward MS SQL audit logs to the indexers?? Or is there any alternative way to send MS SQL audit logs without using Splunk DB Connect?

hectorvp · ‎10-29-2020

Hi @hughkelley , How did you resolve this issue, any idea?

hectorvp · ‎10-29-2020

Hello Splunkers, We need to fetch MS SQL logs into our Splunk. We aren't allowed to use Splunk DB Connect for some security reasons. Currently what we are deciding is to log all MS SQL audit events, trace events in Standard Windows Application events logs. And fetch logs from Windows Application Logs by UF. Will this "Splunk Add-On For MS SQL Server" helpful for us in any way now? I guess this add on has dependency on "Splunk DB Connect". I'm afraid with one more aspect that is will all fields of MS SQL audit events be parsed???? Please check this community link, regarding parsing issue https://community.splunk.com/t5/Splunk-Enterprise-Security/Monitoring-MS-SQL-logs-from-Windows-Event-viewer/m-p/439031

hectorvp · ‎10-28-2020

Hi @richgalloway , Citrix Netscaler is getting enabled to send events using UDP. We would hardly have 5 Netscaler appliances, setting up dedicated syslog server which would store events in file and monitoring those files by UF is what I understand a recommended way. Just setting up syslos server for 5 appliances is what we would be concerned. Rest are all servers events where UF would be forwarding using TCP. Is their any alternative way where we can send events (UDP) directly to HF? Or is can this be done to install rsyslog and HF on same box to achieve it???

hectorvp · ‎10-28-2020

Hello Splunkers, We need to fetch events from Netscaler devices. After investigation I found that Netscaler can be configured to send events in syslogs format. I had read somewhere inorder to fetch such events it is better that we use one syslog server and install UF to monitor and send events written in syslog server. However I saw their is an add-on named as "Splunk add-on for Netscaler Citrix", if we use this add on at our HF or indexer and Search Head, and send events directly from Netscaler device to indexer/HF, will it work effectively and will it be reliable? Or do we need syslog server in any case (Installing UF on top of it on syslog server)??? And if we send events directly from Netscaler to Indexer, I guess we may not need UF anywhere (Does add on helps us here in any way)?? What should be approach here? In documentation I saw we can install add on on UF as well but what is it for??? PS: I'm new with Netscaler aspects and quiet beginner with Splunk as well.

hectorvp · ‎10-18-2020

Hi splunkers, We have integrated disk backup by default when we procure disks in server. For our indexers, shall we go for Raid10 but this will double our disk space? Or is it fine to have only Raid0 as we have integrated disk back up in place?

Posts	80
Solutions	2
Karma Given	42
Karma Received	12
Member Since	‎07-11-2020

Online Status	Offline
Date Last Visited	‎09-23-2021 05:23 AM

Sufficiency of Hardware Specs at indexer layer

Indexer on RAID 0

Single Site or Multi Site Indexer Clustering

Filtering logs at HF to get a single event of each...

Events Source Validation

Does Splunk DB Connect need UF?

Splunk DB Connect: Logs forwarding

Splunk Add-On For MS SQL Server

Getting NetScaler Data In

Disk configuration for indexer

Sufficiency of Hardware Specs at indexer layer

Re: Indexer on RAID 0

Re: Indexer on RAID 0

Indexer on RAID 0

Re: Single Site or Multi Site Indexer Clustering

Re: Single Site or Multi Site Indexer Clustering

Single Site or Multi Site Indexer Clustering

Re: Filtering logs at HF to get a single event of ...

Re: Filtering logs at HF to get a single event of ...

Re: Filtering logs at HF to get a single event of ...

Filtering logs at HF to get a single event of each...

Re: Events Source Validation

Re: Events Source Validation

Events Source Validation

Re: Splunk DB Connect: Logs forwarding

Re: Splunk DB Connect: Logs forwarding

Re: Splunk Add-On For MS SQL Server

Re: Splunk Add-On For MS SQL Server

Does Splunk DB Connect need UF?

Splunk DB Connect: Logs forwarding

Re: Monitoring MS SQL logs from Windows Event view...

Splunk Add-On For MS SQL Server

Re: Getting NetScaler Data In

Getting NetScaler Data In

Disk configuration for indexer