About lisaac

lisaac · ‎01-20-2017

Thanks for this last example. This is what I was looking for in a separate search.

lisaac · ‎01-08-2017

That is a good question. The goal is to have Splunk "auto" discover a sourcetype based on a known list. The Splunk UI doesn't have this capability yet, so I have been trying to find methods for automation. I may have missed this in my first post, but I hope this update helps to solidify my first question. I would like for Splunk to auto discover the sourcetype based on a known list. I can then setup a response to a customer or an admin, based on the results of a Splunk automated sourcetype recognition of a "monitor" input.

lisaac · ‎01-08-2017

Splunk has a list of pretrained sourcetypes (http://docs.splunk.com/Documentation/Splunk/latest/Data/Listofpretrainedsourcetypes). I have installed a new Splunk Supported Add-on called the Splunk Add-on for Tomcat. I am using Splunk Enterprise version 6.5.1. The list of pre-trained sourcetypes includes a sourcetype of catalina. The description for the sourcetype shows "Output produced by Apache Tomcat Catalina (System.out and System.err)" The Splunk Add-On for Tomcat (http://docs.splunk.com/Documentation/AddOns/released/Tomcat/Sourcetypes) has the following sources identified as sourcetype tomcat:runtime:log: Catalina.log, localhost.log, manager.log, host-manager.log I have a file called /tmp/catalina.log, and I would like Splunk to automatically sourcetype the value as tomcat:runtime:log. In testing, I have a dev splunk instance with the Splunk Add-On for tomcat installed. I have a monitor statement as follows for testing: [monitor:///tmp/catalina3.log] index = main Attempts to auto learn the sourcetype to a value of tomcat:runtime:log fail. Splunk will always try and set the sourcetype to a value of catalina-#. A # shows the number as an incremental # assigned to the sourcetype. In tests, I used data sets with 200 lines or 2000 lines. I supposed that I could setup a rule that would check the file name and classify accordingly, but I figured that a Splunk supported Add-On would update the list of pretrained sourcetypes. Is this a feature add that splunk could add in the future? Is it possible to easily update the list of pretrained sourcetypes?

lisaac · ‎12-28-2016

Splunk has an option of a disk based persistent queue on a TCP input. The option is not available for splunktcp inputs. I have 10-20K hosts with a UF, and the UFs send into an HF layer (17 hosts running 2 HFs each). We do this to route data to hadoop, splunk enterprise, or the null bucket. The splunk version is 6.4.4 for the HFs. I was thinking of updating my UFs to not send CookedData into the HF layer and change the Splunktcp inputs to TCP inputs with disk based persistent queues. Here is the setting from the outputs.conf.spec file: sendCookedData = [true|false] * If true, events are cooked (have been processed by Splunk). * If false, events are raw and untouched prior to sending. * Set to false if you are sending to a third-party system. * Defaults to true. The goal is to gain the benefit of Persistent queuing to store data in an input queue to disk. This can help prevent data loss if the forwarder or indexer gets backed up. Does anyone have any thoughts on this item? There will be extra overhead on the HF layer due to uncooked data. I would overcome this by using a combination of Splunktcp and tcp inputs. TCP inputs would handle any data item that would need an extra level of data loss prevention. I have not tried increasing the input queue memory size. This could be an alternate approach to disk based queues.

lisaac · ‎11-09-2016

Splunk has to fit into a structured and unstructured world without duplicating effort. Splunk currently serves as a standard log collection mechanism, but I would like to see Splunk support the feeding of additional third party solutions. To this end, I would like to know if the following is supported or if anyone has tried these options: 1) Roll coldToFrozen AND warmToCold buckets into hadoop and keep a copy indexed into Splunk. This data would need to be searchable via Splunk Enterprise and hadoop utilities like pig or hive. 2) I would like Splunk, in the form of a UF or an HEC, collect data, and I would like for the data to be routable to Kafka in addition to Splunk Enterprise. I was thinking of having an HF layer, which already exists at my company, route to Fluentd via Splunk's tcp or UDP out. Fluentd could then route to Kafka. The other option is to remove the UF or an HEC and replace with Fluentd. Fluentd has community developed plug-ins to send to Kafka, HDFS, and Splunk Enterprise. I have not tested this solution. Any thougths from anyone?

lisaac · ‎07-08-2016

Here is what i arrived at based on input from my Remedy team. I listed out the original Nix example along with a suggested MS-Windows example: [monitor:///opt/remedy_data/user/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\?] “I don’t know what files this might map too? Any thoughts on this one?” disabled = false sourcetype = remedy_user [monitor:///opt/remedy_data/api/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\arapi.log] disabled = false sourcetype = remedy_api [monitor:///opt/remedy_data/ar-error/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\arerror.log] disabled = false sourcetype = remedy_arerror [monitor:///opt/remedy_data/filter/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\arfilter.log] Do you think this might map to the file CAIFilterPluging.log as well? disabled = false sourcetype = remedy_filter [monitor:///opt/remedy_data/escalation/] “I don’t know what files this might map too? Any thoughts on this one?” disabled = false sourcetype = remedy_escalation [monitor:///opt/remedy_data/approval/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\Approval*.log] disabled = true sourcetype = remedy_approval [monitor:///opt/remedy_data/ar-monitor/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\armonitor.log] disabled = true sourcetype = remedy_armonitor [monitor:///opt/remedy_data/arplugin/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\ar*plugin.log] covers arjavaplugin.log and arftsplugin.log covers arftsplugin-std*.log & arjavaplugin-std*.log disabled = true sourcetype = remedy_arplugin [monitor:///opt/remedy_data/cmdb/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db*cmdb*.log] covers arcmdb.log and cmdbengdebug.log disabled = true sourcetype = remedy_cmdb [monitor:///opt/remedy_data/fulltextsearch/] “I don’t know what files this might map too? Any thoughts on this one?” disabled = true sourcetype = remedy_fulltextsrh [monitor:///opt/remedy_data/mid-tier/] [monitor:/// mware\vfabric-tc-server-standard-2.6.5.RELEASE\BMC_MT_64Bit_V7\logs\armmidtier*.log] disabled = true sourcetype = remedy_midtier [monitor:///opt/remedy_data/servergroup/] “I don’t know what files this might map too? Any thoughts on this one?” disabled = true sourcetype = remedy_servergroup [monitor:///opt/remedy_data/sql/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\arsql.log] disabled = true sourcetype = remedy_sql [monitor:///opt/remedy_data/thread/] [monitor:/// E:\Program Files\BMC Software\ARSystem\Arserver\Db\arthread_BMC.log] disabled = true sourcetype = remedy_thread [monitor:///opt/remedy_data/tomcat/] [monitor:/// mware\vfabric-tc-server-standard-2.6.5.RELEASE\BMC_MT_64Bit_V7\logs\catalina*.log] disabled = true sourcetype = apache_tomcat

lisaac · ‎07-08-2016

Is there a sample TA inputs.conf for this application running on an MS-Windows host? Has anyone run this application on a Windows host?

lisaac · ‎04-07-2016

I have a monitor that that isn't working. I turned debug on in log.cfg, and the Universal Forwarder reports no match on whitelist. The following has been tried: [monitor://E:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\LogFiles] disabled = false index = app_ops_prod whitelist=ReportServerService*.log sourcetype = mssql:ilink:rptsvrsvc ignoreOlderThan = 3d OR [monitor://E:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\LogFiles\ReportServerService*.log] disabled = false index = app_ops_prod sourcetype = mssql:ilink:rptsvrsvc ignoreOlderThan = 3d Splunkd.log says that is matches and then skips. As noted above, I tried 2 configurations. 04-07-2016 17:34:20.348 +0000 DEBUG TailingProcessor - Item '' matches stanza: /E:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\LogFiles. 04-07-2016 17:34:20.348 +0000 DEBUG TailingProcessor - Not using stanza for this item (File did not match whitelist 'ReportServerService*.log'.). The file is E:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\LogFiles\ReportServerService__test.log . The host is Windows Server 2012 r2, and the UF is at verison 6.2.6.

lisaac · ‎03-03-2016

I am working on automating enrollments for a deployment server. I am working with a CMDB that has 25K hosts with over 1k applications. Each host in the CMDB is mapped to an application. For now, assume there is a one to one relationship of host to application. Each application has a unique 4 digit code, and the code isn't reused. I plan to include the 4 digit code in the name of a forwarder app and in the name of the server class. The serverclass whitelist will include hosts mapped to the application. As you could imagine, the serverclass.conf could very easily grow to 25-50K lines. Does anyone know if the size of the serverclass.conf file will be an issue? Does anyone in the field have similiar sized serverclass.conf file? Our environment consists of 5-8 deployment servers. We have a single primary deployment server, and we synchronize secondary deployment servers from the primary with rsync. The splunk version is 6.3.

lisaac · ‎03-02-2016

I have 5-7 shared search heads that are not synced with configuration items. I am migrating all search heads to a shared search head cluster. Does anyone have any best practices or recommendations on how to best consolidate 5-7 shared search heads into a new SH cluster? At this point, we are looking to migrate user created apps and public artifacts.

lisaac · ‎12-22-2015

Thank you very much for the thought process on the reply. All events are forwarded to a pool of indexers. The License Master SH has a distributed search into each of the indexers. It turns out that 1 of the 5 was disabled. My search on the license master now matches the license dashboard on the license master. I enabled the distributed search to the 5th indexer. My search now shows as follows: Search Total=2452GB License Portal=2289GB Difference: 163GB+ I am now thinking why the difference the other way.

lisaac · ‎12-21-2015

We have a centralized license server running on a host called sl55splunk1. The result of the following search doesn't match the license portal: index=_internal source=*license_usage.log host=sl55splunk1* type=Usage | timechart span=1d sum(b) as total | eval total=total/1024/1024/1024 Search Total=1979GB License Portal=2289GB Difference: 309GB Which value is correct? I would expect the two values to match up. I have checked dates from November 25 through Dec. 20, and I see a daily variation that ranges anywhere from 59GB to 433GB. The Splunk Enterprise version is 6.2.6.

lisaac · ‎12-01-2015

This solution here worked link text

lisaac · ‎11-24-2015

I like this app! Is this app a wrapper for the same data preview option that is available in data on-boarding?

lisaac · ‎11-10-2015

This is good information. Do you know of 6.2.x or 6.3.x supports persistent output queues to disk vs. using more memory?

lisaac · ‎11-10-2015

The outputs.conf.spec shows a default value of "auto". The Splunk Universal Forwarder version is 6.2.3 on RHEL 6.6. What is the algorithm used to determine the amount of memory to use? I have OS personnel asking what is the possible memory maximum for the agent.

lisaac · ‎11-06-2015

Will a flat file gateway be sufficient to keep up with a busy tiered Omnibus architecure? Is there an alternate to the flat file gateway for the app?

lisaac · ‎11-05-2015

The documentation shows that a Netcool flat file gateway is needed. I am assuming the netcool flat file gateway will produce a file that will be ingested by the forwarder. How does Splunk know to automatically the schema changes for alerts.status?

lisaac · ‎11-02-2015

I have a distributed non-clustered Splunk Enterprise environment. I am planning to implement Search Head clustering and a multisite indexer cluster. I know that the current Nix and Windows Infra apps haven't been certified yet by Splunk in a SH cluster, and there could be limited support from Splunk in the event of an issue. I am sure there are users that have deployed the Apps in a SH cluster. Are there any caveats that people have seen on the deployment of the current shipping applications in a Splunk 6.3 SH cluster?

lisaac · ‎10-28-2015

I suppose that I could run 2 UFs on the same host, but I would prefer to skip this approach.

lisaac · ‎10-28-2015

I am batching gzipped files into Splunk. The files reside in 4 directories. Splunk, per splunkd.log, appears to be reading only the files in the first batch statement. Is there a way to run parallel batches? I have a Linux 64 bit Universal Forwarder with Splunk 6.2.6. I have set maxKBPS to 0 in limits.conf, and I have reniced the Splunk UF prdata2sses to a priority of -20 on the Linux VM. I have batch statements listed as follows: [batch:///leroylogs2/multicast/archive/data11/*PROD*.gz] [batch:///leroylogs2/multicast/archive/data12/*PROD*.gz] [batch:///leroylogs2/multicast/archive/data21/*PROD*.gz] [batch:///leroylogs2/multicast/archive/data22/*PROD*.gz] [batch:///leroylogs2/multicast/archive/data11/*CAS*.gz] [batch:///leroylogs2/multicast/archive/data12/*CAS*.gz] [batch:///leroylogs2/multicast/archive/data21/*CAS*.gz] [batch:///leroylogs2/multicast/archive/data22/*CAS*.gz] 10-28-2015 09:37:37.372 -0700 INFO ArchiveProcessor - Finished processing file '/leroylogs2/multicast/archive/data11/2015-08-29-08_30-PRODtrans_svs.log.gz', removing from stats 10-28-2015 09:37:37.433 -0700 INFO ArchiveProcessor - handling file=/leroylogs2/multicast/archive/data11/2015-08-29-07_10-CAStrans_svs.log.gz 10-28-2015 09:37:37.434 -0700 INFO ArchiveProcessor - reading path=/leroylogs2/multicast/archive/data11/2015-08-29-07_10-CAStrans_svs.log.gz (seek=0 len=32496625) 10-28-2015 09:37:37.551 -0700 WARN TcpOutputProc - The event is missing source information. Event : 10-28-2015 09:37:38.655 -0700 ERROR ArchiveContext - From archive='/leroylogs2/multicast/archive/data11/2015-08-29-07_10-CAStrans_svs.log.gz': gzip: stdout: Broken pipe

lisaac · ‎10-12-2015

My environment has just patched the Splunk for VMWare app to 3.1.4. I have a new physical DCN with 32 cores. Is it possible to configure more than 8 worker processes for this one DCN, of 8, with a higher number of worker processes? The other virtualized DCNs do not have 32 cores. The GUI in the splunk for vmware application doesn't allow for more than 8 worker processes.

lisaac · ‎10-02-2015

I have a very busy search head that complains : DistributedPeerManager - Unable to distribute to peer named slxxxxxxxxx:9089 at uri https://xxxxxxxx037:9089 because peer has status = "Down" The messages will start in splunkd.log at 22:08:10.971 and finish at 22:09:46.994, but the message is reported about 60 times during short time period. A telnet from the SH to the indexer on 9089 shows no connectivity issues. This has happened off and on for all indexers configured in distributed search. I am wondering if there is a setting that could be adjusted that to prevent these messages from occurring, or if there is a conf value that could be adjust to improve performance under high load. The SH is 10vpcus by 32gig, and there is a high load average on the SH and indexers (lots of searches). There appears to be no negative impact to the messages, since searches are working. Users are not reporting any issues.

lisaac · ‎09-13-2015

I have setup a 6.2.5 SH cluster. The SH cluster consists of 3 SHs and an additional host functioning as a SH deployer. I have successfully configured the SH cluster to communicate with an index cluster. What is the best way to add a single non-clustered indexer to a SH cluster? It looks like I would need to use the CLI and run the command splunk add search-server on each of the SH cluster members. Any thoughts from anyone?

lisaac · ‎09-11-2015

Does Splunk for PAN App work on a search head cluster ?

Posts	53
Solutions	0
Karma Given	0
Karma Received	15
Member Since	‎04-07-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

Add a sourcetype from a Splunk Supported Add-On to...

Persistent queues tcp

Using Splunk as a enterprise data platform

Is there a sample TA inputs.conf for this applicat...

Why is my universal forwarder on Windows server 20...

Splunk Deployment Server: What is the largest poss...

Best Practices Migrating from shared search heads ...

Why doesn't the license_usage.log summed value mat...

Is the Preview app a wrapper for the same data pre...

Splunk 6.2.3 Universal Forwarder maxQueueSize: Wha...

Re: How do I edit my if else case eval statement t...

Re: Add a sourcetype from a Splunk Supported Add-O...

Add a sourcetype from a Splunk Supported Add-On to...

Persistent queues tcp

Using Splunk as a enterprise data platform

Re: Is there a sample TA inputs.conf for this appl...

Is there a sample TA inputs.conf for this applicat...

Why is my universal forwarder on Windows server 20...

Splunk Deployment Server: What is the largest poss...

Best Practices Migrating from shared search heads ...

Re: license_usage.log summed value doesn't match l...

Why doesn't the license_usage.log summed value mat...

Re: Duplicate labels causing conflict

Is the Preview app a wrapper for the same data pre...

Re: Splunk 6.2.3 Universal Forwarder maxQueueSize:...

Splunk 6.2.3 Universal Forwarder maxQueueSize: Wha...

IBM Tivoli flat file gateway question

Splunk for Tivoli Netcool: Is the Netcool Flat Fil...

Are there issues with deploying the Splunk App for...

Re: Batching gzipped files residing in 4 directori...

Batching gzipped files residing in 4 directories i...

Splunk App for VMWare DCN - Greater than 8 worker ...

Why is the splunkd.log reporting lots of "Distribu...

Configure Search Head Cluster Peers to search a si...

PAN App on Search Head cluster