About tlmayes

tlmayes · ‎08-04-2021

I have an index1/source1/sourcetype1 of events that is several "million" records each day. I have a second index1/source1/sourcetype2 that is several hundred records each day Several times a day I must execute a JOIN command to associate (1) sourcetype1 field with (1) sourcetype2 field, with each run of the query covering the last 2 weeks. The associations between query1 and query2 change or are updated with each run. The output is not static (changes with each run), which means the output of the last query is no longer valid since the data in query2 changes. Is there a better way to address this? KB or Lookup won't work since the output of query2 changes the outcome, and saving the output of query1 is not practical (millions of events) index=index1 sourcetype=sourcetype1 field=common | join common [ search index=index1 sourcetype=sourcetype2 field=common field=changing] | table common, changing, field3, field4, field5, ......

tlmayes · ‎08-07-2018

Agree regarding use of a different partition, but is an inherited design so have to deal with what I have (for now). Primary is @ /opt/splunk/var/lib/splunk and secondary separate. Very good feedback, which investigation shows this is the right track. Checked btool as suggested. Shows "../system/default/indexes.conf tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary" . Since there is no setting for volume _splunk_summaries assuming at this point that it is defaulting to the primary volume. Since summary is defaulting to the primary volume is it (_splunk_summary) size controlled by the "maxVolumeDataSizeMB" setting for primary or does it require its own setting?

tlmayes · ‎08-02-2018

[volume:primary] path = /opt/splunk/var/lib/splunk maxVolumeDataSizeMB = 630001 (is 64,512 bytes) /opt/splunk/var/lib]# du -cks * | sort -rn|head 661052116 total 661052116 splunk df -h Filesystem Size Used Avail Use% Mounted on ............... .... ... ... ... ........ /dev/mapper/vg01-opt_splunk 700G 673G 28G 97% /opt/splunk To your question about accelerated datamodels so checked indexes.conf on the SH's and sure enough it is set to the same path, but a different value (740G). Since it is on the SH vice indexer, does it really have an effect? Built by others so am having to reverse engineer.

tlmayes · ‎08-01-2018

Primary volume set to 650GB. MC reports that primary volume is 615/650, so all is good... But the volume consumption as reported by "df -h" reports 710GB being used, and nothing is stored in the path but the primary volume (indexes). What am I missing? What is using the extra 60GB that is not being recognized by Splunk?

tlmayes · ‎07-03-2018

Thanks for the response Starcher. I have no ES background, but am responsible for the core architecture. ES was deployed by Splunk PS and is managed by an MSS. I asked this questions several times of PS, and the answer was always the same: ADMIN is required for the MSS, which contradicts the documentation. The documentation as you point out does indicate that ADMIN is not required. I am more interested in what others are doing in practice (what works). What you you? Do you use the roles effectively as the document indicates? Without having to provide other than a core administrative function of the Splunk ES (same as you would on any Splunk SH?)?

tlmayes · ‎06-29-2018

We have a growing Splunk environment with one ES SH, and a SH cluster. We have an MSS that is going to manage our ES server as part of the managed SOC, we manage/administer everything else internally. I understand the ES "best practices" dictate that for ES to be properly managed admin access is required. The problem is that this gives the managed SOC complete access to 100% of our data, including indexes that have nothing to do with ES. What have others done to overcome this requirement of admin access, yet still allow ES and those that manage ES to work properly?? Or is there no alternative that works?

tlmayes · ‎06-07-2018

Our Splunk architecture is using SAML. Users are randomly added to our SAML mapped groups without authorization (too many cooks in the Azure kitchen). Has anybody found a way to control SAML accounts @ the Splunk level? Or is this exclusively out of the control of Splunk?

tlmayes · ‎05-18-2018

Question about powershell, and a DEFAULT installation of a Universal Forwarder on a Windows Server. In this instance, does Splunk use powershell for any process, assuming no UF configuration other than collecting Windows Events.

tlmayes · ‎02-14-2018

To jtrucks, interested in whether or not you pursued this. II'm working in an environment with ~20k (and growing) UF's and a range of geographies and sub-geographies requiring many white/black lists and apps. Deployment Manager GUI is no longer useful so moving to CLI. The downer is as discussed here is a better way to manage this file and its relationships (Apps & white/blacklists). The use of lookup table is very interesting, but mostly interested in what you and others may have found as a "best" solution.

tlmayes · ‎01-22-2018

No. Load balancer in front of SH cluster only. Your suggestion of installing Apache locally is a great idea and will investigate. Sometimes the forest gets in the way of the trees, but for simplicity purposes would be ideal if a single proxy (maybe one dedicated to management servers) could host all management servers. Will investigate as well as an option. Additional challenge as I work through this. The answer may be in the deployment as discussed above, but don't know yet. On management servers, there is a requirement to login via CLI in order to execute package deployments, etc. This requires a "local" account, which bypasses SSO/SAML. This also creates a backdoor for all SSO users (they are all admins on these servers). Defeats the purpose of having SSO. Have found no way around this using SSO/SAML directly between mgmt servers and IDP, but the local proxy solution might help. Will follow-up with results

tlmayes · ‎01-19-2018

Read through the docs (and several others) regarding dual use of SAML & LDAP, which all require offloading of SAML to the proxy/load balancer. Is either of your setups protecting the management servers as well with SSO using the same reverse proxy, or are you addressing only the SHC? Our target at this point are our management servers, and will later address our SHC. Interested in hearing more.

tlmayes · ‎01-19-2018

Interested in how you were able to achieve both SSO and LDAP. Currently have LDAP working and integrated with Splunk roles, all defined in Splunk Authentication/Authorization.conf. Have one production server working with SSO, Ping Identity as the IDP, controlling access to the Splunk WFE. The challenge is that our IDP solution administratively allow only (2) roles so restricted to Splunk Admin, & Splunk Users. Extremely limiting. Any help on how I might be able to leverage both LDAP (for authorization/roles) and the IDP (for authentication) would be appreciated, knowing that our IDP solution is administratively very limited.

tlmayes · ‎01-09-2018

Thanks... Found that same link as well a few minutes ago and agree that the answer is to create a new site, and search against both.

tlmayes · ‎01-09-2018

I figured as much, but asking never hurt (learn from somebody else, before causing bigger problems). Regarding replication, what is to keep the old indexers from replicating with the new? I do not want the new indexers to know about the old indexed events.

tlmayes · ‎01-09-2018

I have an indexer challenge that was hoping to get help with. We have 4 indexers with a significant amount of historical data. We are adding 4 new indexers with significantly more resources to overcome performance problems. Is it possible to do the following and if so what would be the best way to address this? Write all new events to the 4 new indexers Keep the 4 old indexers online and searchable, but do not write new events to these indexers Search is possible against all 8 indexers NO replication between the 4 old, and 4 new indexers. Only replication within their group. Thanks in advance for the help

tlmayes · ‎11-19-2017

Thanks for the response. Learned the hard way what may have been obvious, that what I was trying to do was not possible. I came to the same conclusion as you did, that an additional HF is necessary, which I am deploying now. Thanks again

tlmayes · ‎11-19-2017

After testing, seems the "fix" for the bug didn't work. CEF forwarding v2.0 & v2.0.1 does not work, even with developer support, on what I consider to be a simple deployment

tlmayes · ‎11-17-2017

I have a requirement to send certain windows events to BOTH the indexers AND a remote syslog using TCP. - The indexers should receive the events in standard Windows multiline format - The remote syslog should receive the events in single line format I have a configuration that works, forwarding all events in one or the other formats (Windows multiline, or syslog single line) but not both. How can I write props.conf so the same event is sent to the indexers in one format, and the syslog in another? If I remove the SEDCMD, everything is received in both locations in Windows format. If I include, everything is single line. How can have both depending on destination, not source? PROPS.CONF [default] TRANSFORMS-routing=Everything [source::WinEventLog:*] TRANSFORMS-routing=send_to_syslog SEDCMD-rmlines=s/[\n\r\t]/ /g

tlmayes · ‎11-17-2017

Sometimes it is the simple things... Removing the recursive line fixed it

tlmayes · ‎11-17-2017

I know this should be simple, but for whatever reason, it's not working Have a production Windows 2012 server where we are collecting application logs from a log file. The path is C:\Program Files\somepath..... so created an inputs.conf as follows [monitor://C:\Program Files\somepath\] index=someindex sourcetype=somesourcetype whitelist=\logfile.*$ Restarted the Windows UF service, no errors, but no events either (yes, confirmed there are events). So suspected permissions, and instead used: [monitor://C:\Test Folder\somepath\] index=someindex sourcetype=somesourcetype whitelist=\logfile.*$ Again, nothing, so used: [monitor://C:\TestFolder\somepath\] index=someindex sourcetype=somesourcetype whitelist=\logfile.*$ Without the whitespace, and works as expected. Put the whitespace back in, modified the log file so as to force collection, and again nothing. Was able to reproduce all on a 2012 test server. There is no provision I am aware of in inputs.conf to account for whitespace since it is supposed to be automatically recognized. What am I missing?

tlmayes · ‎11-06-2017

Have been using CEF App for Splunk for ~ 5 months now, and 2 "bugs" identified by the Splunk CEF App team. We do use this app for forwarding CEF events, even though it consistently forwards only 80% or less of the filtered events it should (another "bug" according to Splunk). We are hopeful these issues will be resolved soon. This requirement is to forward RAW un-formatted data. Would appreciate any suggestions. Your responses are appreciated since for us this is difficult to reproduce Question1: what about using the same outputs/transforms/props configuration @ the indexers instead of the HF's? Would doing so avoid the "feature" causing indexing to stop, or would the filling up of the queues still ultimately cause the issue? Question2: Since this is only a problem of filling up of queues, once the remote host starts accepting connections, do the queues now empty?

tlmayes · ‎11-06-2017

Recently we configured outputs.conf/props.conf/transforms.conf on our Heavy Forwarders to forward 3 specific events to a remote Syslog collector. The configuration worked for several days, and then we noticed that our daily indexing rate started dropping rapidly from ~ 700GB/day to under 200GB/day. Investigation of splunkd.log on the Heavy Forwarders showed that TcpOutputProc was throwing errors and failing on the connection attempt to our indexers. Engaged Splunk support. Was told that there was a known bug that affected ALL versions such that when using such a setup (forwarding to remote host) that if for any reason the connection is not made as listed in outputs.conf, that ALL forwarding will stop, including forwarding to the indexers!!! So, we choose not to use outputs.conf from our HF's since any minor connection issues with remote Syslog collectors will cause major problems. Interested in opinions on other ways to forward filtered events in raw format to a remote collector (ArcSight) in short 5 minute intervals...

tlmayes · ‎10-20-2017

Regarding apps, forwarders, and indexers.... 80 apps on 9 clustered HF's receiving from 10,000 UF's feeding 6 indexers. Don't want to route ONLY to Syslog. In my originally posted configs the long line of servers in outputs.conf (server=*****) was simply commented out. At the end of this conf was what I thought was the "additional" route statement that would handle this. Maybe the problem is not with my outputs.conf, but rather with the other configs. Or the fact that my inputs.conf is simply a listener for the 10,000 UF's. Since indexing does not occur until further down the line, there is no "content", simply receive and forward. Am I over thinking this 😞 INTPUTS.CONF [splunktcp-ssl:11001] compressed=true sslVersions = *,-ssl queueSize = 128MB persistentQueueSize = 30GB disabled = false

tlmayes · ‎10-20-2017

Thanks... got my config (such as it is) from reading the referenced doc, and according to the doc it 'should' work. The difference between yours and mine: you have an inputs.conf that identifies a source of data, which is not discussed in the document 😕 In a HF, where data/events is a pass through with no indexing or local consumable content, how does such a config in the ref. doc get the raw event for forwarding? If I DID in fact need an inputs.conf, what would it pull from?

tlmayes · ‎10-20-2017

I am being asked to forward events from a Heavy Forwarder, to a remote ArcSight server as raw events. Our HF's receive events from UF's un-indexed, and they pass-through the HF's un-indexed. Is this possible what I am trying to do? Below the config from one of our HF's, with my basic forwarding config (will refine once I make it work). ----------------------------------------- OUTPUTS.CONF ----------------------------------------- [tcpout:site-hub] server = ********* sslPassword = password sslCertPath = ******** sslRootCAPath = ******** [tcpout] defaultGroup = site-hub indexAndForward = false useACK=true maxQueueSize=128MB useClientSSLCompression = true sslVersions = tls1.1, tls1.2 heartbeatFrequency=167 autoLBFrequency = 10 [syslog:windows_events_alert] server = <REMOTE SYSLOG IP>:5166 type = tcp ----------------------------------------- PROPS.CONF ----------------------------------------- [default] TRUNCATE = 100000 [source::WinEventLog:Security] TRANSFORMS-windows_security_events = send_to_arcsight ---------------------------------------- TRANFORMS.CONF ----------------------------------------- [send_to_arcsight] REGEX = "EventCode=4725" DEST_KEY = _SYSLOG_ROUTING FORMAT = windows_events_alert

Posts	149
Solutions	3
Karma Given	12
Karma Received	20
Member Since	‎01-25-2014

Online Status	Offline
Date Last Visited	Tuesday

Splunk 9.3 deployment server page

Splunk message to teams 'chat'

SAML users unable to load reports or alerts

Access request to IA4S

Deployment Server clients have wrong apps

DMC not displaying server metrics

Splunk Enterprise upgrade to 9.1.0.1, all users di...

Anybody here running Splunk Enterprise on an IBM E...

Write to summary index, Why are some fields are no...

How to write timechart output that drops most rece...

Recurring JOIN

Re: primary volume capacity and "df -h" do not agr...

Re: primary volume capacity and "df -h" do not agr...

primary volume capacity and "df -h" do not agree

Re: Enterprise Security admin privileges, why

Enterprise Security admin privileges, why

SAML whitelist and blacklist

Splunk Powershell

Re: Managing a growing serverclass.conf

Re: SAML authentication with LDAP authorization

Re: SAML authentication with LDAP authorization

Re: SAML authentication with LDAP authorization

Re: Add new indexers, keeping old for historical

Re: Add new indexers, keeping old for historical

Add new indexers, keeping old for historical

Re: Match 2 stanzas in props.conf

Re: CEF output forwarding everything from all inde...

Match 2 stanzas in props.conf

Re: inputs.conf and Windows path

inputs.conf and Windows path

Re: Forward raw data to remote host - avoiding out...

Forward raw data to remote host - avoiding outputs...

Re: Heavy Forwarder send events to remote syslog

Re: Heavy Forwarder send events to remote syslog

Heavy Forwarder send events to remote syslog