About tyronetv

tyronetv · ‎10-23-2019

Found it. The problem was in "Splunk App for Unix" . . . File "/opt/splunk/etc/apps/splunk_app_for_nix/appserver/modules/CFHiddenSearch/CFHiddenSearch.py", line 65 except splunk.ResourceNotFound, e:

tyronetv · ‎10-23-2019

Saw this in the logs: ERROR UiPythonFallback - Appserver at http://127.0.0.1:8065 never started up! 10-23-2019 08:32:48.695 -0400 ERROR UiPythonFallback - Appserver running on port 8065 exited unexpectedly: exited with code 1 10-23-2019 08:32:48.695 -0400 ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted 10-23-2019 08:32:48.695 -0400 INFO UiHttpListener - Shutting down webui 10-23-2019 08:32:48.695 -0400 INFO UiHttpListener - Shutting down webui completed 10-23-2019 08:32:48.696 -0400 WARN UiHttpListener - Web UI now stopped

tyronetv · ‎10-23-2019

Most recent restart attempt. . . ============== splunk restart Stopping splunkd... Shutting down. Please wait, as this may take a few minutes. [ OK ] Stopping splunk helpers... [ OK ] Done. splunkd.pid doesn't exist... Splunk> Australian for grep. Checking prerequisites... Checking http port [80]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _metrics _telemetry _thefishbucket cim_modactions firedalerts history iarchive jmx main os perfmon plw puppet-enterprise summary testing101 uat unix_summary windows wineventlog Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-8.0.0-1357bef0a7f6-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done [ OK ] Notice. No attempt to start webserver or splunkd.

tyronetv · ‎10-23-2019

Did an upgrade to 8.0 on my development box (Windows w/Splunk 7,.3.1) and the upgrade went without a hitch (as normal). Did an upgrade to 8.0 on my production Linux box (w/Splunk 7.3.1) and although the upgrade appears to work, it fails to start a webserver. It doesn't even try. I have verified web.conf: [settings] startwebserver = 1 httpport = 80 enableSplunkWebSSL = false It has not changed in a number of years. Yet no web-server. When I restart all of splunk, it never generates the "waiting on splunk web . . . " When I attempt a 'splunk restart splunkweb' it gives me a very long wait before it says, "Splunk's web interface has been restarted." But there is no interface. When I do a 'netstat -tulpn | grep :80' I see ports 8089, 8051, and 8089 only. Not 80 or 8000 (in case it reverted for some reason). And Ideas on what I missed ?

tyronetv · ‎12-16-2016

tyronetv · ‎07-16-2015

When I run this search on my search head: earliest=-24h index=os sourcetype=ps tag=uat COMMAND=httpd|dedup PID | timechart span=1h count by host The resulting search causes my indexers to go from 1.2G of ram to 12.0G of ram (how much memory they have) and cause all kind of faults. Errors talking to license server, completing search, etc., etc., to the point of making any other search, scheduled or ad-hoc, fail as well. I have two indexers and am only ingesting OS data from about 50 unix/Linux hosts currently. Has anyone else seen anything like this?

tyronetv · ‎06-30-2015

I managed to get the following report from Splunk (excuse the lines, trying to format it for viewing): mrSTATUS----------------------All-------------------DBA----------------ProdEng Pending Confirmation------------6 Outstanding From Prior---------25 Closed-------------------------26 In Progress--------------------36 Touched------------------------62 Pending Confirmation----------------------------------1 Closed------------------------------------------------3 Touched----------------------------------------------11 In Progress-------------------------------------------8 Outstanding From Prior--------------------------------4 Pending Confirmation-----------------------------------------------------------5 Closed------------------------------------------------------------------------25 Touched-----------------------------------------------------------------------54 In Progress-------------------------------------------------------------------29 Outstanding From Prior--------------------------------------------------------22 How can I get them to collapse to: mrSTATUS-----------------------All------------------DBA---------------------ProdEng Pending Confirmation-------------6--------------------1---------------------------5 Outstanding From Prior----------25--------------------3--------------------------25 Closed--------------------------26-------------------11--------------------------54 In Progress---------------------36--------------------8--------------------------29 Touched-------------------------62--------------------4--------------------------22 Any ideas? The query I used is below. Might be a more efficient way to do it, but not sure how, yet. | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%DBA%' OR mrassignees LIKE '%dba1%' OR mrassignees LIKE '%dba2%' OR mrassignees LIKE '%PE1%' OR mrassignees LIKE '%Production__bEngineering%' OR mrassignees LIKE '%PE2%' OR mrassignees LIKE '%PE3%' OR mrassignees LIKE '%PE4%' OR mrassignees LIKE '%PE5%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_' AND mrUPDATEDATE < DATEADD( day, -7, CURRENT_TIMESTAMP ) ) OR ( mrUPDATEDATE >= DATEADD( day, -7, CURRENT_TIMESTAMP ) ) )" | multireport [ stats count(eval(mrSTATUS="Pending__bConfirmation")) as All | eval mrSTATUS="Pending Confirmation" ] [ stats count(eval(mrSTATUS="Closed")) as All | eval mrSTATUS="Closed" ] [ stats count as All | eval mrSTATUS="Touched" ] | append [ | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%DBA%' OR mrassignees LIKE '%dba1%' OR mrassignees LIKE '%dba2%' OR mrassignees LIKE '%PE1%' OR mrassignees LIKE '%Production__bEngineering%' OR mrassignees LIKE '%PE2%' OR mrassignees LIKE '%PE3%' OR mrassignees LIKE '%PE4%' OR mrassignees LIKE '%PE5%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_') )" | stats count as All | eval mrSTATUS="In Progress" ] | append [ | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%DBA%' OR mrassignees LIKE '%dba1%' OR mrassignees LIKE '%dba2%' OR mrassignees LIKE '%PE1%' OR mrassignees LIKE '%Production__bEngineering%' OR mrassignees LIKE '%PE2%' OR mrassignees LIKE '%PE3%' OR mrassignees LIKE '%PE4%' OR mrassignees LIKE '%PE5%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_' and mrSUBMITDATE < DATEADD( day, -7, CURRENT_TIMESTAMP )) )" | stats count as All | eval mrSTATUS="Outstanding From Prior" ] | append [ | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%DBA%' OR mrassignees LIKE '%dba1%' OR mrassignees LIKE '%dba2%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_' AND mrUPDATEDATE < DATEADD( day, -7, CURRENT_TIMESTAMP ) ) OR ( mrUPDATEDATE >= DATEADD( day, -7, CURRENT_TIMESTAMP ) ) )" | multireport [ stats count(eval(mrSTATUS="Pending__bConfirmation")) as DBA | eval mrSTATUS="Pending Confirmation" ] [ stats count(eval(mrSTATUS="Closed")) as DBA | eval mrSTATUS="Closed" ] [ stats count as DBA| eval mrSTATUS="Touched" ] ] | append [ | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%DBA%' OR mrassignees LIKE '%dba1%' OR mrassignees LIKE '%dba2%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_') )" | stats count as DBA | eval mrSTATUS="In Progress" ] | append [ | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%DBA%' OR mrassignees LIKE '%dba1%' OR mrassignees LIKE '%dba2%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_' and mrSUBMITDATE < DATEADD( day, -7, CURRENT_TIMESTAMP )) )" | stats count as DBA | eval mrSTATUS="Outstanding From Prior" ] | append [ | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%PE1%' OR mrassignees LIKE '%Production__bEngineering%' OR mrassignees LIKE '%PE2%' OR mrassignees LIKE '%PE3%' OR mrassignees LIKE '%PE4%' OR mrassignees LIKE '%PE5%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_' AND mrUPDATEDATE < DATEADD( day, -7, CURRENT_TIMESTAMP ) ) OR ( mrUPDATEDATE >= DATEADD( day, -7, CURRENT_TIMESTAMP ) ) )" | multireport [ stats count(eval(mrSTATUS="Pending__bConfirmation")) as ProdEng | eval mrSTATUS="Pending Confirmation" ] [ stats count(eval(mrSTATUS="Closed")) as ProdEng | eval mrSTATUS="Closed" ] [ stats count as ProdEng | eval mrSTATUS="Touched" ] ] | append [ | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%PE1%' OR mrassignees LIKE '%Production__bEngineering%' OR mrassignees LIKE '%PE2%' OR mrassignees LIKE '%PE3%' OR mrassignees LIKE '%PE4%' OR mrassignees LIKE '%PE5%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_') )" | stats count as ProdEng | eval mrSTATUS="In Progress" ] | append [ | dbquery footprints "SELECT mrID, mrSTATUS, mrassignees FROM footprints.dbo.MASTER49 WHERE ( mrassignees LIKE '%PE1%' OR mrassignees LIKE '%Production__bEngineering%' OR mrassignees LIKE '%PE2%' OR mrassignees LIKE '%PE3%' OR mrassignees LIKE '%PE4%' OR mrassignees LIKE '%PE5%' ) AND ( ( mrSTATUS <> 'Closed' AND mrSTATUS <> '_DELETED_' and mrSUBMITDATE < DATEADD( day, -7, CURRENT_TIMESTAMP )) )" | stats count as ProdEng | eval mrSTATUS="Outstanding From Prior" ] | fields mrSTATUS, All, DBA, ProdEng

tyronetv · ‎05-29-2015

See above. I included a splunkd.log output.

tyronetv · ‎05-29-2015

"test alert email preview" Could you explain that to me? I did do an attempted send via "sendemail" from the search line and received the same errors.

tyronetv · ‎05-29-2015

From splunkd.log 05-29-2015 07:53:02.696 -0700 ERROR HttpListener - Handler for /en-US/api/shelper?snippet=true&snippetEmbedJS=false&namespace=search&search=search+sourcetype%3Dt2_app+OR+sourcetype%3Ddis+iview2+%22%3Fdoc%3D%22+OR+%22%26doc%3D%22+%7C++timechart+span%3D1h+count+by+host+%7C+sendemail+to%3D%22tyrone.vaughn%40transcentra.com%22&useTypeahead=true&useAssistant=true&showCommandHelp=true&showCommandHistory=true&showFieldInfo=false&_=1432911162651 sent a 0 byte response after earlier claiming a Content-Length of 2160! ============= This was working until recently. No changes that I am aware of on the machine or the environment, but I can't send email out any more. Above is the log entry when I execute that search from the interface. The interfaces responds with No connection could be made because the target machine actively refused it while sending mail to: (my email address) . I can send email from Powershell using: $EmailFrom = "(source address)" $EmailTo = "(my address, same as above)" $Subject = "Test from CLI" $Body = "Testing.. 1 2 3" $SMTPServer = "(my relay, same one in Splunk config) $SMTPClient = New-Object Net.Mail.SmtpClient($SMTPServer, 25) $SMTPClient.Send($EmailFrom, $EmailTo, $Subject, $Body) So I don't think it is a machine issue.

tyronetv · ‎06-04-2014

Have an environment where a directory is used to 'stage' files waiting for an update. Essentially, a file is sent to a vendor and a local status file (of zero or more bytes) is placed in a 'pending' directory. Once the vendor has completed their work and a good return is received the local status file is moved to the 'done' directory. (This is done so a second process can use the listing to determine what status file to check for on it's run). What I want to do is monitor the '/pending' directory for new files only being worried about WHEN they showed up. Then, if they haven't 'moved' in a predetermined amount of time (depending upon environment this could be two - eight hours) raise an alert. This would also allow me to say "_time seen" and "_time gone" to build some metrics on vendor performance. All that being said, I am having issues getting it set up to forward from my linux universal forwarder. My inputs.conf stanza looks like: [fschange:/path/to/dir/] signedaudit = false index=_audit I read that in one of the previous 'answers' questions. Unfortunately, I am seeing no records in _audit or _internal for that host using the search head. Ideas?

tyronetv · ‎05-20-2014

I have a request that is sent out in the following format: ?doc=A0RF7S:36518:2;A0RET7:36254:1;A0REQ2:38161:2;A0REJ8:129192:6;A0RDYJ:35301:2 Or, it can be: ?doc=A0RDSZ:9865:2 Or any number of what we call 'doc triplets' separated by semi-colons. What I want to do is extract the three component pieces whether it happens once or n+1 times. The doc triplet is JOBID:OFFSET:PAGES I would like to pull, at a minimum, the JOBID and PAGES. This way I can report on PAGES requested per JOBID, or unique JOBID's requested, or total pages per time-period, etc. Any suggestions on how to write that regex? Thanks!

tyronetv · ‎05-06-2014

Of course I can identify the sourcetype via the command line. The test was to check whether the props.conf on the indexer would do the identification so I can deploy an app to gather these logs from various machines and various users (hence the * in the path).

tyronetv · ‎05-05-2014

Given this in the props.conf on my indexer: [source://c:\Documents and Settings\*\AppData\Roaming\Ipswitch\WS_FTP\Logs\ws_ftp.log] sourcetype = wsftp_log [source://c:\Documents and Settings\*\AppData\Roaming\Ipswitch\WS_FTP\Logs\*.rtf] sourcetype = wsftp_session [wsftp_log] TIME_PREFIX = ^ TIME_FORMAT = %Y\.%m\.%d %H:%M MAX_TIMESTAMP_LOOKAHEAD = 19 SHOULD_LINEMERGE = FALSE LINE_BREAKER = ([\n\r]+)(?=\d{4}.\d{2}.\d{2}\s\d{2}:\d{2}} TRUNCATE = 99999 [wsftp_session] TIME_PREFIX = ^\cf2 \[ TIME_FORMAT = %Y\.%m\.%d %H:%M:%S\.%3N SHOULD_LINEMERGE = FALSE MAX_TIMESTAMP_LOOKAHEAD = 30 LINE_BREAKER = ([\n\r]+)(?=^\cf2\s\[) TRUNCATE = 999999 When I run this: $SPLUNK_HOME\bin\splunk add oneshot "C:\documents and settings\(my user id)\Appdata\roaming\ipswitch\ws_ftp\logs\ws_ftp.log" -index testing OR $SPLUNK_HOME/bin/splunk add oneshot "C:\documents and settings\(my user id)\Appdata\roaming\ipswitch\ws_ftp\logs\salem_file1.rtf" -index testing It doesn't identify the sourcetype at all. Why?

tyronetv · ‎05-01-2014

(?<HOST>[^/(?<!Ii)]+) That sort of works.. but concerned if the FQDN has an 'i' in it.

tyronetv · ‎05-01-2014

I have a log file that has the host as either an IP address OR a FQDN. This is easy, right. Nope. The format can be: 192.168.1.1/inbox/to/file or 192 .168.1.1inbox/to/another-file or host.com/inbox/to/file or host.cominbox/to/another-file I can regex the ip easy enough \d{1,3}.\d{1,3}.\d{1,3}.\d{1,3} so that handles the first two . . For the third one, I can do a greedy up to / [^/]+ problem is the last one, actually... The perfect solution would be one regex for the extract regardless. Written out it would be: everything up to either / or the word 'inbox' That way, I can do the extract as : (?<HOST>(provided regex)) Thanks! The full entry is gathered by: ^\d{4}.\d{2}.\d{2}\s\d{2}:\d{2}\s(?<Transmissionformat>\w)\s(<Fileone>[^-]+)\s(?<DIRECTION>\S{3})\s(?<HOST>???????)(?<Filetwo>[^$]+)$ I need the ???? part replaced with a regex that will do as described above. 2014.04.28 14:10 B n:\path\one\to\fileone <-- 172.21.1.2Inbox/path/filetwo

tyronetv · ‎02-04-2014

Yeah. I understand this as a solution but more often then not (and shame on me training them as such) the users are used to seeing a sequential listing of the user's activity and I've been asked to mark the one that Error'd as part of the report. So, I find the user, then their RID's, and then generate a report. I need the report to show the ERROR on the line item of the users activity.

tyronetv · ‎02-04-2014

I have a unique ID (RID) for a each client click. A single click can execute between 3 and, sometimes, over 100 log entries. Sometimes, in the middle of all of that is a situation where an error occurs. I currently have a search that generates a list of actions by userid/RID but the report creates every line as an "INFO" response (I guess because the first and last line of the transaction are always INFO log level). What I need to do is set the corresponding report to acknowledge the transaction as an ERROR level based upon the ERROR existing someone in the middle of the transaction. How would I do that? 2014-02-04 05:58:14,473 INFO [ndler-HTTPThreadGroup-289] RID=1391522013732-458538 2014-02-04 05:58:14,462 ERROR [ndler-HTTPThreadGroup-289] RID=1391522013732-458538 2014-02-04 05:53:33,776 DEBUG [ndler-HTTPThreadGroup-289] RID=1391522013732-458538 2014-02-04 05:53:33,760 DEBUG [ndler-HTTPThreadGroup-289] RID=1391522013732-458538 2014-02-04 05:53:33,760 INFO [ndler-HTTPThreadGroup-289] RID=1391522013732-458538 The above is for a client search request. The request timed out after 5 minutes. When I report on this transaction as a summary of actions taken by the client I currently report on the action and the duration. I would like to include the "Error_Level" as a column with a drill down to the actual transaction(RID) experiencing the error. But first, I just want to be able to identify it as an 'error' transaction.

tyronetv · ‎01-29-2014

So, you are saying, in props I could do something similar to: EXTRACT-subfields = SOURCE_KEY=SFIL (? \w+).(?P \w{3}\d{3}).(? \d{3})(?P \d{2})

tyronetv · ‎01-29-2014

I have an application sourcetype that is a mix of normal informational data and also houses a subset of web requests and web responses all in XML format. I would like to present the XML data as a searchable element, i.e, account=1234, while at the same time allowing the current key/value pairs to be searched. As I read the docs, kv_mode is basically all or nothing. In that, it's one mode only. How would I go about capturing both field elements from my logs using the splunk field identification process and not having to write thousands of extract statements?

tyronetv · ‎01-29-2014

I do something today that matches this and it works just fine. What I want is something I can put in props/transforms and allow other users to run the searches without teaching them this level of syntax. I.e., just teach the support guy to type 'login = ??' The system records the login ID only once, on the entry wherein the token is assigned. From that point forward, every log entry references the token only.

tyronetv · ‎01-29-2014

When my users log into my system they are identified with a token number that follows all of their activity going forward during that session. What I want to do is say login=john.smith@hostname.com and have it track all the log entries that have the corresponding token. (-459867834847833) Each time a user log's in, they get a new token. Suggestions?

tyronetv · ‎01-27-2014

I read the linked answer and it points the user towards the doc for transforms.conf. Specifically the SOURCE_KEY value. But, I'm afraid I need more information. The spec file says it can be used to meet my need but I don't understand how it is entered in the transforms.conf to expose the values to my end users without teaching them rex, etc.

tyronetv · ‎01-27-2014

This works and I appreciate pointing me to this but I have to figure out how to set it up in transforms.conf so it's visible without teaching the end users to use rex, etc.

tyronetv · ‎01-27-2014

I mean, instead of only doing SFIL=* I would like to do app=* or site=* or client=*. For example, client=abc124 site=34 |stats count by hour

Posts	66
Solutions	4
Karma Given	5
Karma Received	23
Member Since	‎09-07-2011

Online Status	Offline
Date Last Visited	‎08-11-2020 12:50 PM

Splunk 8.0 upgrade has no web server running

How to edit my search to reflect stats related the...

Why is the Splunk Add-on for Unix and Linux consum...

Formatting a multi column report.

email not sending from windows search head

File system changes (not log monitoring)

Repeating Field Extraction

source type identification in props.conf

regex difficulty with host / path discrimination

Finding an Error in the middle of a Transaction

Re: Splunk 8.0 upgrade has no web server running

Re: Splunk 8.0 upgrade has no web server running

Re: Splunk 8.0 upgrade has no web server running

Splunk 8.0 upgrade has no web server running

How to edit my search to reflect stats related the...

Why is the Splunk Add-on for Unix and Linux consum...

Formatting a multi column report.

Re: email not sending from windows search head

Re: email not sending from windows search head

email not sending from windows search head

File system changes (not log monitoring)

Repeating Field Extraction

Re: source type identification in props.conf

source type identification in props.conf

Re: regex difficulty with host / path discriminati...

regex difficulty with host / path discrimination

Re: Finding an Error in the middle of a Transactio...

Finding an Error in the middle of a Transaction

Re: Fields within fields - search time extract

Can I use kv_mode = auto AND kv_mode = XML for sou...

Re: dynamic lookup based upon dynamic token

dynamic lookup based upon dynamic token

Re: Fields within fields - search time extract

Re: Fields within fields - search time extract

Re: Fields within fields - search time extract