About butzowj

butzowj · ‎11-30-2016

Hello, My management (and me as well, of course) loves the way the visualizations for real time searches look. But from a system administration perspective, it's a nightmare, as we are all well aware of the impact real time searches have on the system. To clarify, a real time search, when it updates its data, seamlessly and continuously transforms the current data point to the next data point as new data streams through Splunk. However, if we use a historical search with an auto-refresh, there is sort of 'flash' of blank space as the search runs and populates the visualization with the newly retrieved data. I am looking for a solution to replace the real time searches with a historical search without impacting the visualization - in other words, a historical search that displays like a real time search (only, of course without the continual updates to the values). Is this possible, has any one tried this or have any ideas? Again, the idea is to give management the 'smooth' visualizations they have come to expect from the real time searches without having to actually run a real time search. Thanks, Joel B

butzowj · ‎08-05-2016

Hello - I have successfully installed the JMS Messaging Modular Input with the intention of monitoring Websphere MQ statistics. (e.g. Queue Depth, Oldest Message Age, etc...). I am able to successfully pull test messages off the queue and index them with Splunk. However, when I pull messages off the statistics queue, the format is crazy, and looks like this: Fri Aug 05 13:21:06 EDT 2016 name=QUEUE_msg_received event_id=ID:414d512051756575654d6772202020205d9a9f572010d5f4 msg_dest=SplunkStatsQueue msg_body=$� D�0QueueMgr � 2016-08-05 � 13.20.36 � 2016-08-05 � 13.21.06 ��K D�0SplunkQueue � 2016-08-01 �13.26.01�� K D�0SYSTEM.ADMIN.STATISTICS.QUEUE � 2016-08-01 �13.23.02�� "�� |� �|�� K D�0SYSTEM.ADMIN.COMMAND.QUEUE � 2016-08-01 �13.23.01�� X� �X�� K D�0AMQ.MQEXPLORER.416946394 � 2016-08-05 �10.15.48��+ �\ �� u�� u�� Is there something I am missing that will translate this message format into readable statistics? Perhaps a setting on the MQ side? Thanks JB

butzowj · ‎07-29-2016

Hi! Can you point out where you got the jar files from? Thanks!

butzowj · ‎07-21-2016

Yes, I am only the splunk administrator, so I have limited understanding of okta. This is how I have configured things so far; for dev, QA, & stage environment, I supplied the following: URL, ( e..g - dev-123456.okta-preview.com) all non-production URLs contained the word "preview" API Token, (e.g. - ASDHJKGHD123123146346etc....) unique API token for each URL For prod I did the same, only with a different URL & API key. So I suppose I am trying to determine if there is a difference in how an okta-preview URL would be set up compared to a production okta URL. Thanks!

butzowj · ‎07-20-2016

Hello - I am using the Splunk App and Add-on for Okta. Works pretty great so far, EXCEPT: The production Okta data is not being returned by the Splunk app, but all of the non-production data is (dev, stage, qa). Is there anything obvious I could change from the Splunk side that would make this call work? Thanks! JB

butzowj · ‎07-08-2016

Hello - I am working on getting the Splunk App and Add-on for Okta working for Splunk 6.2.4. Currently I am pulling in some data with the okta:user sourcetype, but I just keep pulling back the same records... What can I do to get the app to stop pulling in duplicate records? Thanks!

butzowj · ‎04-20-2016

Wanted to thank you for this. I as able to use your example to solve a problem I had, and you also helped me understand how subsearches work! Thanks!

butzowj · ‎04-04-2016

This works for now - it may not be the best practices solution in the long term, but forcing the dashboard to refresh will definitely alleviate the crashes. Thanks!

butzowj · ‎04-04-2016

Can you explain what is meant by "they probably are not working the way you expect anyway"?

butzowj · ‎04-04-2016

Hello, We have Splunk 6.2, and we have a dashboard that utilizes 9 real-time searches and 4 historical searches. On both Chrome and Firefox, the dashboard causes the browser to crash intermittently throughout the day. Is there anything that can be done to alleviate this issue? We'd like to keep the dashboard running throughout the day in our office, and this issue is preventing us from doing so. Thanks, JB

butzowj · ‎03-24-2016

Hello Splunkers - I'd like to include the time/date range of a search in my report. If I have a report that runs at midnight showing all the results for yesterday, I'd like to be able to see the actual date on the report, not just the word "yesterday". Is there any way this can be done? Thanks.

butzowj · ‎02-02-2016

Hey all, We have seen these crashes happen on two servers the past few days. Is there anything in the crash log that would help identify a root cause? I have browsed through the log but nothing jumps out at me. See logs below... Thx, JB [splunk@rtlvpxawsb splunk]$ cat crash-2016-01-31-00\:10\:01.log [build 271043] 2016-01-31 00:10:01 Received fatal signal 6 (Aborted). Cause: Signal sent by PID 10897 running under UID 49436. Crashing thread: MainTailingThread Registers: RIP: [0x0000003E7CA32625] gsignal + 53 (/lib64/libc.so.6) RDI: [0x0000000000002A91] RSI: [0x0000000000002AB0] RBP: [0x0000000001612A40] RSP: [0x00007FD2AA5F9278] RAX: [0x0000000000000000] RBX: [0x00007FD2B3ECB000] RCX: [0xFFFFFFFFFFFFFFFF] RDX: [0x0000000000000006] R8: [0xFEFEFEFEFEFEFEFF] R9: [0x00007FD2B3F61F60] R10: [0x0000000000000008] R11: [0x0000000000000202] R12: [0x00000000015B46B5] R13: [0x0000000001614660] R14: [0x00007FD2AA5F9B30] R15: [0x00007FD2A98A42C0] EFL: [0x0000000000000202] TRAPNO: [0x0000000000000000] ERR: [0x0000000000000000] CSGSFS: [0x0000000000000033] OLDMASK: [0x0000000000000000] OS: Linux Arch: x86-64 Backtrace: [0x0000003E7CA32625] gsignal + 53 (/lib64/libc.so.6) [0x0000003E7CA33E05] abort + 373 (/lib64/libc.so.6) [0x0000003E7CA2B74E] ? (/lib64/libc.so.6) [0x0000003E7CA2B810] __assert_perror_fail + 0 (/lib64/libc.so.6) [0x000000000099145A] ? (splunkd) [0x000000000098D582] _ZNK11TailWatcher12setupConfigsER15WatchedTailFile + 1474 (splunkd) [0x000000000098D692] _ZNK11TailWatcher19initializeFileStateER15WatchedTailFileRK8Pathname + 66 (splunkd) [0x00000000009904B2] _ZN11TailWatcher11fileChangedEP16WatchedFileStateRK7Timeval + 242 (splunkd) [0x0000000000EC2602] _ZN30FilesystemChangeInternalWorker15callFileChangedER7TimevalP16WatchedFileState + 114 (splunkd) [0x0000000000EC3F90] _ZN30FilesystemChangeInternalWorker12when_expiredERy + 464 (splunkd) [0x0000000000F53B2D] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval + 301 (splunkd) [0x0000000000EBD818] _ZN9EventLoop3runEv + 744 (splunkd) [0x000000000098E9ED] _ZN11TailWatcher3runEv + 141 (splunkd) [0x000000000099428A] _ZN13TailingThread4mainEv + 154 (splunkd) [0x0000000000F5165E] _ZN6Thread8callMainEPv + 62 (splunkd) [0x0000003E7CE079D1] ? (/lib64/libpthread.so.0) [0x0000003E7CAE88FD] clone + 109 (/lib64/libc.so.6) Linux / rtlvpxawsb.labcorp.com / 2.6.32-504.16.2.el6.x86_64 / #1 SMP Tue Mar 10 17:01:00 EDT 2015 / x86_64 Last few lines of stderr (may contain info on assertion failure, but also could be old): 2015-10-14 16:39:28.077 -0400 splunkd started (build 271043) Conf mutator lockfile has disappeared; error condition possible. 2015-10-15 15:07:01.431 -0400 splunkd started (build 271043) Conf mutator lockfile has disappeared; error condition possible. 2015-10-15 16:25:07.121 -0400 splunkd started (build 271043) Conf mutator lockfile has disappeared; error condition possible. 2015-10-29 14:25:14.432 -0400 splunkd started (build 271043) splunkd: /home/build/build-src/6.2.4/src/pipeline/input/Tailing.h:120: bool StatWrap::isDir() const: Assertion `_valid' failed. 2015-12-15 08:16:03.012 -0500 splunkd started (build 271043) splunkd: /home/build/build-src/6.2.4/src/pipeline/input/Tailing.h:120: bool StatWrap::isDir() const: Assertion `_valid' failed. /etc/redhat-release: Red Hat Enterprise Linux Server release 6.6 (Santiago) glibc version: 2.12 glibc release: stable Last errno: 2 Threads running: 30 argv: [splunkd -p 8089 start] Thread: "MainTailingThread", did_join=0, ready_to_run=Y, main_thread=N First 8 bytes of Thread token @0x7fd2b1c6d150: 00000000 00 a7 5f aa d2 7f 00 00 |.._.....| 00000008 First 512 bytes of Timeout object @0x7fd2aa5f9a88: 00000000 10 f2 6d 01 00 00 00 00 00 00 00 00 00 00 00 00 |..m.............| 00000010 38 98 5f aa d2 7f 00 00 00 00 00 00 00 00 00 00 |8._.............| 00000020 00 00 00 00 00 00 00 00 29 97 ad 56 00 00 00 00 |........)..V....| 00000030 3e 94 0d 00 00 00 00 00 00 00 00 00 00 00 00 00 |>...............| 00000040 80 9a 5f aa d2 7f 00 00 20 9c 5f aa d2 7f 00 00 |.._..... ._.....| 00000050 01 00 00 00 01 00 00 00 c0 a7 14 a9 d2 7f 00 00 |................| 00000060 80 d6 1c b0 d2 7f 00 00 c0 28 10 a9 d2 7f 00 00 |.........(......| 00000070 00 43 8a a9 d2 7f 00 00 00 9b 5f aa d2 7f 00 00 |.C........_.....| 00000080 00 9b 5f aa d2 7f 00 00 10 9b 5f aa d2 7f 00 00 |.._......._.....| 00000090 10 9b 5f aa d2 7f 00 00 20 9b 5f aa d2 7f 00 00 |.._..... ._.....| 000000a0 20 9b 5f aa d2 7f 00 00 c0 41 8a a9 d2 7f 00 00 | ._......A......| 000000b0 40 3f 8a a9 d2 7f 00 00 00 00 00 00 00 00 00 00 |@?..............| 000000c0 00 e0 0d ab d2 7f 00 00 14 10 00 00 00 00 00 00 |................| 000000d0 51 f3 4b 55 00 00 00 00 78 49 61 01 00 00 00 00 |Q.KU....xIa.....| 000000e0 d8 f6 16 b0 d2 7f 00 00 00 00 00 00 00 00 00 00 |................| 000000f0 00 00 00 00 00 00 00 00 80 4b 1f ab d2 7f 00 00 |.........K......| 00000100 10 4c 1f ab d2 7f 00 00 c0 4d 1f ab d2 7f 00 00 |.L.......M......| 00000110 0c 00 00 00 00 00 00 00 00 e4 1e b0 d2 7f 00 00 |................| 00000120 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |d...............| 00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000140 b8 9b 5f aa d2 7f 00 00 b8 9b 5f aa d2 7f 00 00 |.._......._.....| 00000150 00 00 00 00 00 00 00 00 80 63 1a b0 d2 7f 00 00 |.........c......| 00000160 e0 8d 82 af d2 7f 00 00 00 00 00 00 00 00 00 00 |................| 00000170 18 2e 45 b2 d2 7f 00 00 00 e4 1e b0 d2 7f 00 00 |..E.............| 00000180 00 48 0c ab d2 7f 00 00 30 48 0c ab d2 7f 00 00 |.H......0H......| 00000190 40 48 0c ab d2 7f 00 00 26 00 00 00 10 00 00 00 |@H......&.......| 000001a0 80 a7 14 a9 d2 7f 00 00 00 00 00 00 00 00 00 00 |................| 000001b0 88 9a 5f aa d2 7f 00 00 00 00 00 00 00 00 00 00 |.._.............| 000001c0 25 00 00 00 d2 7f 00 00 40 6b 2e b0 d2 7f 00 00 |%.......@k......| 000001d0 10 00 00 00 aa aa aa aa 00 00 00 00 00 00 00 00 |................| 000001e0 00 00 00 00 d2 7f 00 00 60 99 5f aa d2 7f 00 00 |........`._.....| 000001f0 00 a7 5f aa d2 7f 00 00 00 00 00 00 00 00 00 00 |.._.............| 00000200 FilesystemChangeWatcher: _timeoutActive=Y, _throttled=N, _waitingForNotifyCount=1 EMPTY Q: waitingForTimeout=N, noAction=N, stat=Y, immediateStat=Y, readdir=Y, notify=N WatchedTailFile-WatchedFileState: path="/etc/httpd/logs/stage-phoenix.labcorp.com-ssl-request.log.5", flags=0x24023 First 144 bytes of PathnameStat @0x7fd2a98a4348: 00000000 30 2c 20 73 6f 75 72 63 65 50 6f 72 74 3d 38 30 |0, sourcePort=80| 00000010 38 39 2c 20 64 65 73 74 49 70 3d 31 30 2e 31 31 |89, destIp=10.11| 00000020 31 2e 31 2e 31 39 35 2c 20 64 65 73 74 50 6f 72 |1.1.195, destPor| 00000030 74 3d 39 39 39 37 2c 20 5f 74 63 70 5f 42 70 73 |t=9997, _tcp_Bps| 00000040 3d 32 34 31 39 2e 34 37 2c 20 5f 74 63 70 5f 4b |=2419.47, _tcp_K| 00000050 42 70 73 3d 32 2e 33 36 2c 20 5f 74 63 70 5f 61 |Bps=2.36, _tcp_a| 00000060 76 67 5f 74 68 72 75 70 75 74 3d 32 2e 33 36 2c |vg_thruput=2.36,| 00000070 20 5f 74 63 70 5f 4b 70 72 6f 63 65 73 73 65 64 | _tcp_Kprocessed| 00000080 3d 37 31 2c 20 5f 74 63 70 5f 65 70 73 3d 31 2e |=71, _tcp_eps=1.| 00000090 FilesystemChangeWatcher: _timeoutActive=Y, _throttled=N, _waitingForNotifyCount=1 EMPTY Q: waitingForTimeout=N, noAction=N, stat=Y, immediateStat=Y, readdir=Y, notify=N Timeout: _when = 2321382613982983482.5641075399597568045, _initialMsec = 8247328199548096326 file-in: _initialized=Y, _lastCharWasNewline=N, _lastReadHadNulls=N, _wasCrcConflict=N, _warned=N _nullsWarned=N, _wasTooNew=N, _exists=N, _noDebug=N _hadExplicitSource=N, _crossedInitCrcLenBoundary=N, _classifiedAtLeastOnce=N, _fileReplaced=N, _readPathAfterRealEOF=N _onlyNotifiedOnce=Y, _isArchive=N, _isCached=111213, _unowned=N, _deleteOnEOF=N _overrideDeleteOnEOF=N, _doNotDeleteChildren=N, _readFromEnd=N, _readIrregardless=N _fileCheckMethod=0, _crcSalt=<null>, _origPath=<null> _bytesRead=0, _storingBytesRead=0, _initCrc=0x0, _seekCrc=0x0 _filenameCrc=0x16ab246dab3357c1, _fallbackCrc=0x0, _lastEOFTime=<zero>, _modTime=<zero> _eofSeconds=3, _ignoreThresh=<zero>, _initCrcBytes=256, _initCrcForBatch=0x0 _pendingMetadata=<null> _prevFd=-1, _pdModels=[0 PDs] _rescheduleDelay=1000, _rescheduleTarget=<zero>, _name=/etc/httpd/logs/stage-phoenix.labcorp.com-ssl-request.log.5, _statusName= _st=[dev=64773, ino=36, mode=100644, size=7204328, mtime=1453796447, owner=0, group=3000] _toStringPrefix=state=0x0x7fd2a98a42c0, _backoff=0 _stdataInputHeaderProcessing=[] _detectTrailingNulls=N, _detectReadingFromOffSet=N, _readAndSkipHeader=N, _uniqueId=439908 _rawPath= x86 CPUID registers: 0: 0000000D 756E6547 6C65746E 49656E69 1: 000206D7 02010800 9E982203 0FABFBFF 2: 76035A01 00F0B2FF 00000000 00CA0000 3: 00000000 00000000 00000000 00000000 4: 00000000 00000000 00000000 00000000 5: 00000000 00000000 00000000 00000000 6: 00000077 00000002 00000009 00000000 7: 00000000 00000000 00000000 00000000 8: 00000000 00000000 00000000 00000000 9: 00000001 00000000 00000000 00000000 A: 07300401 0000007F 00000000 00000000 B: 00000000 00000000 000000FD 00000002 C: 00000000 00000000 00000000 00000000 D: 00000000 00000000 00000000 00000000 80000000: 80000008 00000000 00000000 00000000 80000001: 00000000 00000000 00000001 28100800 80000002: 20202020 49202020 6C65746E 20295228 80000003: 6E6F6558 20295228 20555043 342D3545 80000004: 20303436 20402030 30342E32 007A4847 80000005: 00000000 00000000 00000000 00000000 80000006: 00000000 00000000 01006040 00000000 80000007: 00000000 00000000 00000000 00000100 80000008: 00003028 00000000 00000000 00000000 terminating...

butzowj · ‎01-25-2016

I am having the same issue. First we try to run the deployer command to push the app to the cluster members, but we get the error above. I then created the app locally on each search head cluster member, but we still see the same issue: [splunk@ bin]$ ./splunk apply shcluster-bundle -target https://ClusterMember3.labcorp.com:808 9 -auth admin:labcorp1 Warning: Depending on the configuration changes being pushed, this command might initiate a rollin g restart of the cluster members. Please refer to the documentation for the details. Do you wish t o continue? [y/n]: y Error while deploying apps to target=https://ClusterMember3.labcorp.com:8089 with members=3: ConfDeploy mentException: Error while updating app=admin_app on target=https://10.111.1.192:8089: Non-200/201 status_code=404; {"messages":[{"type":"ERROR","text":"Application does not exist: "}]} Error while deploying apps to target=https://ClusterMember3.labcorp.com:8089 with members=3: ConfDeploy mentException: Error while updating app=admin_app on target=https://10.111.1.193:8089: Non-200/201 status_code=404; {"messages":[{"type":"ERROR","text":"Application does not exist: "}]} Thanks, JB

butzowj · ‎01-07-2016

HI Madan - Thx for the response. Our search heads write data locally right now, because we don't know how to configure it any other way. Ideally, we would write send this data to the index cluster to be indexed with the rest of our data. It sounds like we need to utilize an outputs.conf file to forward the locally indexed data to the index cluster. Which outputs.conf file would we use, and would this have any other potential impacts to the system? Thanks, JB

butzowj · ‎01-07-2016

Hi, We are running S.o.S - Splunk on Splunk in a search head clustering environment. Is it possible to get the resource utilization information from all search heads to be searchable from a single search head using the S.o.S add-on for Linux? Right now, I have to log in to Splunk Web for each search head to get the data for that search head, (i.e. I have to log in to searchhead1:8000 to view info for searchhead1). Thanks, JB

butzowj · ‎12-04-2015

Thanks for the help Sunda, I actually ended up using the if function to do what I needed, as below: index=f5 pool rtlvpxaw01 | head 1 | eval status=if(F5_TCPStatus="up",10,0) | stats sum(status) AS severity | rangemap field=severity severe=0-9 low=10-11 default=elevated Cheers, JB

butzowj · ‎12-04-2015

Hi Splunkers! I am running the following search to try and apply a "low" rangemap value if a string matches "up", and a "severe" rangemap value if the string matches "down", but I can' t get the eval field to show in the search results. Here is the search I am using: index=f5 pool | head 1 | eval alert_level = case(F5_TCPStatus=="up",1,F5_TCPStatus=="down",0) | rangemap field=alert_level low=1-1 elevated=2-2 severe=3-3 Thanks for any help! JB

butzowj · ‎10-27-2015

What I did here was add the following line to my CSS file (this was for a single-value chart type): .dashboard-element { width: 100% !important; } This overrides the default width of 33.33% with "100%".

butzowj · ‎10-02-2015

Rodrigo - Thanks for the assistance, it turns out that it was a net-snmp dependency issue on two of our three search heads. I am still seeing the error above when I manually execute the script, but when the script is fired as part of a Splunk alarm, it works properly. Thanks for your help! JB

butzowj · ‎10-01-2015

Rodrigo - I am having the exact same problem as the poster above, using the Splunk-provided SNMP perl script (pasted below). #!/usr/bin/perl # # sendsnmptrap.pl: A script to enable using Splunk alerts to send an SNMP trap. # # Modify the following code as necessary for your local environment. # $hostPortSNMP = "qa-tm1:162"; # Host:Port of snmpd or other SNMP trap handler $snmpTrapCmd = "/usr/bin/snmptrap"; # Path to snmptrap, from http://www.net-snmp.org $TRAPOID = "1.3.6.1.4.1.27389.1.2"; # Object IDentifier for traps/notifications $OID = "1.3.6.1.4.1.27389.1.1"; # Object IDentifier for objects, Splunk Enterprise OID is 27389 # Parameters passed in from the alert. # $1-$9 is the positional parameter list. $ARGV[0] starts at $1 in Perl. $searchCount = $ARGV[0]; # $1 - Number of events returned $searchTerms = $ARGV[1]; # $2 - Search terms $searchQuery = $ARGV[2]; # $3 - Fully qualified query string $searchName = $ARGV[3]; # $4 - Name of saved search $searchReason = $ARGV[4]; # $5 - Reason saved search triggered $searchURL = $ARGV[5]; # $6 - URL/Permalink of saved search $searchTags = $ARGV[6]; # $7 - Always empty as of 4.1 $searchPath = $ARGV[7]; # $8 - Path to raw saved results in Splunk instance (advanced) # Send trap, with the parameter list above mapping down into the OID. $cmd = qq/$snmpTrapCmd -v 2c -c public $hostPortSNMP '' $TRAPOID $OID.1 i $searchCount $OID.2 s "$searchTerms" $OID.3 s "$searchQuery" $OID.4 s "$searchName" $OID.5 s "$searchReason" $OID.6 s "$searchURL" $OID.7 s "$searchTags" $OID.8 s "$searchPath"/; system($cmd); If you could provide any assistance I would greatly appreciate it. Thanks! JB

Posts	20
Solutions	0
Karma Given	32
Karma Received	2
Member Since	‎09-18-2015

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

How to replace a real time search with a historica...

How to configure the JMS Messaging Modular Input t...

Splunk App and Add-on for Okta: Why are we not see...

How to get the Splunk App and Add-on for Okta to s...

Splunk Dashboard Causing Browsers to crash?

Include date on a Splunk report

Splunk 6.2.4 Light Forwarder Crashing

Is it possible to get resource utilization informa...

How to apply a rangemap to string values?

How to replace a real time search with a historica...

How to configure the JMS Messaging Modular Input t...

Re: Why am I getting an error connecting to WebSph...

Re: Splunk App and Add-on for Okta: Why are we not...

Splunk App and Add-on for Okta: Why are we not see...

How to get the Splunk App and Add-on for Okta to s...

Re: Search based on results of another search?

Re: Splunk Dashboard Causing Browsers to crash?

Re: Splunk Dashboard Causing Browsers to crash?

Splunk Dashboard Causing Browsers to crash?

Include date on a Splunk report

Splunk 6.2.4 Light Forwarder Crashing

Re: Splunk apply shcluster-bundle fail

Re: Is it possible to get resource utilization inf...

Is it possible to get resource utilization informa...

Re: How to apply a rangemap to string values?

How to apply a rangemap to string values?

Re: How to control the whitespace or panels size i...

Re: Problem with sending SNMP traps to other syste...

Re: Problem with sending SNMP traps to other syste...