I have not tried changing the search mode to "fast". How would I do that in advanced xml? And I'm extracting many fields (in props.conf) at search time but that's about it. ... View more

ok so how about the output I just posted.. doesn't this look fine implying there is no issue with the server and thus it must be something on the Splunk app level. ... View more

We are using Raid 0, The output below is the disk IO with 1 sec poll intervals during a search: Linux 2.6.32-431.el6.x86_64 (slc-srv-splunk.glg.local) 03/28/2014 x86_64 (4 CPU) Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 8.89 9.65 113.38 5699919 66941072 dm-0 28.62 9.63 113.38 5684777 66940924 dm-1 0.00 0.00 0.00 1288 0 dm-2 0.00 0.00 0.00 925 132 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 34.00 0.00 492.00 0 492 dm-0 123.00 0.00 492.00 0 492 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 3.00 0.00 12.00 0 12 dm-0 3.00 0.00 12.00 0 12 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 17.00 0.00 640.00 0 640 dm-0 160.00 0.00 640.00 0 640 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 15.00 0.00 600.00 0 600 dm-0 150.00 0.00 600.00 0 600 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 15.00 0.00 232.00 0 232 dm-0 58.00 0.00 232.00 0 232 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 15.00 0.00 544.00 0 544 dm-0 136.00 0.00 544.00 0 544 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 11.00 0.00 92.00 0 92 dm-0 23.00 0.00 92.00 0 92 dm-1 0.00 0.00 0.00 0 0 dm-2 0.00 0.00 0.00 0 0 ... View more

I have no scheduled searches. I just have multiple dashboards in an app I made. Is there any setting that would cause my dashboard searches to behave like a scheduled search? Also, my "events" are more like one line a piece (they're not long at all). Not sure if my Linux admin will be able to help me with disk storage; I asked him to check the health of my Splunk server yesterday and he said everything was "perfect"... is there any other direction I can go in or can anyone give me more specifics on how to check the health of my "disk storage"? Thanks. ... View more

Ok.. I'm assuming this is a Linux admin deal. Is that correct or is what you're talking about ("disk storage working correctly") specific to Splunk? If so, would you be able to direct me to "best-practice" documentation for setting disk storage up? Thanks. ... View more

Job inspector output for one of the slow searches (note that it only took 14 sec. but that's only for the last 15 minutes) This search has completed and has returned 23 results by scanning 309 events in 13.604 seconds. The following messages were returned by the search subsystem: DEBUG: base lispy: [ AND index::sitescope sourcetype::sitescope_daily_log ] DEBUG: search context: user="admin", app="glg", bs-pathname="/opt/splunk/etc" (SID: 1395958659.213) Execution costs Duration (seconds) Component Invocations Input count Output count 0.002 command.fields 4 38 38 0.001 command.presort 1 23 23 0.001 command.rename 1 23 23 13.403 command.search 4 - 38 0.005 command.search.index 4 - - 0.004 command.search.filter 3 - - 0.003 command.search.fieldalias 3 309 309 0.002 command.search.calcfields 3 309 309 0 command.search.index.usec_1_8 50 - - 0 command.search.index.usec_8_64 2 - - 13.382 command.search.kv 3 - - 0.013 command.search.rawdata 3 - - 0.006 command.search.typer 4 38 38 0.004 command.search.tags 4 38 38 0.003 command.search.lookups 3 309 309 0 command.search.summary 4 - - 0.001 command.sort 1 23 23 0.004 command.stats.execute_input 5 - - 0.001 command.stats.execute_output 1 - - 0.001 command.table 1 23 46 0.002 dispatch.check_disk_usage 2 - - 0.056 dispatch.createProviderQueue 1 - - 0.12 dispatch.evaluate 1 - - 0.119 dispatch.evaluate.search 1 - - 0.001 dispatch.evaluate.rename 1 - - 0.001 dispatch.evaluate.sort 1 - - 0.001 dispatch.evaluate.stats 1 - - 0.001 dispatch.evaluate.table 1 - - 13.405 dispatch.fetch 5 - - 0.001 dispatch.preview 1 - - 0.001 dispatch.preview.command.presort 1 23 23 0.001 dispatch.preview.command.rename 1 23 23 0.001 dispatch.preview.command.sort 1 23 23 0.001 dispatch.preview.command.table 1 23 46 0.001 dispatch.preview.stats.execute_output 1 - - 0.001 dispatch.preview.write_results_to_disk 1 - - 13.403 dispatch.stream.local 4 - - 0.016 dispatch.timeline 5 - - 0.013 dispatch.writeStatus 7 - - 0.062 startup.handoff 1 - - Search job properties canSummarize 1 createTime 2014-03-27T16:17:39.000-06:00 cursorTime 2014-03-27T16:02:00.000-06:00 custom {'dispatch.earliest_time': '-15m', 'display.general.type': 'events', 'search': 'index=sitescope sourcetype=sitescope_daily_log sis_monitor_host=* | stats count by sis_monitor_host sis_monitor_status | table sis_monitor_host sis_monitor_status count | sort sis_monitor_status,-count | rename sis_monitor_host as MonitoredHost, sis_monitor_status as "Status", count as "Count"', 'dispatch.latest_time': 'now'} defaultSaveTTL 604800 defaultTTL 600 delegate None diskUsage 237568 dispatchState DONE doneProgress 1.0 dropCount 0 eai:acl {'sharing': 'global', 'app': 'glg', 'modifiable': '1', 'perms': {'read': ['admin'], 'write': ['admin']}, 'owner': 'admin', 'can_write': '1', 'ttl': '600'} earliestTime 2014-03-27T16:02:39.000-06:00 eventAvailableCount 38 eventCount 38 eventFieldCount 30 eventIsStreaming True eventIsTruncated False eventSearch search index=sitescope sourcetype=sitescope_daily_log sis_monitor_host=* eventSorting desc indexEarliestTime 1395957779 indexLatestTime 1395958558 isBatchModeSearch False isDone True isFailed False isFinalized False isGoodSummarizationCandidate 1 isPaused False isPreviewEnabled True isRealTimeSearch False isRemoteTimeline False isSaved False isSavedSearch False isZombie False keywords index::sitescope sis_monitor_host::* sourcetype::sitescope_daily_log label None latestTime 2014-03-27T16:17:39.000-06:00 meanPreviewPeriod 13.604000 messages {'debug': ['base lispy: [ AND index::sitescope sourcetype::sitescope_daily_log ]', 'search context: user="admin", app="glg", bs-pathname="/opt/splunk/etc"']} modifiedTime 2014-03-27T16:18:35.742-06:00 normalizedSearch litsearch index=sitescope sourcetype=sitescope_daily_log sis_monitor_host=* | fields keepcolorder=t "" "bkt" "_cd" "_si" "host" "index" "linecount" "prestats_reserved" "psrsvd_" "sis_monitor_host" "sis_monitor_status" "source" "sourcetype" "splunk_server" numPreviews 1 performance {'dispatch.evaluate.search': {'duration_secs': '0.119000', 'invocations': '1'}, 'command.search.index.usec_1_8': {'invocations': '50'}, 'command.search.typer': {'input_count': '38', 'invocations': '4', 'output_count': '38', 'duration_secs': '0.006000'}, 'command.search.fieldalias': {'input_count': '309', 'invocations': '3', 'output_count': '309', 'duration_secs': '0.003000'}, 'command.search.index': {'duration_secs': '0.005000', 'invocations': '4'}, 'dispatch.evaluate.sort': {'duration_secs': '0.001000', 'invocations': '1'}, 'dispatch.preview.command.sort': {'input_count': '23', 'invocations': '1', 'output_count': '23', 'duration_secs': '0.001000'}, 'command.search.filter': {'duration_secs': '0.004000', 'invocations': '3'}, 'command.search.index.usec_8_64': {'invocations': '2'}, 'dispatch.preview.command.table': {'input_count': '23', 'invocations': '1', 'output_count': '46', 'duration_secs': '0.001000'}, 'dispatch.createProviderQueue': {'duration_secs': '0.056000', 'invocations': '1'}, 'command.rename': {'input_count': '23', 'invocations': '1', 'output_count': '23', 'duration_secs': '0.001000'}, 'command.search.lookups': {'input_count': '309', 'invocations': '3', 'output_count': '309', 'duration_secs': '0.003000'}, 'dispatch.evaluate.rename': {'duration_secs': '0.001000', 'invocations': '1'}, 'command.search.calcfields': {'input_count': '309', 'invocations': '3', 'output_count': '309', 'duration_secs': '0.002000'}, 'command.fields': {'input_count': '38', 'invocations': '4', 'output_count': '38', 'duration_secs': '0.002000'}, 'dispatch.evaluate': {'duration_secs': '0.120000', 'invocations': '1'}, 'command.sort': {'input_count': '23', 'invocations': '1', 'output_count': '23', 'duration_secs': '0.001000'}, 'startup.handoff': {'duration_secs': '0.062000', 'invocations': '1'}, 'command.stats.execute_output': {'duration_secs': '0.001000', 'invocations': '1'}, 'dispatch.preview.command.rename': {'input_count': '23', 'invocations': '1', 'output_count': '23', 'duration_secs': '0.001000'}, 'dispatch.evaluate.stats': {'duration_secs': '0.001000', 'invocations': '1'}, 'command.table': {'input_count': '23', 'invocations': '1', 'output_count': '46', 'duration_secs': '0.001000'}, 'dispatch.check_disk_usage': {'duration_secs': '0.002000', 'invocations': '2'}, 'dispatch.evaluate.table': {'duration_secs': '0.001000', 'invocations': '1'}, 'dispatch.writeStatus': {'duration_secs': '0.013000', 'invocations': '7'}, 'command.search.kv': {'duration_secs': '13.382000', 'invocations': '3'}, 'dispatch.preview.stats.execute_output': {'duration_secs': '0.001000', 'invocations': '1'}, 'command.presort': {'input_count': '23', 'invocations': '1', 'output_count': '23', 'duration_secs': '0.001000'}, 'command.search.tags': {'input_count': '38', 'invocations': '4', 'output_count': '38', 'duration_secs': '0.004000'}, 'dispatch.stream.local': {'duration_secs': '13.403000', 'invocations': '4'}, 'dispatch.preview.write_results_to_disk': {'duration_secs': '0.001000', 'invocations': '1'}, 'command.search.summary': {'invocations': '4'}, 'dispatch.fetch': {'duration_secs': '13.405000', 'invocations': '5'}, 'command.search.rawdata': {'duration_secs': '0.013000', 'invocations': '3'}, 'dispatch.preview': {'duration_secs': '0.001000', 'invocations': '1'}, 'command.stats.execute_input': {'duration_secs': '0.004000', 'invocations': '5'}, 'dispatch.preview.command.presort': {'input_count': '23', 'invocations': '1', 'output_count': '23', 'duration_secs': '0.001000'}, 'command.search': {'input_count': '0', 'invocations': '4', 'output_count': '38', 'duration_secs': '13.403000'}, 'dispatch.timeline': {'duration_secs': '0.016000', 'invocations': '5'}} pid 8822 priority 5 reduceSearch sistats count by sis_monitor_host sis_monitor_status remoteSearch litsearch index=sitescope sourcetype=sitescope_daily_log sis_monitor_host= | fields keepcolorder=t "" "bkt" "_cd" "_si" "host" "index" "linecount" "prestats_reserved" "psrsvd_" "sis_monitor_host" "sis_monitor_status" "source" "sourcetype" "splunk_server" reportSearch stats count by sis_monitor_host sis_monitor_status | table sis_monitor_host sis_monitor_status count | sort sis_monitor_status,-count | rename sis_monitor_host as MonitoredHost, sis_monitor_status as "Status", count as "Count" request {'auto_cancel': '30', 'adhoc_search_level': 'verbose', 'custom.dispatch.earliest_time': '-15m', 'status_buckets': '300', 'rf': '', 'custom.dispatch.latest_time': 'now', 'indexedRealtime': None, 'search': 'search index=sitescope sourcetype=sitescope_daily_log sis_monitor_host=* | stats count by sis_monitor_host sis_monitor_status | table sis_monitor_host sis_monitor_status count | sort sis_monitor_status,-count | rename sis_monitor_host as MonitoredHost, sis_monitor_status as "Status", count as "Count"', 'earliest_time': '-15m', 'custom.search': 'index=sitescope sourcetype=sitescope_daily_log sis_monitor_host=* | stats count by sis_monitor_host sis_monitor_status | table sis_monitor_host sis_monitor_status count | sort sis_monitor_status,-count | rename sis_monitor_host as MonitoredHost, sis_monitor_status as "Status", count as "Count"', 'ui_dispatch_app': 'glg', 'preview': '1', 'latest_time': 'now'} resultCount 23 resultIsStreaming False resultPreviewCount 23 runDuration 13.604 runtime {'auto_cancel': '30', 'auto_pause': '0'} scanCount 309 search search index=sitescope sourcetype=sitescope_daily_log sis_monitor_host=* | stats count by sis_monitor_host sis_monitor_status | table sis_monitor_host sis_monitor_status count | sort sis_monitor_status,-count | rename sis_monitor_host as MonitoredHost, sis_monitor_status as "Status", count as "Count" searchEarliestTime 1395957759.000000000 searchLatestTime 1395958659.000000000 searchProviders ['slc-srv-splunk.glg.local'] sid 1395958659.213 statusBuckets 300 ttl 600 Additional info timeline field summary search.log ... View more

Oh and the data is just a typical log file (every new line is a new event). I can post a sample event if needed but I feel like the data itself is irrelevant to my problem since Splunk is meant for searching through log files. ... View more

I have a time range picker for the timerange of all my dashboards (I've created everything using advanced xml).. one search that takes forever is ...| stats count by x y | table x y count | sort - count | rename ... This search takes a long time (like more than a minute) even for a timerange of the last 15 minutes. It wouldn't make sense that it's an inefficient search b/c I've used searches similar to this many times and it's usually not slow (not to mention this is a pretty simple search). ... View more

oh sorry.. the following is the part I'm unsure about but again I'd really just like to know where I can see all of the different options to include in the header. autocancelinterval="90" onunloadcanceljobs="true" refresh="-1" Also, I'll check out the code. Maybe that will help. I've been using the SideviewUtils app and it's been good for some modules but not all. ... View more

Thanks - I knew I must have been missing something. That works! You should make your comment as an answer so I can mark this question as answered. ... View more

Yeah that's true. I guess it depends on whether they are creating their own app or are changing the navigation menu of a Splunk app in which case you would definitely want to change it in the local folder. ... View more

Maybe you have all of the correct syntax and the correct file in the correct place however if you don't you will have issues with your navigation menu. First you should have your navigation menu in a file called "default.xml". This file should be in $SPLUNK_HOME/etc/apps/ /default/data/ui/nav/. Then in default.xml, you should have something like the following: World ... View more

Thanks a bunch. At first I thought the regex you provided didn't work at all (but I was just being stupid). It still doesn't work for PREAMBLE_REGEX but I discovered I was being to greedy so maybe I can use your answer in transforms.conf. Thanks again! ... View more

Actually, I think I figured out my error! thanks! ... View more

Thanks. I didn't know that < is a special character so that helps. However, I'm still getting the same output. I have been cleaning out the index that is designated for these events every time I try something different in props.conf so it's not that I'm just seeing "old" events. But can you please explain with examples maybe what the difference b/w LINE_BREAKER and MUST_BREAK_AFTER? I'm just confused about why I couldn't use MUST_BREAK_AFTER=</event_change> to tell Splunk that I want a new event starting after the keyword "". ... View more

I've never tried to connect Splunk with Hadoop but with an error such as yours I'd first make sure JAVA_HOME is set to the location of the correct version of java needed. Then I'd find the Hadoop command under the bin directory and literally tell Splunk exactly what directory to look in (HADOOP_HOME='/usr/lib/hadoop/bin/'). Have you read all the sections in Splunk's documentation for this? you are probably just missing something. Try this link. But like I've never actually tried to do what your doing - just a couple of ideas to try. ... View more

Thank you! I think that worked pretty well! One question: how can I use that as a regular expression in say props.conf? I tried this: EXTRACT-Audit Type = '(? [^']*)'\s but that didn't work.. It seems to only work with the rex search command. Is there something I need to modify to use it in props.conf? Thanks! ... View more

@sowings - Thanks, that helps. One last question (remember I'm a newbie with the REST API).. I'm assuming that I don't need to set up a forwarder on the server I want to get REST API calls from since these calls use http methods. Am I correct or is a forwarder still needed for whatever reason? ... View more