The following query construct populates a summary index: source=1.log OR source=2.log | eval _time = case(source == "1.log", _time)| stats first(_time) as _time ….other fileds…. dc(source) as dc by id|search dc=2 The resolution _time of the time stamp for each source log is in milliseconds. Example : 2013-06-13 04:00:15,250 Question: Why isn’t the resolution time in the summary index in mill. seconds (e.g., 2013-06-13 04:00:15 +0000)? ... View more

Let's say, I have 5 forwarders. 4 of them are allowed to forward events to the indexer but one of them is not. How can I Blacklist this host at the indexer not at the forwarder or network (eg., iptables)? In this way, no log event should be index from the host that is not allowed to... Thanks, Lp ... View more

Is there a way where I do not have to restart splunk to enable a new custom search command? How to reload commands.conf without restart... Thanks, Lp ... View more

The following query is able to join two source logs where one of the source logs is in json format: (sourcetype="request" AND application=vsp NOT (Agent.007) key_name1 ) OR (sourcetype="response" key_name2) | spath | spath path=your_json_path output=your_output_key_name1 | spath path=your_json_path output=your_output_key_name2 | spath path=your_json_path output=your_output_key_name3 ... | spath path=your_json_path output=your_output_key_name4 stats first(your_output_key_name1) as your_output_key_name1 first(your_output_key_name2) as your_output_key_name2 first(your_output_key_name3) as your_output_key_name3 first(your_output_key_name4) as your_output_key_name4 first(key_name1) as key1 list(key_name2) as key2 dc(sourcetype) as dc by id Problem: JSon path could vary. Therefore, output variables too. Is there a way in Splunk that these could be discovered dynamically? Regards, Lp ... View more

I have the following function that fails when there is not any results returned from the saved search: Saved search results: Stats: { "preview": false, "init_offset": 0, "messages": [], "fields": [], "rows": [] } private static void rss_wc_failed_azkaban_jobs(String s,String sid,ResultsReaderJson rr) throws Exception { HashMap<String, String> event; int total=0,failed=0; while ((event = rr.getNextEvent()) != null ) { if(total>1000) { System.out.println("infinte loop found, quitting"); break; } System.out.println("Event: "+event.size()+": "+event.toString()); total++; if(event.get("Last State").equalsIgnoreCase("failed")) { failed++; } } String title="Service in normal operation state"; if(failed>0) { title="Service with Performance issues"; } if(failed>=5) { title="Service disrupted"; } String desc="Number of jobs with failed state: "+failed; writeRSS(s,title,sid,desc); } The function is called as follow: else if(s.equalsIgnoreCase("rss_wc_failed_azkaban_jobs")){ rss_wc_failed_azkaban_jobs(s,sid,rr); This is the java dump the shows the line where it fails in "com.splunk.ResultsReaderJson" 2013-04-25 07:39:03 Full thread dump OpenJDK 64-Bit Server VM (19.0-b09 mixed mode): "Low Memory Detector" daemon prio=10 tid=0x00007fae0009d800 nid=0x7032 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "CompilerThread1" daemon prio=10 tid=0x00007fae0009b000 nid=0x7031 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "CompilerThread0" daemon prio=10 tid=0x00007fae00098000 nid=0x7030 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00007fae00097000 nid=0x702f waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00007fae00078800 nid=0x702e in Object.wait() [0x00007fadf7af9000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x000000060cdb7528> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) - locked <0x000000060cdb7528> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) "Reference Handler" daemon prio=10 tid=0x00007fae00076800 nid=0x702d in Object.wait() [0x00007fadf7bfa000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x000000060cdb74e8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x000000060cdb74e8> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x00007fae00007000 nid=0x7027 runnable [0x00007fae061dd000] java.lang.Thread.State: RUNNABLE at com.splunk.ResultsReaderJson.readEvent(ResultsReaderJson.java:298) at com.splunk.ResultsReaderJson.getNextEventInCurrentSet(ResultsReaderJson.java:235) at com.splunk.ResultsReader.getNextElement(ResultsReader.java:89) at com.splunk.ResultsReader.getNextEvent(ResultsReader.java:66) at Splunk.rss_wc_failed_azkaban_jobs(Splunk.java:624) -> Code 2 at Splunk.main(Splunk.java:132) -> Code 1 "VM Thread" prio=10 tid=0x00007fae0006f800 nid=0x702c runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fae00012000 nid=0x7028 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fae00014000 nid=0x7029 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fae00016000 nid=0x702a runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fae00017800 nid=0x702b runnable "VM Periodic Task Thread" prio=10 tid=0x00007fae000a0000 nid=0x7033 waiting on condition JNI global references: 1279 Heap PSYoungGen total 178240K, used 170434K [0x00000007580b0000, 0x0000000769020000, 0x0000000800000000) eden space 102208K, 92% used [0x00000007580b0000,0x000000075dce66e0,0x000000075e480000) from space 76032K, 99% used [0x000000075e6c0000,0x00000007630fa470,0x0000000763100000) to space 87872K, 0% used [0x0000000763a50000,0x0000000763a50000,0x0000000769020000) PSOldGen total 343936K, used 81948K [0x0000000608200000, 0x000000061d1e0000, 0x00000007580b0000) object space 343936K, 23% used [0x0000000608200000,0x000000060d2071d0,0x000000061d1e0000) PSPermGen total 21248K, used 10124K [0x00000005fdc00000, 0x00000005ff0c0000, 0x0000000608200000) object space 21248K, 47% used [0x00000005fdc00000,0x00000005fe5e3268,0x00000005ff0c0000) ... View more

I have a query that is able to join two or more source types with the same log format in each source log (all log with key=value pair format). However, I am having difficulties to join two or more source types if the source logs are not in the same format (e.g., one source log is in json format and the other is in key=value pair format). All logs are joined by the id key name found in each source log. This is main construct to join two source types with the same key=value pair format: (sourcetype="request" AND application=vsp NOT (Agent.007) ) OR (sourcetype="response") stats first(key_name1) as key1 list(key_name2) as key2 dc(sourcetype) as dc by id| search dc=2 Can anyone share any example to join one source log with json format and another source log with key=value pair format a common key name? any idea? Thanks, Lp ... View more

I need to back fill an index from a scheduled search but the result set of the scheduled search is quite large. Therefore, the scheduled search is set up to run hourly. I do not want to increase the amount of "srchDiskQuota" in authorized.conf. Is there way to instruct Splunk to delete the scheduled search file in "/opt/splunk/var/run/splunk/dispatch" just after it completed the index process. Any idea ? Thanks, Lp ... View more

I have the following log event : 2013-03-12 10:37:10,205 { "start" : 1, "returned" : 1, "count" : 1, "entities" : [ { "houses" : { "callers" : "IM", "placeid" : 5041447014850446107, "number" : 14, "sourceid" : 5625 }, "entitytype" : "house/street", "title" : [ { "default" : "No Place" } ] } ] } With the following query: | rex field=_raw mode=sed "s/[rn]//g" | rex "<tvsquery id="(?&lt;id">[^>]+)>(?<response_temp>.+?)</tvsquery>" | eval result="{\""."TimeStamp"."\"".":"."\""._time."\"".","."\""."id"."\"".":"."\"".id."\"".","."\""."response"."\"".":".response_temp."}" |table result I am able to get the following result set: result {"TimeStamp":"1364556697.631","id":"58b6bf4d-948b-416b-8d17-cedcbc1059ec","response":{ "start" : 1, "returned" : 1, "count" : 1, "entities" : [ { "houses" : { "callers" : "IM", "placeid" : 5041447014850446107, "number" : 14, "sourceid" : 5625 }, "entitytype" : "house/street", "title" : [ { "default" : "No Place" } ] } ] }} Problem: How can index the value of result in a new summary index? For this particular example I just need this: {"TimeStamp":"1364556697.631","id":"58b6bf4d-948b-416b-8d17-cedcbc1059ec","response":{ "start" : 1, "returned" : 1, "count" : 1, "entities" : [ { "houses" : { "callers" : "IM", "placeid" : 5041447014850446107, "number" : 14, "sourceid" : 5625 }, "entitytype" : "house/street", "title" : [ { "default" : "No Place" } ] } ] }} I do not want this in the index: result = "{"TimeStamp":"1364556697.631","id":"58b6bf4d-948b-416b-8d17-cedcbc1059ec","response":{ "start" : 1, "returned" : 1, "count" : 1, "entities" : [ { "houses" : { "callers" : "IM", "placeid" : 5041447014850446107, "number" : 14, "sourceid" : 5625 }, "entitytype" : "house/street", "title" : [ { "default" : "No Place" } ] } ] }}" Thanks, Lp ... View more

The following URI returns the metadata information related to a saved search named "test" found in application "search": https://myhost:8089/servicesNS/userid/search/saved/searches/test I would like to know what will be the URI to run the saved search "test" found in application "search" where the returned data format is in json. Thanks, Lp ... View more

Let's say we have the following 3 logs sources: request.log : timestamp id=123 q=1 filter=2 query_time="timestamp" response.log: timestamp id=123 q="{1}" response="hello world" performance.log: timestamp id=123 responsetime="1 ms" Then, We need to join these 3 logs with the following query: (request.log) OR (response.log) OR (performance.log)| stats first(q) as q first(query_time) as _time first(response) as response first(responsetime) as responsetime dc(sourcetype) as dc by id|search dc=3 Based on these premises, is there a way in splunk to define the name space of variables? Example: q.response.log q.request.log In this way, I will have the following query construct: (request.log) OR (response.log) OR (performance.log)| stats first(q.request.log) as q_req first(q.response.log) as q_res first(query_time) as _time first(response) as response first(responsetime) as responsetime dc(sourcetype) as dc by id|search dc=3 Any idea? Thanks, Lp ... View more