The most likely issue, since the logfiles reside on your indexer, is that the input is not set correctly. Try this search index=_internal sourcetype=splunkd NOT log_level=INFO This is splunks internal log, and should tell you if there are any problems with the logfiles you are generating. Especially of interest are messages with the filenames of the logfiles you think you have set up, and messages with TailingProcessor in them. Also check the settings of your input and make sure that they are set correctly. ... View more

Is $id is actually set? My guess would be that splunk is running curl -s -H "X-Requested-With: Curl Sample" -b "path=/api; secure" -u "TRUCNATED"https://TRUNCATED&id=" which probably gives you no body. ... View more

Cool thanks! ... View more

Inputs is the first section of the documentation you linked to: http://dev.splunk.com/view/java-sdk/SP-CAAAEJ2#inputs If you are unclear on the difference between submitting data direvtly to an index and using an index, I encourage you to drop your development work and run through the splunk tutorial. It's important that you understand the non SDK methods of getting data into splunk before writing your own application. Have you tried getting data into splunk using the universal forwarder or a syslog listener? A syslog input can handle millions of events an hour if you have enough indexers, so you could just use log4j instead of creating your own library! ... View more

All the answers are in the documentation. It's fairly involved so I recommend having a good read through it. It's not as simple as adding A records to DNS as you would for simple web servers, as there is a great deal of state replication (not to mention the distribution of scheduled searches between cluster members) The benefit of doing this work is great though. You get reliable failover and replication, and it doesn't cost you a cent extra. In my experience, adding members to a cluster seems to be fairly easy once the cluster is set up, so you can scale out easily in the future. Also I strongly recommend putting a load balancer in front of your cluster instead of a round robin dns entry. Check out HAproxy if you don't have one already. ... View more

When you POST to /services/search/jobs, the response from the splunk server should be a search id (sid). You will then need to do a GET request to /services/search/jobs//results to get the results from splunk (when they are ready) Alternatively you can use the /services/search/jobs/export endpoint. This will stream the results back to you. ... View more

What you want is search head clustering. It will replicate knowledge objects between search heads. Unfortunately you're going to need another search head, as you need at least three in a cluster. Check out the docs on search head clustering: http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/AboutSHC ... View more

Whats in the HTTP response? EG: Do you get any response at all from Splunk or are your requests just timing out. ... View more

Can you put in a screenshot of that? It seems pretty big ... View more

You're going to get 3 & 5 as well though. Is that ok? ... View more

What does the _internal index say about those SIDS ? ... View more

No problem. Do you mind accepting the answer? To help future forum users 🙂 Cheers, JP ... View more

Are we still using your example data in your question? I suggested you append a search command to filter down the list, but I'm not sure how you want to filter it. Are you trying to exclude all but a single commit hash? Or do you want to use the commit hashes to search a different log? I'm confused. Originally you said you want records 4 & 6 but now you say you want all the records? Can you provide an example of what you want the result to be? Based on the example data you originally provided. Of course if you just want to use the commit hashes to search another source then make the search string a subsearch. EG sourcetype=my_other_log [search sourcetype=my_log (example.com:9442 OR example.com:9441) (131676 OR Replication) | rex field=_raw "^.*\[(?<commit_hash>\w.*?)\]" | dedup commit_hash | fields commit_hash | rename commit_hash as query] (The rename command at the end of the subsearch removes the 'commit_hash=' from the returned values. See http://docs.splunk.com/Documentation/Splunk/6.3.1511/Search/Changetheformatofsubsearchresults) ... View more

The timestamp issue is probably because Splunk doesn't allow dates from that far in the past without modification. I notice you are adding the data directly to an index. I recommend that you use an input rather than submitting to an index so you can easily change the input settings without changing your sourcecode. Otherwise I'd say you just need to submit the args properly - Have you tried listening to the normal submit event example in the docs and seeing the format of the sting that gets sent? ... View more

Make sure you are supply a value for the field to search, eg: ...| search commit_hash=<some value here> Also stats values(_raw) was just an example. It's probably better to run ...| stats values(*) as * by commit_hash That will make sure you get all the fields, so you can do further aggregation commands. Have a search through splunk ansers for stats values and chart values for some great examples. ... View more

You're using windows? Then you want %SPLUNK_HOME%\etc\apps\search\appserver\static SPLUNK_HOME will be where ever you installed splunk. For example the full path might be: C:\Program Files\Splunk\etc\apps\search\appserver\static Most people here will assume you're running on linux unless you specify otherwise ... View more

You won't have to update the search peer list on the search heads. If you don't have a search head cluster or an index cluster then you don't need to do it ... View more

Interesting question. I'm not sure that you can with Splunk's cron notation. If the search doesn't consume many resources, just run it every hour. Otherwise I suppose you could: Make a saved search Use cron or a bash script on a linux host to run saved/searches/{name}/dispatch http://docs.splunk.com/Documentation/Splunk/6.3.1511/RESTREF/RESTsearch#saved.2Fsearches.2F.7Bname.7D.2Fdispatch Check out http://stackoverflow.com/questions/1417098/cronjob-every-25-hours I'm curious: what use case do you have for this search? ... View more

You could eval a bounding box based off a radius input. But don't forget the distance between degrees of longitude and latitude changes depending on where you are. EG at the north pole there is essentially zero distance between any longitude line. http://factually.gizmodo.com/how-precise-is-one-degree-of-longitude-or-latitude-1631241162 So you're going to have to have a pretty complicated expression. Also you might want to check out the google apis: https://developers.google.com/places/. It might be easier to do the geolocation alerting outside of splunk ... View more

Do you get results for sourcetype=my_log (example.com:9442 OR example.com:9441) (131676 OR Replication) | rex field=_raw "^.*\[(?<commit_hash>\w.*?)\]" | stats values(_raw) by commit_hash ? ... View more