How to find the fiscal year Finding a calendar year in splunk is straightforward using the built-in date functions. But sometimes it is important to classify data based on a fiscal year. For instance, the federal fiscal year starts October 1st and ends September 30th. So what is a poor analyst to do? Floor it! Use the floor function to do some math and always get the right number. Let's take a look at an example search. eval FiscalYear=tonumber(theYear) + floor(tonumber(theMonth) / 10) | eval FiscalYear= "FY" + substr(FiscalYear,3,2) For simplicity, let's start with a date of "2019-11-01" and ignore the hours, minutes, etc. This date represents November 01, 2019. Since it is after October 1st, it is within the Fiscal year of 2020, or FY20. We'll extract the 4-digit year and the month as a 2-digit number using strftime, "%Y", and "%M". For details, see https://docs.splunk.com/Documentation/SplunkInvestigate/Current/SearchReference/DateandTimeFunctions. This is well-documented in other threads, so we will act as though you've got this piece figured out. So now, let's say that you parsed your date of interest into the year, as a string named theYear, and the month, as a string named theMonth. Next, it's important to observe one more thing: because we are finding a fiscal year that starts in October, we need to pay attention to conditions where the months are October, November and December. These months have respective numbers of 10, 11, and 12. They are greater than or equal to 10. And this means that we can resort to integer math! Yeah! Let's review the formula again. Here it is: eval FiscalYear=tonumber(theYear) + floor(tonumber(theMonth) / 10) | eval FiscalYear= "FY" + substr(FiscalYear,3,2) The first eval function has two summed terms, tonumber(theYear) and floor(tonumber(theMonth) / 10). The first term is straightforward. It converts the theYear value from a string to a number, and the result, in this case, is 2019. The second term is where the magic happens. Evaluating from inside to outside, we start by converting the month to a number and then dividing by 10. Remember, 10 is the numeric value of October. In this case, we end up with an intermediate value of 11/10, or 1.1. But then we take that value and we apply the floor function to it. This function simply returns a value by ignoring decimal remainders so our result for this term changes from 1.1 to 1. Then we add the two results together to get FiscalYear = 2019 + 1, or FiscalYear = 2020. That's the right number! The second eval method converts the 4-digit fiscal year into a 4-letter acronym for presentation purposes. It suffers from the Y2K problem, but for most reporting—reporting that isn't interested in the twentieth century—it is sufficient. It takes the newly calculated fiscal year, selects the last two digits, and concatenates it with the letters "FY". eval FiscalYear= "FY" + substr(FiscalYear,3,2) So that 2020 becomes "FY20". One last thought: we divided by 10 because we were interested in the federal fiscal year, which starts on October 1st, and 10 is the numeric value for October. But we could have just as easily divided by 7 to start a different fiscal year with July or chosen any month to make a custom fiscal year of our own. ... View more

I installed splunk-sdk-python version 1.6.5 using pip on Mac OS 10.11.6 and have had moderate success cannibalizing example code from the kvstore.py. But now, I've run into an issue. I expect to throw an error when I use "query_by_id" with a nonexistent key. Unfortunately, my stack trace states that the error object called "splunklib.binding.HTTPError" cannot be found. I can see the object being defined in the "binding.py" file. I have lengthy experience coding in Java, but I am switching to Python, so I could be making a rookie mistake without knowing it. Does anyone see the problem? Thanks. This is the error: Traceback (most recent call last): File "./dc2kv.py", line 108, in main kvRecord = json.dumps(collection.data.query_by_id(recID)) File "/Users/dcrocker/anaconda3/lib/python3.6/site-packages/splunklib/client.py", line 3642, in query_by_id return json.loads(self._get(UrlEncoded(str(id))).body.read().decode('utf-8')) File "/Users/dcrocker/anaconda3/lib/python3.6/site-packages/splunklib/client.py", line 3612, in _get return self.service.get(self.path + url, owner=self.owner, app=self.app, sharing=self.sharing, **kwargs) File "/Users/dcrocker/anaconda3/lib/python3.6/site-packages/splunklib/binding.py", line 289, in wrapper return request_fun(self, *args, **kwargs) File "/Users/dcrocker/anaconda3/lib/python3.6/site-packages/splunklib/binding.py", line 71, in new_f val = f(*args, **kwargs) File "/Users/dcrocker/anaconda3/lib/python3.6/site-packages/splunklib/binding.py", line 669, in get response = self.http.get(path, self._auth_headers, **query) File "/Users/dcrocker/anaconda3/lib/python3.6/site-packages/splunklib/binding.py", line 1167, in get return self.request(url, { 'method': "GET", 'headers': headers }) File "/Users/dcrocker/anaconda3/lib/python3.6/site-packages/splunklib/binding.py", line 1228, in request raise HTTPError(response) splunklib.binding.HTTPError: HTTP 404 Not Found -- Could not find object. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./dc2kv.py", line 134, in <module> main() File "./dc2kv.py", line 109, in main except HTTPError as herror: NameError: name 'HTTPError' is not defined This is the code: #!/usr/bin/python -tt # dc2kv.py import sys import requests import json import splunklib.client as client #... def main(): # ... try: kvStoreConn = client.connect(**kwargs) print("Connected to KVStore Successfully") for collection in kvStoreConn.kvstore: print(" %s" % collection.name) collection = kvStoreConn.kvstore[props.get('dcCollection')] #print(json.dumps(collection.data.query())) #kvStoreConn.logout() except AuthenticationError: print("Failed to connect") sys.exit(1) for record in allRecords: recID = record.get('_id') print(recID) kvRecord = 'None' try: kvRecord = json.dumps(collection.data.query_by_id(recID)) except HTTPError as herror: print ('Found HTTPError: ', herror) print('No record found.') if kvRecord == 'None': collection.insert(dumps(recID)) print('Inserted %s', recID) else: collection.update(id = recID, data = record) print('Updated %s', recID) kvStoreConn.logout() sys.exit(1) kvStoreConn.logout() sys.exit(1) if __name__ == '__main__': main() ... View more

We are trying to inject JSON directly into our KV Store instance while using a defined _key inside the JSON object. We are not able to connect because of the following error: 2016-09-23 10:25:01 ERROR stderr:71 - com.splunk.HttpException: HTTP 400 -- <?xml version="1.0" encoding="UTF-8"?> 2016-09-23 10:25:01 ERROR stderr:71 - <response> 2016-09-23 10:25:01 ERROR stderr:71 - <messages> 2016-09-23 10:25:01 ERROR stderr:71 - <msg type="ERROR">Must supply 'Content-Type' header set to 'application/json'</msg> 2016-09-23 10:25:01 ERROR stderr:71 - </messages> 2016-09-23 10:25:01 ERROR stderr:71 - </response> We have looked for a way to add message headers to the RequestMessage object but we do not see a method in the source. Where do we inject message headers? The method breaks on the following line: response = service.send(kvStorePath, requestMessage); Below is our method: private void writeToKVStore(File uglyJsonFile) throws IOException { String nextJSON = ""; ResponseMessage response = null; BufferedReader br = null; ServiceArgs serviceArgs = null; Service service = null; RequestMessage requestMessage = null; int port = params.getKvStorePort(); String kvStorePath = params.getKvStorePath(); String kvStoreScheme = params.getKvStoreScheme(); String kvStoreUsername = params.getkvStoreUsername(); String kvStorePassword = params.getkvStorePassword(); String kvStoreHost = params.getKvStoreHost(); String kvStoreApp = params.getKvStoreApp(); try { HttpService.setSslSecurityProtocol(SSLSecurityProtocol.TLSv1_2); serviceArgs = new ServiceArgs(); serviceArgs.setScheme(kvStoreScheme); serviceArgs.setUsername(kvStoreUsername); serviceArgs.setPassword(kvStorePassword); serviceArgs.setHost(kvStoreHost); serviceArgs.setPort(port); serviceArgs.setApp(kvStoreApp); // kvStoreApp = "kvstore" service = Service.connect(serviceArgs); // TODO set up requestMessage // Must supply 'Content-Type' header set to 'application/json' requestMessage = new RequestMessage("POST"); // service.send(params.getKvStorePath(), requestMessage); br = new BufferedReader(new FileReader(uglyJsonFile)); while ((nextJSON = br.readLine()) != null) { requestMessage.setContent(nextJSON); response = service.send(kvStorePath, requestMessage); } } catch (IOException e) { LOGGER.error(e.getMessage()); e.printStackTrace(); } catch (Exception ex) { LOGGER.error(ex.getMessage()); ex.printStackTrace(); if (response != null) { String nextLine = ""; LOGGER.error("Status: " + response.getStatus()); BufferedReader responseContent = new BufferedReader(new InputStreamReader(response.getContent(), "UTF-8")); while((nextLine = responseContent.readLine()) != null) { LOGGER.error(nextLine); } } } finally { try { if (br != null) { br.close(); } } catch (IOException e) { LOGGER.error(e.getMessage()); e.printStackTrace(); throw e; } } } ... View more