About dpadams

dpadams · ‎12-26-2011

I've got a collection of Web log data where we like to see the URLs counted by host: sourcetype="access_common" | chart count over uri_path by host We've got a couple of different server types handling different sorts of URLs, so the table we get out of this query is a helpful way to see the URLs-by-machine very easily. You get a table with the uri_path (URL) listed in the left-hand column and then one column for each of our servers. So: uri_path Server_1 Server_2 /action/do_this 897 12345 /action/do_that 23456 18001 The "Server_1" and "Server_2" names are extracted from the host field for the event and the counts are calculated by Splunk. Can anyone tell me how to format the resulting numbers? Ideally, I'd like to use commas in large numbers and put a - or an empty string for zeros. I've been eperimenting with fieldFormat, eval, and tostring and, in this case, had no luck. I think part of the problem is that I don't know how to address the generated columns. They're not fields, they're values extracted by fields. If I try applying formatting to "Server_1" and so on (not an ideal solution) it doesn't work. Is there a way to apply numeric formatting to an extracted/calculated series like this? Thanks for any suggestions or pointers.

dpadams · ‎01-19-2011

I've been struggling a bit with external lookups. It's potentially a fantastically useful feature, but I've been hampered largely by a lack of Python knowledge and a shortage of good examples. (The docs are decent and I have found some examples, but not enough for me with my limited knowledge of Python and Splunk.) I've got nothing against learning Python, but also don't have the time to do it. It occurred to me that a nice feature would be to add URL-based lookups as an external lookup system. Then, you could integrate any existing service that deals with HTTP name-value pairs. If you need to write a new back-end service and have a Web/application server, you can use any language you like to make it all work. Again, nothing against Python apart from a shortage of time. Hopefully, this feature would make external lookups accessible to a broader range of people than they are today. I've written up a fake props.conf and transforms.conf (below) to show how such a feature might work on the configuration side. I'm not quite sure how to submit feature requests for Splunk. I submitted an 'enhancement request' but, without a license (still in the eval phase), I'm not sure if my submissions get any attention (18 days and counting on my one tech support question.) So, I'm posting the idea here for comment and tagging it 'feature-request'. Thanks for any suggestions or comments. If I've missed an existing feature that will do what I need, I love to learn about it. -- Thanks props.conf sample below: [application_log] # Take application_log to be a custom souretype. lookup_table = user_market_lookup appuser_uuid OUTPUT market_id, market_name transforms.conf sample below: [user_market_lookup] # The external URL is specified below. # ID is the name that will reach the exteranl Web/app server. # appuser_uuid is a Splunk field name and is filled in automatically # when this script is invoked by Splunk as an external lookup. # Note: It's easy enough to put HTTP Basic 'credentials' into a URL # but some people might want additional security. I don't need it myself # at the moment. It would be good to support SSL in the connection. external_url = https://www.example.com/GetMarketDataFromUserUUID?ID=appuser_uuid # I supposed you could support POST as well as GET external_type = get # The Web/app server returns name-value pairs, like # market_id=5&market_name=North%20Dakota # Splunk should convert these into fields using the # names specified in the fields_list and extract # the matching values from the Web results. So: # market_id = 5 # market_name = North Dakota fields_list = market_id, market_name # Often enough, a log line with have no entry or a value such as - # instead of a meaningful field value. Splunk recognizes this # in search results as it is. It would be nice if the external # lookup could be short-circuited to avoid needless calls to the # Web/app server. You could have a default value returned without # making the external call. The default string could be a "", # a meaningful default, an error message or any other string. # Some nice optional directives for this could go like so: result_if_input_equals_none = "" result_if_input_is_empty = "" # I'm happy with URL-encoded name-value pairs but others # might want JSON or XML. You could have an optional # format declaration with constants like # XML, JSON, or HTTP name_value_format = http

dpadams · ‎01-09-2011

I think I'm actually left with possibly two problems: 1) Something is wrong in my configuration statements as the field does not show up in the field list or picker. 2) I may be returning the data in the wrong format. I'd like to return the wrong data but have also tried MarketID+CRLF+data+CRLF Getting as far as seeing the requests come in and the response go out is a huge win, but I'd still be grateful for help figuring out how to get everything wired into Splunk properly. P.S. I don't see that it matters but I put my script into $SPLUNK_HOME/etc/searchscripts/user_to_market_id.py

dpadams · ‎01-09-2011

Added a stanza to props.conf tied to the sourcetype, like [int_web_log] lookup_table = marketIDLookup user_id OUTPUT MarketID AS market_id Added a stanza to transforms.conf, like: [marketIDLookup] external_cmd = user_to_market_id.py user_id MarketID external_type = python fields_list = user_id, MarketID Restarted Splunk. When I load data from the int_web_log sourcetype, the new field does not show up in the list. I've experimented with trying to do the lookup directly from the search bar using a pipe, but get no results their either.

dpadams · ‎01-09-2011

Thanks for the code! That's a huge help, thanks very much. I've adapted what you posted and Splunk is now calling my Web interface correctly. I can see the requests coming in with the right arguments and am able to respond. Right now, nothing is showing up in Splunk, but I can see that the data is going over the wire. So, it looks like I still have a Splunk configuration step done incorrectly. I've been going through samples and the docs and have done the following:

dpadams · ‎01-09-2011

I've been looking at Splunk's external lookup features and they sound ideal for several of my logs. For example, I've got a log with a user ID where I'd like to be able to do counts based on their sales region. I can easily get the region name or ID from the user by calling a simple URL like the illustrative example below: http://www.foo.com/splunk/lookup/GetSalesRegionNameFromUserID?user_id=1b4545x5 With a result of an empty string or an ID. Can anyone point me to an example that uses HTTP GET or POST to call a URL to satisfy an external lookup? As far as the various configuration file modifications and packaging requirements, I figure I'll use example from the following article by Nimish Doshi as a starting point: http://blogs.splunk.com/2009/09/14/enriching-data-with-db-lookups-part-2/ That article use SQL calls but, otherwise, the idea is much the same. It might be relevant to mention that I'm pretty clueless about Python but that I have total control over the Web interface. (The URL, inputs and outputs.) I've already written and tested the Web+DB side of this, I just need to figure out how to call the URL from Python and get the inputs/outputs sorted out with Splunk. Getting a working example would be incredibly helpful to me as, starting from that, I could build out many other similar functions. Thanks in advance for any help, pointers, or suggestions. P.S. I looked at GET workflow actions but they seem to be about adding interactivity to search results. What I'm after right now is the ability to do statistics based on external lookup field values as well as improve the meaningfulness of log entries in results.

dpadams · ‎01-09-2011

Just to close this question out, I thought I'd report back. Our Splunk server is on a virtualized copy of Windows so we imaged the machine, upgraded it and restarted. That took less than half an hour and at the end of it, only the IP address had changed. So, in this particular case, the checklist was: Shut down Splunk on the old server, new server, and forwarders. Reconfigure inputs.conf on the new server to match the new machine's address. Start Splunk on the new server, log in and check that the data and other elements are there. Reconfigure outputs.conf to point to the new server's address. Distribute the customized outputs.conf to the forwarders. Restart the forwarders. Check that the forwarders are working, either by checking splunkd.log locally or (what I did), letting it run for a bit and then checking the last fifteen minutes of events. All of the forwarder host addresses should show up. It was pretty quick and painless, but this is about a simple an upgrade as you could hope for. Specifically, no files were actually moved and all of the forwarders are reading from physical logs, so no ephemeral events are lost during the downtime. Getting Splunk onto a better piece of (virtual) hardware has really improved the search speed on our roughly 8,000,000 events.

dpadams · ‎01-09-2011

command.search dispatch.fetch dispatch.timeline It seems that the point of a field extraction is not so much to speed up a search. Is the idea to make the search statements clearer and to make reports possible? For example, it's easy to imagine a search that then wants to sum/count/top by a custom field such as internal_user_id or sales_region. Do you reckon I'm on the right track here? I've started to get religion about Splunk but am still pretty new to it. I've been at it for a few weeks but most of the time has been spent prepping logs, configuration files and automating distribution.

dpadams · ‎01-09-2011

Thanks very much for your answer - I've been enjoying some of your videos recently. I had not noticed the search inspector, it looks great. I've tried out a small comparison and am unclear what the results indicate. I've got a JSON payload in my logs in an extract field called json_post_data. If I want to find the phrase "red rover", I can do a free search for "red rover" or a field-specific search for json_post_data="red rover". The search inspector shows somewhat different execution paths for the two searches, but overall similar performance. The most time consuming elements appear in both

dpadams · ‎01-09-2011

Error below - I've added a line break between the lines for clarity. Error : Traceback: Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\jsonutils\bin\jsonkvrecursive.py", line 53, in handle_dict(r, json.loads(json_text)) File "C:\Program Files\Splunk\Python-2.6\Lib\json\__init__.py", line 307, in loads return _default_decoder.decode(s) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 319, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 336, in raw_decode obj, end = self._scanner.iterscan(s, **kw).next() File "C:\Program Files\Splunk\Python-2.6\Lib\json\scanner.py", line 55, in iterscan rval, next_pos = action(m, context) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 171, in JSONObject raise ValueError(errmsg("Expecting property name", s, end)) ValueError: Expecting property name: line 1 column 1 (char 1) Error : Traceback: Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\jsonutils\bin\jsonkvrecursive.py", line 51, in json_text = raw[ raw.index( '{' ) : raw.rindex( '}' )+1 ] ValueError: substring not found

dpadams · ‎01-09-2011

The error is too long for comments, I'll add an answer instead.

dpadams · ‎01-08-2011

No, there aren't any new fields in the field picker. Could it be because the source data is quoted? "{'foo':'bar'}"? I quoted the text to make it easier to insert the POST payload into a single line log format. For jsonkvrecursive, there's what seems to be a traceback stack dump (or something of that sort) that I'll quote in a second comment.

dpadams · ‎01-08-2011

Thanks very much for answering. I'm still not quite there yet. I've tried piping a search through jsonkv but don't get any visible change in my results. jsonkvrecursive throws a series of errors and gives up. Could the problem be that my data includes a JSON array, not only simple name-value pairs? Imagine I've got an array of favorite foods. In my idea world, I'd like to be able to find events where "banana" is a favorite food and then get the top ten most commonly occurring foods in those events. Does that sound even remotely possible? Thanks for any help.

dpadams · ‎01-07-2011

After some completely undignified trial-and-error with a simpler sample set, I've got a pattern that works in Splunk: (?i)^(?:[^"]*"){3}(?P[^"]+) This captures the {any contents} and not the double-quotes around the outside. For example, starting from the text below: "{'user_id':'abc123','action':'add','names':['hello','world']}" the pattern should match {'user_id':'abc123','action':'add','names':['hello','world']} Depending on your your whitespace is organized, the above pattern may need a tweak or two.

dpadams · ‎01-07-2011

I've got a custom log format using a format similar to an Apache access log but with different data. I've used the interactive field extractor to teach Splunk all of the fields in the table. As my Splunk is quite slow, I'm wondering if there's a speed impact from each field extraction rule. Should field extraction rules degrade search performance? If so, can anyone point me to the best way to optimize this? As far as I could see from the docs, the normal practice is to define field extractions this way rather than at index time. (That seems backwards to me as a database person but the docs seemed pretty clear on this point.) Thanks for any guidance, still finding my feet with Splunk.

dpadams · ‎01-06-2011

The jsonutils application sounds like it may help considerably with my current project as we're POSTing a lot of JSON. (Likewise, xmlutils could help with some of my other projects.) I'm new enough to Splunk to have a basic question: How do I use jsonutils in a search? Do I need to use jsonutils to build some field extraction rules in a config file, or is it possible to use jsonutils in search statements on the fly? I'd also be curious to figure out how to extract a JSON payload in its entirity from a POST. I've got the raw regex pattern I need but haven't figured out how to adapt it to Splunk. I've submitted a question about this already: http://answers.splunk.com/questions/10363/extracting-json-from-post-data. Any would be very much appreciated! Also, thanks to vbumgarner for writing and sharing these apps.

dpadams · ‎01-06-2011

Thanks very much, that's as easy as I'd hoped. Apologies for not finding this in the documentation myself. I'd already customized props.conf to understand the sourcetype, but didn't now about the [yoursourcetypehere] stanza.

dpadams · ‎01-06-2011

I've got log data that includes JSON text that's sent up using POST to a Web server. A raw regex pattern to match the JSON data is: \{([^}]*)\} That gets me the curly braces and everything inside. Perfect. Unfortunately, I've been unable to figure out how to adapt this to field extraction system's patterns. I've not managed to get Splunk to recognize my POSTed JSON data using the interactive field extraction screen, which is why I'm trying to get the raw regex into place. Thanks very much for any assistance.

dpadams · ‎01-06-2011

Thanks for the confirmation on this point. All of the Windows machines are virtualized already. We're OS X/Windows developers and haven't felt like putting a linux machine on-line...but are considering it now.

dpadams · ‎01-06-2011

I've got Splunk configured to assign some custom sourcetypes to files when they're uploaded automatically from a watch directory. Some users would prefer to upload files by hand using the Splunk GUI and I want to make sure that they assign the correct sourcetype. They can do so by selecting "manual" for the sourcetype and then typing in the correct value. Unfortunately, this makes it easy to make a mistake. Is it possible to customize the list of sourcetypes in the 'automatic' list shown in the Splunk Web GUI? I've googled a bit and looked around in the manuals but didn't follow if this is possible or not. Thanks for any help.

dpadams · ‎01-05-2011

I'm evaluating Splunk and have the server running on Windows. I'd like to be able to send emails with PDF results, but this appears to be a linux-only feature. Is there any workaround to this limitation? I have full control of the Windows machine and can install CutePDF, a command-line PDF printer, a Python library, or anything else that might be required. If the answer is "no", does anyone know if PDF support on Windows has been announced as being on the Splunk roadmap somewhere? Thanks very much.

dpadams · ‎01-05-2011

I've been experimenting with Splunk for a few weeks and now have nine machines working as light forwarders sending several logs each over TCP. So far, so good - but Splunk is quite slow on the target server. Before I try and move the installation to a new machine, I'd like to double-check that I understand the steps. Briefly: Shut down Splunk on the old server, new server, and forwarders. Copy all or part of /Splunk/var/lib/splunk/ to the new machine. Reconfigure my customized outputs.conf to point to the new server. Distribute the updated outputs.conf to the forwarders. Start the new server and see that it looks okay. Restart the forwarders and see that they look okay too, either by checking the splunkd.log locally or by checking that they all seem to be posting events to Splunk. If I've got anything wrong there or have skipped as step, I'd be grateful for advice. Also, I'm not sure of the following: What exactly do I need to copy from /Splunk/var/lib/splunk/? Everything? Where are custom searches and reports stored? I don't have many but figure I should sort this part out. I've saved these through the Splunk Web GUI and have not hand-edited, moved (or even found) any savesearches.conf files or the like. We're on Win32 right now and are contemplating moving either to a fast Win machine or a Linux machine. Any serious gotchas to be aware of there? Thanks again for any help.

dpadams · ‎12-23-2010

Thanks for the clarification. I'll pass on trying to reverse engineer an undocumented internal format!

dpadams · ‎12-23-2010

Thanks for the suggestions. Yes, writing the socket code directly is probably just asking for trouble. Thanks for the reference to scripted inputs as I hadn't considered them. The environment I'm in for this job doesn't naturally talk to stdin/stdout so I'll do something else, but I like the idea of scripted inputs very much. In this case, I think I'll have one client connect to the database, set a semaphore that it's processing the error log table and then proceed to push out errors to a physical file. Splunk can monitor it normally.

dpadams · ‎12-22-2010

I've got a basic Splunk setup to consolidate four different Web logs from eight machines running two Web servers each. As a next step, I'd like to integrate in output from an internal error log that's stored centrally in a database. I could write the data out to a log and let a regular Splunk forwarder handle everything. While that's likely to be what I do, I thought I'd ask if there's any sense to writing a custom bit of code to forward the data directly to Splunk from my application without writing to a physical log. I'm new enough to Splunk that I didn't manage to find the relevant docs. I pushed an existing log into Splunks bulk upload folder and traced out the send data and it looks as though: Depending on how the server is configured, the data can be sent up just as it is in the log (raw) or pre-parsed (splunk-cooked-mode-v2, in my quick test.) I'd use a raw format to keep my code simple. There are at least a handful of meta-data name-value pairs (or something) at the start of the message, even in raw mode. At a quick glance, the raw mode format looks like it's mostly a simple stream of data from the log to Splunk. Can anyone point me to some advice, documentation, blogs, etc. on the pros/cons/details of writing custom code to push log data directly to Splunk over TCP? Thanks very much. I'm new to Splunk and have already gotten some great answers from Splunkanswers members. Much appreciated!

Posts	55
Solutions	2
Karma Given	1
Karma Received	15
Member Since	‎12-21-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

Forwarder and props.conf troubleshooting

Problem overriding host on an indexer

Is it safe to downgrade from 4.2.5 to 4.2.2?

Migrating a stand-alone linux server to a shared L...

Extracting stats from different fields in the same...

Can you customize the heatmap colors?

How can you format numbers in a ctable?

Is there an efficient way to learn Splunk?

Can you create a dashboard with an adjustable time...

How do you format numbers produced by 'count over'...

How do you format numbers produced by 'count over'...

Feature Request: Support GET and name-value pairs ...

Re: Example of doing an external lookup using HTTP...

Re: Example of doing an external lookup using HTTP...

Re: Example of doing an external lookup using HTTP...

Example of doing an external lookup using HTTP GET...

Re: Splunk migration: Tips for moving data, saved ...

Re: Speed impact of field extractions?

Re: Speed impact of field extractions?

Re: Example of jsonutils add-on use in a search?

Re: Example of jsonutils add-on use in a search?

Re: Example of jsonutils add-on use in a search?

Re: Example of jsonutils add-on use in a search?

Re: Extracting JSON from POST data

Speed impact of field extractions?

Example of jsonutils add-on use in a search?

Re: Is it possible to customize the automatic sour...

Extracting JSON from POST data

Re: Is there any way to print reports to PDF on Wi...

Is it possible to customize the automatic sourcety...

Is there any way to print reports to PDF on Window...

Splunk migration: Tips for moving data, saved sear...

Re: Using custom code to push log data directly to...

Re: Using custom code to push log data directly to...

Using custom code to push log data directly to Spl...