About Lowell

Lowell · ‎01-18-2011

Basically, if you hide (or remove) the input field used your lookup , then the lookup for a specific event isn't executed because there is no value to lookup. That makes sense. You can leverage this behavior to do a conditional lookup by conditionally hiding the lookup input field before you call lookup . I think and example would be helpful. Let's say your base lookup search looks like this: ... | lookup mylookup code OUTPUT desc | ... So lookup is using the "mylookup" lookup table, "code" is the input field and the lookup returns a "desc" field. Now, let's say you have a field called "bool" that enabled/disables this lookup. (The lookup is enabled when "bool=1"). Then you can use a search like this: ... | eval _copy_code=code | eval code=if(bool==1, code, null()) | lookup mylookup code OUTPUT desc | rename _copy_code as copy | ... Note: If you don't need to preserve "code" then you can drop out the first and last search commands. The following question & answer may help too: http://answers.splunk.com/questions/4079/making-a-lookup-optional-or-how-to-build-a-multi-level-lookup

Lowell · ‎01-18-2011

Can you provide a high-level overview of what you are trying to accomplish? Also if you can provide some reason(s) why you can't simply use splunk event forwarding which traditionally the suggested way of distributing events across splunk instances.

Lowell · ‎01-17-2011

Off the top of my head, EXTRACT s (in props.conf ) are done first, then REPORT s are evaluated. (Both of these can take a "class" name, which is simply sorted in lexagrapical order.) Of course, another simple way to assure fields are extracted in the proper order like this is to list out both extractions in the order you want them using a single REPORT entry, like so: props.conf : [my_sourcetype] REPORT-myfield = fields1, fields2 transforms.conf : [fields1] REGEX = (?<field1.) .... (?<field4>...) [fields2] SOURCE_KEY = field4 REGEX = (?<field5>) .... (?<field8>) In this example, "fields1" will always be evaluated before "fields2". Does that help? (If not, please provide some sample events and the related props.conf and transforms.conf .) Update: Based on your updated example, this is one solution that should work: (Assuming all your regex and everything else is fine, I didn't look that closely; and without sample event's it's hard to say anyways.) [cloudfront_http] REPORT-cloudfront = cloudfront-cdn-http, cdn-uri-v1, cdn-page-v1 The answer provided by southeringtonp should work fine too. It really comes down to your preference. If you had a more complex situation (For example, if "cdn_uri" could be found by two different field extractions based on variations in your events, then it would certainly be better to go with the explicit priority approach that southeringtonp pointed out.), but as is, either should work fine.

Lowell · ‎01-13-2011

It seems only right to the following blog page, since this question inspired it: http://blogs.splunk.com/2011/01/11/maintaining-state-of-the-union/comment-page-1

Lowell · ‎01-13-2011

I'm not sure what "webping" is. Can you update your question and add the exact URL that your browser is pointing to? (Click the "update" link below your question.)

Lowell · ‎01-13-2011

I'm not 100% sure what all you've tried. I would start with simply setting up an appropriate props.conf entry, for example: [the_source_type] TIME_FORMAT = %H:%M:%S,%3N %m/%d TIME_PREFIX = ^ MAX_TIMESTAMP_LOOKAHEAD = 40 # Assuming your files are on a single line, so you should set this too: SHOULD_LINEMERGE = False Do you regularly load these files from the previous year? Or are you concerned about when the year rolls over? If you have splunk actively monitoring your files (either locally, via splunk forwarder, or over a network share) and aren't doing any kind of batching process (or log file shipping), then you should be fine as splunk read the files pretty quickly. Having said that, I haven't really tested this configuration too closely, as splunk seem to just get it right. That, and the year change doesn't happen all that often...

Lowell · ‎01-13-2011

That is correct. I'm looking for connected ports and listening ports, and seeing the PIDs is nice too. I think netstat -a -o will give me a good place to start.

Lowell · ‎01-13-2011

BTW, I haven't actually tested the regex shown above, and the "..." is because I have no clue what those last fields are for (and it's also a valid regex expression. 😉 Yeah, I'm a geek.) If you post a few example events and indicate what the fields are for, I or someone else can help you out with the regex. You can use the IFX (interactive field extractor) for this too, but IMHO, for strictly formated events like this you're better off writing a single all-in-one regex than creating many single-field regexes as would happen with IFX.

Lowell · ‎01-13-2011

Yes, only new events would be assigned to this sourcetype. However, you can always use the same name, I chose "afl" as an example.) If you do use the same sourcetype that splunk used automatically, then the field extractions will work for old and new events (because EXTRACT- is a search-time setting. Whereas TIME_* and SHOULD_LINEMERGE are index-time settings. (If you're new to splunk, it will payoff long-term to become familiar with the difference between search-time and index-time settings; I remember being confused at first; fortunately the docs are more clear on this point now.)

Lowell · ‎01-12-2011

You should setup a sourcetype if you don't have one already. This is where you will setup some field extractions to extract the values you are looking for; once you have that done, getting the specific search your are looking for is pretty easy once you get familiar with the tool. Let's say your log file is located in /var/log/afl.log and you want your sourcetype to be called afl . Setup the following entries in props.conf : [source::.../afl.log] sourcetype = afl [afl] TIME_PREFIX = \[ TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z MAX_TIMESTAMP_LOOKAHEAD = 128 SHOULD_LINEMERGE = False EXTRACT-fields = ^(?<clientip>\d+\.\d+\.\d+\.\d+) \S+ \S+ \[[^\]]+\] "(?<user>[^"]+)" ... Once that's setup, you can restart splunkd and setup splunk to index your afl log file. Once you have events in splunk, you should be able to see them with a search like so: sourcetype=afl Assuming that's working, you can move on to getting the report you're looking for like so: sourcetype=afl | top user

Lowell · ‎01-12-2011

Have you tried simply removing those double-quotes? That could be your problem. You also don't need a . (period) after %S (Oh, and this is not "regex" syntax, it a strptime format string.)

Lowell · ‎01-12-2011

Has anyone setup the windows "netstat" command as an input? I like the "netstat" source provided in the unix app, and it seems like it should be possible to capture most of the same information using the built-in "netstat.exe" that comes with windows. I did notice that some of the columns are too wide (long host names or port names). Is there a way to get the same information with a WMI source?

Lowell · ‎01-12-2011

See my comment about MAX_* settings on this question: http://answers.splunk.com/questions/7924/matching-timestamp-outside-of-the-acceptable-time-window. I understand how out-of-order timestamp can cause issues, but I don't think that's what's happening here.

Lowell · ‎01-07-2011

I just tried running the Splunk Web restart command manually ( H:\splunk\bin\splunk restart splunkweb ), and I see the issue now. This appears to be an upgrade of Splunk. -------------------------------------------------------------------------------- Splunk has detected an older version of Splunk installed on this machine. To finish upgrading to the new version, Splunk's installer will automatically update and alter your current configuration files. Deprecated configuration files will be renamed with a .deprecated extension. ... Perform migration and upgrade without previewing configuration changes? [y/n] So it looks like it's waiting for an answer its never going to get. And even it you pass in --answer-yes , then you simply get the message: An error occurred: In order to migrate, Splunkd must not be running. Fun. I guess I'll be updating all of these manually. I thought this was taken care of automatically with the Windows install. I'm used to dealing with this manually after each splunk upgrade on Linux.

Lowell · ‎01-07-2011

After deploying an updated app to a deployment client, instead of the client restarting splunkd (as requested), it shuts down compleetly. As a result, I have to connect to the boxes and manually restart the splunkd service. For a few months I only had this problem with a single forwarder (I tried a few upgrades, but that made no difference). Now more recently several additional servers are having this problem too. Here are the splunkd.log message after a sample deployment: 01-06-2011 17:02:50.896 INFO DeployedApplication - Checksum mismatch 1889753244061277121 <> 15808103211093779078 for app: co_windows_extras. It will be reloaded again from: splunk.co.com:8089/services/streams/deployment?name=default:windows-server:co_windows_extras 01-06-2011 17:02:50.927 INFO DeployedApplication - Remote repository has resolved to: splunk.co.com:8089/services/streams/deployment?name=default:windows-server:co_windows_extras 01-06-2011 17:02:50.974 INFO DeployedApplication - Downloaded url: splunk.co.com:8089/services/streams/deployment?name=default:windows-server:co_windows_extras to file: H:\splunk\var\run\windows-server\co_windows_extras-1294350540.bundle 01-06-2011 17:02:51.396 WARN DeployedApplication - Installing app: co_windows_extras to location: H:\splunk\etc\apps\co_windows_extras 01-06-2011 17:02:53.114 WARN DeploymentClient - Restarting Splunk Web... 01-06-2011 17:03:04.145 WARN DeploymentClient - Couldn't execute H:\splunk\bin\splunk restart splunkweb 01-06-2011 17:03:05.755 ERROR DeploymentClient - Couldn't ensure that index dirs were created. 01-06-2011 17:03:05.755 WARN DeploymentClient - Restarting Splunkd... ... (other non-related activity) ... 01-06-2011 17:09:05.753 ERROR DeploymentClient - forcing shutdown since it didnt complete in 360 seconds I'm not sure why it's trying to restart Splunk Web on a deployment client, since I have this disabled. Any ideas on what's hanging this up?

Lowell · ‎01-07-2011

After some network packet tracking with wireshark and a bunch of head scratching; I've finally found the answer: The messages are a result of the "Splunk Monitoring" app that I installed some time ago and had forgotten about. The app is setup to poll a given list of splunk servers (which was out of date on my system, hence not ALL of my splunk instances were reporting this.) The app uses a scripted input that runs on a regular interval (10 minutes on my system.) The basic ping-like approach used by the app actually uses nmap to look for a response on port 8089 (which is the the splunkd HTTPS listener). So because there is no actual request being made, nothing gets logged in the splunkd_access log, and because there is no attempt to handle the SSL stuff, you end up with errors coming from splunkd about an invalid connection. Which is accurate. So the bottom line is that these messages are not harmful, but there may be a better way to monitor for up/down activity that using nmap . Perhaps there is some internal "ping" like URL that could be used with wget or curl or something like that. Since this took a while track down, I figure it was worth sharing. And of course any other kind of port scanning device or simple TCP monitoring service would most likely also trigger this kind of messages too. Perhaps this will save somebody some time.

Lowell · ‎01-07-2011

I've been seeing a bunch of error message sequences like this: 01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:0 01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0) 01-07-2011 09:45:40.106 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0 01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL Error for fd from HOST:splunk.my.domain, IP:192.168.1.99, PORT:3959 This is happening not only on my splunk indexer, but also on most (but not all) of my splunk forwarders too. Observations: The "HOST" and "IP" are always pointing to my central splunk indexes. It seems like errno:0 could indicate that there is NO error (since low-level OS calls often use a return code 0 to mean "success", and any other value indicates a specific error code.) Sometime I see SSL_ERROR_SYSCALL ret errno:104 instead of errono:0 but that occurs much less frequently (and the subsequent errors show above are the same) Any ideas? Should I be concerned about these?

Lowell · ‎01-06-2011

Agreed. I think this is probably the only option. I would suggest that if you are going to sum up the bytes, you may as well also store a count of the events; to give you values more comparable to the builtin metrics. You could do something like: | stats eval(round(sum(size)/1024,2)) as total_kb, count as events you could from there calculate average rates too since you know the time span.

Lowell · ‎01-06-2011

I think I disabled all the of internal searches for this very reason. But supposing I missed one, can you think of any reasons why I'm not seeing anything logged in searches.log , scheduler.log (both of these are 0 bytes on the forwarder), audit.log has nothing about running searches, and there's nothing in dispatch folder. Is there anything else I should look for?

Lowell · ‎01-06-2011

BTW, this can be a problem with summary indexing too. I know that in 4.x "source" is set to the name of the saved search, which helps this problem, but it seems like people are still running into this; for example if your doing per-day summary; then the first second of the day can be given a very large number of events.

Lowell · ‎01-06-2011

@dmlee, you may want to open a new question. This question is specifically referring to a change in behavior after an upgrade; but there is a more generic question here: How to avoid 125000 events limit in my summary index? If you ask a question like that you me able to get some additional attention. I don't have a good answer, especially without seeing the related searches. I added a link of interest to my answer that may be helpful too.

Lowell · ‎01-06-2011

BTW, you need a higher score before you can add a comment; it's not your browser.

Lowell · ‎01-06-2011

@sranga, sorry for getting back so late, I must have missed your comment. While you can use addinfo and collect and all that, you don't have to, and that's not what I'm suggesting that you change. I'm simply pointing out that you can change your search to use | stats instead of using | sistats the si -prefixed search operators sometimes give you extra value add and generally lets you do less thinking upfront, but I've found that blindly choosing sistats over stats can lead to other problems if you aren't paying attention to how much output it creates.

Lowell · ‎01-06-2011

BTW. I think the interactive field extraction (IFX) tool uses a python search command (and therefore the python regex engine) whereas Splunk uses PCRE for built-in field extractions (as well as the rex search command), so it is conceivable to get some subtle rexex flavor difference like this when using IFX (although, the normally don't seem subtle when your looking at them). It's also possible something else was going on.

Lowell · ‎01-06-2011

Yeah. I do see that I made a typo on the second one. You don't need the [\r\n] before Request , but it may be helpful to tell the regex engine to always expect the word Request to be the start of a line. Glad you have a working solution.

Posts	1343
Solutions	217
Karma Given	1014
Karma Received	1581
Member Since	‎03-26-2010

Online Status	Offline
Date Last Visited	‎05-26-2021 05:52 PM

Is passAuth still supported for scripted Inputs?

Does inputcsv honor dispatch=true?

Can a search detect it's own sample ratio?

How do you update lookups on a SHC while running S...

How do create indexed fields in a summary index?

What is a managed app in Splunk Enterprise Securit...

How to use a custom token filter within Simple XML

Can you use appendpipe and map together?

Is there a way to pick a point in time (not a time...

How do I run a programmatic search against a Searc...

Re: Can lookups be made conditional on the value o...

Re: Better Hot-To-Warm Roll Methods?

Re: n-level nested transform and extraction

Re: Complex Query question

Re: 500 internal server error on localhost

Re: Parsing Date/Time questions (biggest issue is ...

Re: Anyone setup a netstat like input for Windows?

Re: Looking at AFP logs

Re: Looking at AFP logs

Re: Looking at AFP logs

Re: How to define a time format using regex?

Anyone setup a netstat like input for Windows?

Re: What would cause the "date_*" fields to be mis...

Re: Deployment client fails to restart splunkd - ...

Deployment client fails to restart splunkd - manu...

Re: Why am I seeing TcpInputFd SSL Errors every 10...

Why am I seeing TcpInputFd SSL Errors every 10 min...

Re: How can I determine my indexing volume not by ...

Re: LookupOperator errors on a forwarder?

Re: what's max # events I can have timestamped wit...

Re: IndexScopedSearch Error

Re: Error in 'IndexScopedSearch': The search faile...

Re: IndexScopedSearch Error

Re: Search extract grabbing more than tested/verif...

Re: Search extract grabbing more than tested/verif...

Join the Conversation