About jeff

jeff · ‎11-16-2014

I had a call with the developers and after we chatted for awhile it became apparent that the the posted documentation was somewhat inaccurate- it's since been updated. SA-Hydra and SA-Utils aren't to be installed on dedicated indexers (see the updated Splunk App for VMware component reference)

jeff · ‎11-08-2014

I run Splunk 6.1.4 in a distributed environment: dedicated search head dedicated indexers dedicated deployment server (mostly) universal forwarders In short: there's surprisingly little documentation for the SA-Hydra and SA-Utils apps (even the READMEs aren't very telling)- they both seem really heavy and I'm ambivalent about having them broadly deployed without a more thorough understanding... in particular, other than the event parsing in props.conf I don't see anything in either app that specifically is needed on the indexers (true?)... There looks to be a lot happening in the SA-utils app and I don't have a warm and fuzzy on what's really going on. Consider: [script://./bin/tsidx_clean_up.py] disabled = false passAuth = splunk-system-user # Once per day at 3 AM interval = * 3 * * * index = _internal sourcetype = tsidx:clean_up Isn't Splunk doing this on its own already? I'm not clear why this is needed... There's a lot going on under the covers here and without understanding this a little better I'm a little uncomfortable deploying it. On the other hand the Splunk App for VMWare is one of the top requests from my infrastructure folks so I want to support them. Can anyone shed some light on these apps? ========== TL;DR: I'm a bit of a minimalist when it comes to the configs at each stage of the pipeline. I don't like to have irrelevant configs where they have no business (for instance, an indexes.conf on a forwarder, inputs.conf on the indexers... don't even get me started on props and transforms). To that end, when I'm looking at a new app I dig through and dissect apps so that only relevant configs are present. I know... Splunk is good at ignoring irrelevant configs if they don't apply, and having these bundled apps is easier on the developers with all of the various Splunk architectures, but I find it is a lot easier, for me, to look at and resolve conflicts on an ongoing basis if I minimize the configs up front. So when I look at the Splunk App for VMware component reference and where the various components need to be installed, it's making my head hurt. There's a lot going on in there and I just really need to understand more deeply what all of these pieces are really doing in support of the VMWare data.

jeff · ‎05-03-2014

Question from my backup guys and I couldn't find a good answer in the docs- I don't understand the structure of the data model data on the system. Indexes with a data model defined have a datamodel_summary directory: [splunk@splunk3 splunk]$ ll ./firewall total 60 drwx------. 37 splunk splunk 4096 May 2 13:26 colddb drwx------. 340 splunk splunk 24576 May 2 13:35 datamodel_summary drwx------. 306 splunk splunk 20480 May 3 10:06 db drwx------. 2 splunk splunk 4096 Aug 17 2013 thaweddb In the _internaldb index directory, I seem to have one of these and another "summary" directory that looks like it's associated somehow with the splunk deployment monitor: [splunk@splunk3 splunk]$ ll _internaldb/ total 532 drwx------. 2216 splunk splunk 126976 May 3 09:47 colddb drwx------. 2519 splunk splunk 184320 May 3 09:55 datamodel_summary drwx------. 306 splunk splunk 28672 May 3 10:08 db drwx------. 2519 splunk splunk 184320 May 3 09:50 summary drwx------. 2 splunk splunk 4096 Aug 16 2013 thaweddb [splunk@splunk3 splunk]$ ll _internaldb/summary/998_163BFC27-2C4C-4CDE-83CD-F8B48C29BA80/20D17CF6-2E61-47A1-B3A4-FF57509916DF/ total 596 drwx------. 2 splunk splunk 32768 Dec 23 05:10 splunk_deployment_monitor_nobody_1a56f43bf8d5bf20 drwx------. 2 splunk splunk 32768 Dec 23 05:10 splunk_deployment_monitor_nobody_26e747c470c62ba8 <snip several lines /> drwx------. 2 splunk splunk 24576 Jan 11 14:08 splunk_deployment_monitor_nobody_NSd0dc3ea132443bbf From the backup perspective, the backups are throwing a thousands of errors each night for non-existant files (were there when the drive was scanned, but not when it came time to back up). I'm fairly sure it's okay to tell them to exclude the datamodel_summary (and summary) directories entirely since they can be recreated after a restore, but for my own sanity I'd like to understand the structure a bit more. Can we exclude the data models from backup? What is that extra summary directory in _internaldb all about? Likewise, it can be excluded?

jeff · ‎12-03-2013

Presuming you mean Splunk v6, it already is: http://docs.splunk.com/Documentation/VMW/latest/Release/Features.

jeff · ‎12-03-2013

what did you change it too? I would think keeping it in the default location would work best since Splunk is monitoring that directory and adding it to the _internal index... Of course I'm not getting any log data now- looking at the script it appears to only log when it encounters an error...

jeff · ‎12-02-2013

@kevenking- yep, I found it was working in my case because I had installed faup on the indexers once upon a time early on. After removing the app from the indexers my edit did not work. I found a hard coded path in the faup.py script. See my edited answer above for the solution. Hopefully @stricaud_splunk can take a look at these changes and incorporate into the next faup release.

jeff · ‎11-15-2013

@kevinking- did you restart your search head? On your indexers, check $SPLUNK_HOME/var/run/searchpeers/ - /apps/faup... is the opt directory there?

jeff · ‎11-06-2013

solved... The knowledge bundle from the search head to the indexers did not include the apps/faup/opt directory. When using bucket, the search results were retuned to the search head before the lookup so it worked. The following changes need to be made to the app: create apps/faup/default/distsearch.conf to distribute the opt directory to search peers [replicationWhitelist] faup-opt = apps/faup/opt/... The faup.py script in bin has a hard coded path to the faup app in $SPLUNK_HOME/etc/apps/.. edit apps/faup/bin/faup.py to use a relative path def where_is_faup(): #replace the next four lines with those below to use a relative path from this python script #if platform.system() == "Darwin": # return os.environ['SPLUNK_HOME'] + "/etc/apps/faup/opt/faup-darwin" #if platform.system() == "Linux": # return os.environ['SPLUNK_HOME'] + "/etc/apps/faup/opt/faup-linux" if platform.system() == "Darwin": return os.path.join(os.path.dirname(__file__), "..","opt","faup-darwin") if platform.system() == "Linux": return os.path.join(os.path.dirname(__file__), "..","opt","faup-linux") # I don't know, so let's trust the system return "faup" These two changes allowed the search to run properly without installing the faup app on each indexer.

jeff · ‎10-30-2013

In installed the faup addon after seeing it at .conf2013 but have gotten inconsistent results. index=wsa earliest=-60s | bucket x_wbrs_score | lookup faup url AS cs_url works as expected: I get the appropriate url_* fields extracted. index=wsa earliest=-60s | lookup faup url AS cs_url does not work... the url_* fields are not created, although 100% of events have the cs_url field. I am perplexed. Splunk 5.0.4 on the SH and Indexers.

jeff · ‎05-09-2013

I am trying to run a timechart against a summary index (the summary is populated once an hour) and split into 24 hour segments: index=summary search_name=my_summary earliest=-30d@h latest=@h | timechart span=24h count by target If I set span=1d the buckets are split at midnight. I really want them to be split into 24 hour buckets that align with the search when run though... whether I set span=24h or span=1440m or span=86400s though, the segment is split at 8pm each day (regardless of when the search is run), resulting in 31 bins (29 with 24 events per target, and 2 with whatever adds up to 24 (11+13 for instance) at the beginning and end of the span. I've also tried bins=30 by itself (2 bins split at month markers) and span=24h bins=30 (bins ignored). Splunk 4.3 Any thoughts or sanity checks welcome and appreciated.

jeff · ‎04-25-2013

Sounds like you're looking for timechart. (your search) | timechart span=30s limit=0 avg(workerO) as workerO by nodeID should get you close to what you're looking for.

jeff · ‎04-01-2013

Where are you defining your regex? If you are doing multivalued extraction at search time, you'll need to define that in transforms.conf (and refer to it in props.conf) on your search head.

jeff · ‎04-01-2013

Shoot... thought that might have been it. I ran into something similar way back in the day. REPEAT_MATCH only applies at index time, so it's a non-issue. What does the output from splunk cmd btool transforms list <your stanza> --debug on your search head look like?

jeff · ‎04-01-2013

Try this instead: (?im)"(?<FIELDNAME>[^"]+)".+[\r\n]{0,2} If all of the lines in your log are captured in a single event, then ^ marks the beginning of your record. You may also try your (?im-s) explicitly turning off single-line mode...

jeff · ‎04-01-2013

see the host_regex under [monitor://<path>] in http://docs.splunk.com/Documentation/Splunk/5.0.2/admin/Inputsconf

jeff · ‎03-13-2013

Something like this should work: "query" | regex _raw="(?i)query=\"*\"" but that'll be awful inefficient since it will return all lines with "query" (presumably every record in your index for this sourcetype) then filter based on the regex. You'll be better off if you can add additional criteria to filter it, or replace the * character at index time using SEDCMD, for instance...

jeff · ‎02-26-2013

I don't see anything inherently wrong with the regex... so I'll need to ask the obvious question - are you sure the Some.Component-log4net.log is being correctly typed as "mysourcetype"? Check by adding sourcetype to your table above: index=prod | rex field=source "^.\\([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})?(?<SourceFilenameTmp>[^\\])$" | table sourcetype source SourceFilename SourceFilenameTmp

jeff · ‎02-21-2013

Your regex I think should work- but you need to escape the backslashes in your source path: [source::d:\\TGNI\\Logs*.log] From the props.conf.spec: **Considerations for Windows file paths:** When you specify Windows-based file paths as part of a [source::<source>] stanza, you must escape any backslashes contained within the specified file path.

jeff · ‎02-19-2013

Option 1: Install with the command line parameter STARTSPLUNK=0 to not have Splunk forwarder start automatically (with caveats- see command line installer documentation. Option 2: Install splunk, point to a deployment server during installation. Keep all other defaults (no receiving indexer, no logs to monitor, etc). On your deployment server, create one or more apps with all of the appropriate settings for your environment... this is what we do. See deployment server documentation.

jeff · ‎02-18-2013

What does the output from splunk cmd btool indexes list --debug look like? You can run this tool for each of your config files to see if an app is overriding your system/default settings.

jeff · ‎02-18-2013

Two courses of action to go through: Try statically adding the correct hostname in etc/system/local/inputs.conf as jonuwz suggests below on one of your forwarders to see if this resolves the problem. Whether or not it does, file a bug report with Splunk at http://www.splunk.com/page/submit_issue.

jeff · ‎02-15-2013

To be clear... are your forwarders running 4.3? You can set host to $decideOnStartup in inputs.conf starting with Splunk 5.0... it's not an option in 4.3 (so if you have it set on a 4.3 forwarder, you're setting it to the literal value $decideOnStartup). Also, verify that you followed the instructions for installing a forwarder on a system image. edit Okay, based on some of what I read above (specifying perfmon counters in inputs.conf for example) and another post it looks like you are indeed running Splunk 5... I note there was an issue fixed in 5.0.2 with $decideOnStartup for web-uploaded content. I wonder whether this might be another bug? Does the other data from your forwarders come across with the correct hostname?

jeff · ‎02-14-2013

jonuwz is right here... my answer above assumes a user logged in sometime between 40 and 30 days ago- it won't report any users that didn't log in at all or outside of that time. You could dump all AD users to a csv and use inputlookup to load them. Note that lastLogon isn't replicated, so to be accurate you would need to get this value from every Domain Controller in your environment. You are better off querying lastLogonTimestamp. It is replicated, but is only updated on any given object if its value is over seven days (I think) old at the time of successful authentication.

jeff · ‎02-14-2013

You'll probably find that this search will be expensive in Splunk without summary indexing enabled. Generate a query to search the Windows security logs for successful logons and schedule to run every x minutes or hours, adding the data to the summary indexing. Depending on what fields you choose to push to the summary index, you may just want to record one logon per user over the timespan... something like: sourcetype=sourcetype=WinEventLog:Security EventCode=4648 | rex field=_raw "(?si)New Logon:.*Account Name:\W(?{user}[^ \r\n]+)" | stats max(_time) as lasttime by user (note:* {user} above should really be <user>... markdown wasn't cooperating with me for some reason.)* scheduled to run over an hour and added to a summary index hourly. Then... index=summary_ad_logons earliest=-40d@d latest=-30d@d NOT [search index=summary earliest=-30d@d latest=now | dedup user | fields user] Now, having said all of this, I'm pretty sure you can get this out of the native Microsoft plugins by looking at a given user's lastLogonTimestamp attribute. The Windows Server 2008 AD Administrative Center plugins has specific pre-bulit queries just for this purpose.

jeff · ‎02-14-2013

When you say you want to "filter", you are implying you want to take action (change indexer, drop data) based on the regex of the host? You would do this by specifying the host-matching pattern for the host in props.conf, which is regex-based (with the exception of ..., ., and *). props.conf [host::(my.+[1-6])|(otherhost[0-9]{0,})] TRANSFORMS-drop = drop_event transforms.conf [drop_event] REGEX = . DEST_KEY = queue FORMAT = nullQueue

Posts	127
Solutions	18
Karma Given	26
Karma Received	127
Member Since	‎2010-03-25

Online Status	Offline
Date Last Visited	‎07-17-2025 10:07 AM

timestamp not being recognized for Proofpoint Add-...

Stream: 10G capture using Intel x520 on linux help...

Splunk Stream: Is it possible to pull SSH fingerpr...

Splunk local account login to splunkweb when SAML ...

Must all cluster nodes be offline when upgrading a...

Splunk App for Enterprise Security 3.3.1: Why are ...

Splunk DB Connect 2 - dbxquery only returns 1001 r...

SA-Hydra and SA-Utils in a distributed environment...

data model storage and backups

faup lookup not working as expected...

Re: SA-Hydra and SA-Utils in a distributed environ...

SA-Hydra and SA-Utils in a distributed environment...

data model storage and backups

Re: Splunk App for VMware

Re: faup lookup not working as expected...

Re: faup lookup not working as expected...

Re: faup lookup not working as expected...

Re: faup lookup not working as expected...

faup lookup not working as expected...

timechart day offset

Re: help Extracting data from logs for a line grap...

Re: Multiple fields per event with different data

Re: Multiple fields per event with different data

Re: Multiple fields per event with different data

Re: How to obtain hostname/eventtype from file nam...

Re: Search for field with '*'

Re: Regular expression works inline via 'rex', but...

Re: Splunk Regex Expression

Re: Inputs.Conf - Historical Logging - Chicken and...

Re: Indexes not showing data

Re: $decideOnStartup Remote Perfmon

Re: $decideOnStartup Remote Perfmon

Re: How do I track unused logons in Windows?

Re: How do I track unused logons in Windows?

Re: filtering on host

Are you a member of the Splunk Community?