About Chris_R_

Chris_R_ · ‎08-24-2010

I am creating a view with the following code. If I try to insert a < / form > after the </fieldset> it bails with an error. Any ideas why? <?xml version='1.0' encoding='utf-8'?> <form> <label>2 Dashboard</label> <fieldset> <input type="text" token="searchHost"> <label>Enter host to search against (can be wildcard)</label> <seed>fgw*</seed> </input> <input type="text" token="searchTerm"> <label>Enter keyword</label> </input> <input type="time" /> <default>Last 4 hours</default> </fieldset> <row> <event> <searchTemplate>host=$searchHost$ $searchTerm$ </searchTemplate> <option name="showPager">true</option> <option name="count">10</option> </event> </row> <row> <single> <searchName>SV_countvulnerable Last 24f</searchName> <title>Attacks against Vulnerable Hosts Detected Last 24 Hours</title> <option name="classField">range</option> <option name="linkFields">result</option> <option name="linkSearch">search SFDC "Impact: Vulnerable"</option> <option name="linkView">flashtimeline</option> </single> <table> <searchName>SF_last_24h</searchName> <title>IDS Alerts Last 24h</title> <option name="showPager">true</option> <option name="count">25</option> </table> </row> <row> <chart> <searchName>RT_suspicious</searchName> <title>Top Suspicious Last Hour - Pie</title> <option name="charting.chart">pie</option> </chart> <table> <searchName>TopTenInternet_LastHour</searchName> <title>Top 10 Internet-facing Source Ports Last Hour</title> <option name="showPager">true</option> <option name="count">25</option> </table> </row> <row> <chart> <searchName>pie_AV_infections left alone last 7 days</searchName> <title>AV infections Last 7 days - not cleaned</title> <option name="charting.chart">pie</option> </chart> <table> <searchName>table_AV infections not cleaned Last 7 Days</searchName> <title>AV infections: Last 7 days - not cleaned</title> <option name="drilldown">row</option> <option name="showPager">true</option> <option name="count">25</option> </table> </row> <row> <table> <searchName>vpnpoolusers_last_4hours</searchName> <title>Vortex Users - Last 4 hours</title> <option name="showPager">true</option> <option name="count">25</option> </table> <table> <searchName>FW - All Top Talkers Last 60 minutes</searchName> <title>All Firewalls - Top Talkers last 60 minutes</title> <option name="showPager">true</option> <option name="count">25</option> </table> </row> <row> <event> <searchName>ASAFO1</searchName> <title>Firewall Failover Events Last Hour</title> </event> </row> </form>

Chris_R_ · ‎08-24-2010

The crcSalt was set because the websphere logs all have a really big header which is identical in all the rotated logs, and splunk wouldnt index in the next SystemOut.log when it rotated. Ill check on the duplicate events w/_indextime value, Thanks

Chris_R_ · ‎07-27-2010

Hello Sunny, Since Splunk syslog's default extractions are taken directly from the event data, you have to override it another way. Try these settings all on your indexer(receiver). $SPLUNK_HOME/etc/system/local/props.conf [host::ip-10-196-173-139] TRANSFORMS = newhost $SPLUNK_HOME/etc/system/local/transforms.conf [newhost] DEST_KEY = MetaData:Host REGEX = . FORMAT = host::newhostname.domain.com If you have more then just syslog data coming from this host, you may need to set a custom sourcetype on your forwarder to filter off instead of your host in props.conf. let me know how it goes.

Chris_R_ · ‎07-27-2010

Lowell, thanks i am getting better results with: curl -k -u admin:changeme -d "search=search * &preview=1&status_buckets=300" https://server:port/services/search/jobs The job seems to stay alive much longer

Chris_R_ · ‎07-22-2010

I am running a pretty basic search such as this email="someemail@domain.com" OR email="someemail@domain.com" The job takes about 30 seconds to complete, but does not return results until the job is done. Is there any way to get partial results of the running job with an api call? Some call's i've tried that dont return the partial job results so far. https://server:8089/services/search/jobs/1279579745.369/results?output_mode=json&preview=1&count=0 https://server:8089/services/search/jobs/1279579330.368/results?output_mode=json&count=0 https://server:port#/services/search/jobs/1276629355.47/results_preview

Chris_R_ · ‎07-16-2010

sorry for the delay i was trying to recommend client using websphere app. Turns out it wont work for them. The indexer is 4.1.3, Its monitoring network shares cifs/ntfs mounts such as: [monitor:///ntfs/kahobtwas39Jlog] disabled = false crcSalt = < SOURCE > host = kahobtwas39.kah.unitrininc.com sourcetype = websphere_trlog_sysout _whitelist = (SystemOut\.log$|SystemErr\.log$) blacklist = (SystemOut_\d+.*|SystemErr_\d+.) I tried adding those whitelist/blacklist entries to filter out the rotated logs. Still the same behavior

Chris_R_ · ‎07-16-2010

TZ=Earth/Orbit :D

Chris_R_ · ‎07-09-2010

GK: These are full forwarders, here's the outputs from a forwarder [tcpout] indexAndForward = false [tcpout:stonegateGroup] disabled = false server=10.20.12.35:9001 [tcpout:fortimailGroup] disabled = false server=10.20.12.33:9997 and the inputs.conf from a indexer [default] index = default host = fortimailsplunk _rcvbuf = 196608 [monitor://$SPLUNK_HOME/var/spool/splunk] move_policy = sinkhole [fschange:$SPLUNK_HOME/etc] signedaudit = true sendEventMaxSize = -1 recurse = true pollPeriod = 600 filesPerDelay = 10 delayInMills = 100 followLinks = false fullEvent = false hashMaxSize = -1 [splunktcp] route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue; Note: I had them remove the tcp route = stanza seems to not be blocking this morning, could be a slower day...but i'll know for sure next week

Chris_R_ · ‎07-09-2010

This configuration is two 3.4.2 forwarders -> two 4.1.2 indexers. Forwarders have two UDP inputs & two seperate assigned sourcetypes on these UDP inputs, props/transforms/outputs entries are doing _TCP_ROUTING to two seperate indexers. Config seems ok for the most part. However they are getting constantly blocked tcpout_connections messages in metrics.log splunkd.log Error on the forwarders 07-07-2010 06:11:29.452 WARN TcpOutputProc - TcpSendThread: Connection to server lost - retrying: Broken pipe 07-07-2010 06:11:29.452 WARN TcpOutputProc - Connection dropped by Indexer. Possible version mismatch with indexer. Please check compatibility with indexer version splunkd.log errors on the indexer 07-08-2010 01:15:13.501 ERROR TcpInputProc - Error encountered for connection from host=< ip address >, ip=< ip address >. Timeout 07-08-2010 01:15:13.501 INFO TcpInputProc - Hostname=< ip address > closed connection 07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel source::udp:515|host::192.168.88.25|somesourcetypel|remoteport::41108" ended without a done-key 07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel "source::udp:514|host::192.168.8.204|somesourcetypee|remoteport::41108" ended without a done-key 07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel "source::udp:515|host::192.168.88.26|somesourcetype|remoteport::41108" ended without a done-key 07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel "source::/opt/splunk/var/log/splunk/splunklogger.log|host::NCCForwarder|splunklogger|remoteport::41108" ended without a done-key 07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel "source::udp:515|host::192.168.88.27|somesourcetype|remoteport::41108" ended without a done-key one other odd entry i see in the inputs.conf of the indexers, seems like this is a older spec file setting to route certain data to queues instead of letting splunk do it automatically? [splunktcp] route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;

Chris_R_ · ‎07-02-2010

A websphere server, in particular the websphere_trlog appear to be getting over indexed by a huge amount Checking http://server:port/en-US/app/search/indexing_volume shows 30gb worth of data on a single /ntfs/kahobtwas39Jlog/PROD/XAG_3_1/SystemOut.log Looking at the log size in the dir has 30MB worth of logs, Splunk appears to have collected more then 30GB worth Logs are rotating based on time, but they should still not be anywhere near 30gigs and the rotated logs are not whitelisted inputs.conf - settings [monitor:///ntfs/kahobtwas38Jlog] disabled = false crcSalt = <SOURCE> host = kahobtwas38.kah.unitrininc.com sourcetype = websphere_trlog_sysout whitelist = SystemOut\.log|SystemErr\.log They are getting a lot of DateParserVerbose errors so it's possible events are getting over indexed by failing date extraction? 07-01-2010 15:50:03.880 WARN DateParserVerbose - Time parsed (Sat Dec 1 14:50:15 2007) is too far away from the previous event's time (Thu Jul 1 15:50:15 2010) to be accepted. If this is a correct time, MAX_DIFF_SECS_AGO (3600) or MAX_DIFF_SECS_HENCE (604800) may be overly restrictive. Context="source::/ntfs/kahobtwas39Jlog/PROD/XAG_3_1/SystemOut.log|host::kahobtwas39.kah.unitrininc.com|websphere_trlog_sysout|" Perhaps just turning off date extraction would help resolve this ala / or any other ideas? /opt/splunk/etc/system/local/props.conf [websphere_trlog_sysout] DATETIME_CONFIG = CURRENT I've tried monitoring with file inputs set to DEBUG but not seeing anything useful

Chris_R_ · ‎07-01-2010

The rule of thumb we go by is usually anything that affects indexing level changes require a splunk restart, while search level changes require a reload. Here's a good guideline on how to determine which is which. http://www.splunk.com/base/Documentation/latest/admin/Indextimeversussearchtime So index creation or settings modifications, props.conf timestamp extractions, or transforms.conf indexed field modifications as well as most .conf manual changes will require a restart. If you make changes with $SPLUNK_HOME/bin/splunk CLI changes or within the UI, it wont require a restart.(unless of course you get prompted for a restart)

Chris_R_ · ‎06-28-2010

We have an index that gets around 2million events/hour and it seems not a sizable number of events are not making it from the manager to our splunk instance. At the very least we are talking about 60,000 events in a 24 hours period. This would seem to be beyond the normal expected loss for connectionless UDP. Is it possible splunk is being inundated with so many events that some are being discarded?

Chris_R_ · ‎06-23-2010

Lowell how's your DS metrics now? I had a bug open due to both of your cases but Neil's was solved due to a rogue outputs.conf in the unix app.

Chris_R_ · ‎06-21-2010

Stephen, what would be some alternatives if a clients AQ is constantly blocked, where too many .tar.gz files are coming in for splunk to process in time. increasing indexthreads seems to be a no no, adding a full forwarder before the indexer perhaps?

Chris_R_ · ‎06-03-2010

I have 10's of thousands of files(tarballs) i want to monitor via batch/sinkhole. [batch:///var/log/archived_files] index = someindex move_policy = sinkhole sourcetype = unique_sourcetype With a batch/sinkhole input, how frequently is the directory checked? Do I need to move the file into the directory atomicly (i.e. a mv from the local filesystem), or can splunk figure out when the file is not open for writing by anyone?

Chris_R_ · ‎05-28-2010

Is there any way to check for forwarders that have not connected recently and include a "sourcetype, source or host" value from that down host? Perhaps a subsearch with the metadata search for down forwarders? | metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time

Chris_R_ · ‎05-26-2010

If your on version 4 try a query such as this to see recently connected forwarders. | metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time

Chris_R_ · ‎05-24-2010

I've installed Splunk build 4.1.2 and this upgrade has introduced the following in the splunkd.log: 05-24-2010 12:58:34.695 INFO TailingProcessor - File descriptor cache is full, trimming... 05-24-2010 12:58:37.695 INFO TailingProcessor - File descriptor cache is full, trimming... 05-24-2010 12:58:40.956 INFO TailingProcessor - File descriptor cache is full, trimming... 05-24-2010 12:58:43.753 INFO TailingProcessor - File descriptor cache is full, trimming... 05-24-2010 12:58:46.465 INFO TailingProcessor - File descriptor cache is full, trimming... 05-24-2010 12:58:48.832 INFO TailingProcessor - File descriptor cache is full, trimming... 05-24-2010 12:58:51.807 INFO TailingProcessor - File descriptor cache is full, trimming... Now that we are using the "time_before_close" parameter within most of the input stanza's we are leaving files open for longer. This I imagine is putting more pressure on Splunk to be able to open and close files as required. As you can see from the log above, whilst not an error it is an indicator that we need to tune our Splunk instance better to cope with the large number of files that Splunk has to monitor (I would imagine around 800 per day) can you please look into this and advise me on how to better tune our Splunk instance? I have also tested setting various different OS values for the number of allowed open file descriptor (ulimit -n) and the Splunk max_fd (within limits.conf) but they have not resolved the issue.

Chris_R_ · ‎05-21-2010

Apro, Here is another answer that should help http://answers.splunk.com/questions/2291/how-to-backup-restore-splunk-db-to-new-system

Chris_R_ · ‎05-20-2010

Try setting your inputs to debug mode, we will need to figure out why splunk's complaining on those files in your etc/log.cfg set these or input them if not already there. category.TailingProcessor=DEBUG category.WatchedFile=DEBUG category.BatchReader=DEBUG category.FileTracker=DEBUG restart splunk, let it run a few minutes and check log/splunkd.log for any of those files like native*.log, splunk should be pretty verbose about what's wrong

Chris_R_ · ‎05-19-2010

Unfortunately this only works with syslog via UDP inputs. If using a tcp input, you would have to set up a props/transforms entry to store these fields.

Chris_R_ · ‎05-19-2010

Splunk recommends indexing anywhere from 3-10mb per second on a single indexer. Please keep in mind the upper limit of 10mbps is on very fast hardware, 15k rpm disks, raid 0+1 array, fast bonnie++ results Your system may be indexing within the reccomendations of < 100gig per day spec'd box, but if you have blocked indexqueue's at certain times you may be indexing in too much data at certain time frames. Check your queues with this search during problem time frames. index=_internal source="*metrics.log*" group=queue | timechart perc95(current_size) by name If you want to drill down and find out the maximum kbps indexed at that time index="_internal" source="*metrics.log*" per_index_thruput | timechart span=1h max(kbps) by series | addtotals You can then identify heavy forwarders sending lots of data index=_internal source="*metrics.log*" per_host_thruput | eval mb=(kb/1024) | timechart span=1h sum(mb) by series | addtotals For further assistance and recommendations on how to increase performance open a case with support.

Chris_R_ · ‎05-19-2010

I've already got my single indexer spec'd to handle under 100Gigs a day and it meets the requirements. However i am getting blocked queue's at certain times of the day. What gives?

Chris_R_ · ‎05-18-2010

Ok great, yeah setting the sourcetype in inputs.conf is usually the way to go.

Chris_R_ · ‎05-18-2010

Do the "rabbit" results returned in the UI match any in $SPLUNK_HOME/etc/apps/learned/local/props.conf Occasionally splunk will learn the wrong sourcetype, you can actually delete entries out of the learned props.conf if it does not match the props.conf entries you want. The sourcetype-too_small entries usually mean the amount of data splunk was trying to learn/create props from was too small in the file, source etc...

Posts	83
Solutions	14
Karma Given	63
Karma Received	117
Member Since	‎01-23-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

view error with </form> after </fieldset>

api REST searching on partial job results

Forwarder tcpout_connections blocked

websphere logs indexing more then they should

What is splunk's capacity for receiving UDP events...

Could you provide more monitoring detail on a dire...

How can i check for forwarders that have not sent ...

4.1.2 upgrade TailingProcessor - File descriptor c...

How much data can i index per second on a single i...

How do i customize scheduled search alert emails

view error with </form> after </fieldset>

Re: websphere logs indexing more then they should

Re: wrong HOST value

Re: api REST searching on partial job results

api REST searching on partial job results

Re: websphere logs indexing more then they should

Re: Can Index on Satellite stay In sync with Inde...

Re: Forwarder tcpout_connections blocked

Forwarder tcpout_connections blocked

websphere logs indexing more then they should

Re: What is the best way to figure out if a config...

What is splunk's capacity for receiving UDP events...

Re: Deployment Server Log Entries Stopped

Re: What does the queue named AQ? How did it get b...

Could you provide more monitoring detail on a dire...

How can i check for forwarders that have not sent ...

Re: Forwarder not reporting more than 2 weeks ago.

4.1.2 upgrade TailingProcessor - File descriptor c...

Re: Backup and restore of Splunk??

Re: 3 monitor stanzas of the same folder but only ...

Re: How do keep splunk from removing syslog priori...

Re: How much data can i index per second on a sing...

How much data can i index per second on a single i...

Re: Yet Another Problem Sending Events to the null...

Re: Yet Another Problem Sending Events to the null...