About Genti

Genti · ‎10-12-2010

things to keep in mind: Disk IO speed: how fast can you write to disk. If this is low, no matter how fast your machine is CPU wise or RAM wise, you will still fall short in indexing speed. CPU: Its needed to write the info to disk, allow you to search, make it possible for you to monitor many files at the same time and parse through the logs. RAM: well known..

Genti · ‎10-11-2010

then i think the best bet would be to 1. install forwarders on all your servers and set them to monitor your logs and send the events to the splunk indexer and 2. set up some saved search that runs every-so-often and alerts you about how many events you have. If you want you could also have a script that runs and perhaps calls to the remote hosts to delete the data. However, you might want to be careful with this as if you delete the logs and for some reason they did not make it into splunk, then you will have data loss..

Genti · ‎10-11-2010

Your question, and environment is a bit unclear. The data is coming from forwarders into the single splunk server (indexer)? If this is the case, then there is no data being "indexed" on the other machines, and hence nothing to delete. It almost sounds like you would like the original raw data to be deleted? If that is the case splunk does not touch that data at all, in the sense, it only monitors the logs but never changes them or manipulates them. Hence there is no way to use splunk to delete raw data from your disk. If i have misunderstood your question you might want to edit it and be a bit more specific..

Genti · ‎10-11-2010

did you successfully authenticate ?

Genti · ‎10-11-2010

Multiple choices here Steve, from your search dashboard you can: - Actions:Save results - for later viewing through Jobs manager page - Actions:Export results - for exporting to .csv or other available formats - Actions:Build report... - to build a report of the data. By default the report gets created as | timechourt count if you would like something different, then click on "Define Data using search language" - Actions:Save search - if you want this to be an automated search. You can have it send you an email alert as well as a csv/pdf of the results. More on the above in the Users Manual pages..

Genti · ‎10-09-2010

Do you see all your files if you check this: https://localhost:8089/services/admin/inputstatus/TailingProcessor%3AFileStatus Note: you might need to modify port in case you have not used the default ones

Genti · ‎10-09-2010

There could be a couple of things going on here: first, the trivial - have you actually created any monitoring stanzas on the forwarder? i.e. are you actually monitoring anything at all? then, when you go to the SEARCH app summary dashboard, under the list of hosts, do you see the forwarder there? Lastly, if you do a search like: index=_internal do you just see logs from your indexer or your forwarder as well?

Genti · ‎10-09-2010

how did you produce the above chart?

Genti · ‎10-09-2010

why was it none before? Customer set? And was it working on 4.1.4? If so, then it was not the upgrade that broke it, just they had never restarted the server since they changed the config file?

Genti · ‎10-08-2010

try this: index="ftp" host="host" "bPasswordOK=1" | fields user, src_ip, log_date | stats count by user, src_ip, log_date | sort -count | dedup src_ip | dedup user Edit: okay, thinking a bit more about this i think the above might not be what you want. It dedups first by one field, and then dedups by the second. Reading your question again it feels like you want to dedup by a "concatenation" of the two. So if both A and B are the same, dedup by it. To achieve that i think this is the search you need: index="ftp" host="host" "bPasswordOK=1" | fields user, src_ip, log_date | stats count by user, src_ip, log_date | sort -count | eval myfield=source.punct | fields myfield | dedup myfield Im no search expert though, so you might want to hear from SS or GK for a better approach.

Genti · ‎10-08-2010

if you run: index=_internal source=*metrics.log* group=queue | timechart perc95(current_size) by name what kind of numbers do you see? If you use heat map, what colors do you see for each of the queues? Lastly, can you confirm that the indexer is on the same timezone as the logs timestamp?

Genti · ‎10-08-2010

yeap, inputcsv editing the answer

Genti · ‎10-08-2010

Deepak, After the indexer goes down, splunk forwarder should actually fully stop forwarding. If indexer is down, in splunkd.log of the forwarder you should be seeing errors regarding its connectivity with the indexer. This will in turn fill up the tcpout queue on the forwarder. Once the queue is full, the forwarder will stop trying to read the data altogether. once the indexer is up again then the forwarder starts sending the data (the one in the queue first). There is no reason for the forwarder to lose data.. (unless you stop/restart splunk while the indexer is down, which will cause the events that are in the queue (blocked) to drop and disappear) Also, this We see that Light Forwarder does forward some data after it recognizes that server went down but NOT all of it. doesnt make any sense. If indexing server is down, how is the forwarder sending the data?

Genti · ‎10-08-2010

problem with this might be that it will index all data, and hence create duplicates in the index.

Genti · ‎10-07-2010

make sure that the syslog messages are actually coming with the priority level included. Tell syslog to write to file and examine the file. If the priority is not showing up there, then it is not a splunk issue. You have to find out how to tell syslog to actually send the priority flag over. Cheers!

Genti · ‎10-06-2010

Since you "desperately" want the data out of the index then you should be able to put some extra work and try this: Export the data that you care for, csv format. Watch out for the size (rows) of the csv file. Once you are sure you have all your data that you care for, roll the index and back up your data. Then, clean all the data for the index using the clean command for this index. Then use inputcsv to get the data back into this clean index. Note again, only 10k data will be imported at once, so you will need to use a little trick to get the data in again. see THIS for more info on importing large CSV files.

Genti · ‎10-06-2010

i would assume as long as there is a bridge to communicate between the two networks, the setup should be the same as all forwarder->indexer configurations. Checking pinging/telneting and data connectivity between the networks is the first step. Then you can set up your indexer on Network B to FORWARD and INDEX the data.

Genti · ‎10-06-2010

ha! bad Maverick, bad!

Genti · ‎10-06-2010

yes, something like: [yoursourcetype] EXTRACT-loginfield = ^\S+\s+(?<login>\S+) EXTRACT-Status = ^\S+\s+\S+\s+(?<status>\S+) ..etc.. props.conf.spec for more info

Genti · ‎10-06-2010

you two are hilarious!!

Genti · ‎10-06-2010

i think something like this should work for you: REGEX = ^\w+ (\w+) Yourfield = $1 If you can send a bigger log file i can be a bit more sure, but using a regex tester this works ok for me. Hope this helps

Genti · ‎10-05-2010

Ambushing one of the UI dev's and torturing him for a while i found out what one needs to do in order to have specific portions, or all, of the right hand side menu in the manager view show up. Browse to: /splunk/etc/apps/search/metadata/ and create/edit the local.meta file to contain the following: [manager/datainputstats] access = read : [ power ], write : [ admin ] [manager/data_inputs_monitor] access = read : [ power ], write : [ admin ] [manager/data_inputs_script] access = read : [ power ], write : [ admin ] [manager/data_inputs_tcp_cooked] access = read : [ power ], write : [ admin ] [manager/data_inputs_tcp_raw] access = read : [ power ], write : [ admin ] [manager/data_inputs_udp] access = read : [ power ], write : [ admin ] Restart Splunk and login as a poweruser. You should now see "Data Inputs" on the right hand side menu, along side the "User Options" - which always shows.. Note, with this config powerusers will only be able to see/list the data inputs and not change the config. If this is what is wanted then change the write: [admin] to [power] and you should be done. This is not really documented and i hope this answers page might help others out there... cheerio! .gz

Genti · ‎10-05-2010

So, a case came in with the following issue: "Even after giving the power user (read role) the capability to edit_monitor list_inputs etc, i still do not see the listing for Data Inputs on the right hand side menu in the manager view" I thought the customer might have done something wrong with his configs but found out that the same was happening for me. And so started an epic war against the UI to try and find out what capabilities i needed to add to a role so that users that belonged to it would be able to see more then just the "User Options" on the right hand side menu of the Manager View. It turns out that no matter what capabilities you add, the other options do not show up. So, what do i need to do in order to have the "Data Inputs" (for example) show up for the poweruser?

Genti · ‎10-05-2010

how about this: index="_internal" source="*metrics.log*" per_host_thruput | timechart max(kbps) by series | addtotals

Genti · ‎10-05-2010

I think what you want to do can be achieved by Routing events to nullQueue Make sure you create the regexes to match what you do not want and you should be set.

Posts	377
Solutions	82
Karma Given	153
Karma Received	345
Member Since	‎05-04-2010

Online Status	Offline
Date Last Visited	‎06-05-2020 02:02 AM

Search help: remove intersection of two sets from ...

When creating a diag, splunk complains about splun...

How can i modify flashtimeline so that i can show ...

How to monitor .dat.gz files?

time format for results in en-US vs en-GB

on a '| metadata type=hosts' search, what do each ...

how to get host, sourcetype and source from a sing...

(search time) extracted field not showing up on th...

date_* fields not being extracted

Using the Rest API endpoint, how do i know if a se...

Re: Splunk throughtput

Re: Can splunk delete remote event logs?

Re: Can splunk delete remote event logs?

Re: New CSV files are being seen, but not indexed

Re: How can I export the events that are the resul...

Re: New CSV files are being seen, but not indexed

Re: New input from light forwarder not appearing

Re: Sort Fields in Charts

Re: I have upgraded from version 4.1.4 to version...

Re: Help with sorting a date

Re: TailingProcessor - Could not send data to outp...

Re: events not deleting with | delete

Re: Light Forwarder Data Loss Recovery

Re: Light Forwarder Data Loss Recovery

Re: "no_priority_stripping = true" is not working

Re: events not deleting with | delete

Re: forwarding accross networks

Re: Why does Splunk have multiple indexes?

Re: REGEX help on field extraction?

Re: How do I increase the number of series display...

Re: REGEX help on field extraction?

Re: Manager view for power user does not list any ...

Manager view for power user does not list any opti...

Re: examples of searches to capture network thrupu...

Re: All Windows Event Logs but not others?

Join the Conversation