About maciep

maciep · ‎09-30-2015

Hi all, I'm looking for some advice on source version control. We have a couple deployers, a deployment server, indexer cluster, bunch of forwarders and various supporting splunk servers. And we of course have a lot of different apps/TAs that need deployed to different parts of the infrastructure, typically with some overlap. From everyone we talked to before and the sessions at conf, it seems like we need to come up with a source control strategy and git seems like a good tool to use. I've been reading the basics of using git (never have before), but trying to understand how best to use it with our splunk infrastructure. Should we just have some central location where all of the apps and their repositories are? And then when we need to make a change, we check it out, copy it to our test instance, modify as necessary, copy back and commit? And then just manually copy the updated directory out to all of the places it needs to go. Or do we actually have repositories on the various deployment servers and sync those up to some master repository. For example, our main indexes app goes out to the both shc and in the cluster master. Would the apps in all 3 locations be actual git repositories? Sharing some master repository? For those that do something similar, any advice or lessons learned you could share would be appreciated. Right now we're just using the old central share approach with a lot of copying around. Thanks!

maciep · ‎09-30-2015

you can use a search command called "rex" to grab a named regex capture group, and splunk will create a new field for it. In this case, I'm trying to grab the 8th digit after the Latency part of the event and putting it in a field called "cool_num". And then you can just create a table of _time (the timestamp) and then new field you extracted. Your base search .... | rex "Latency:\s+(\d+\s+){7}(?<cool_num>\d+)" | table _time cool_num

maciep · ‎09-29-2015

my cron skills aren't that good, but instead was wondering why not just make two separate alerts? Since it does seem like those are two different schedules. Just a thought....

maciep · ‎09-28-2015

I'm still confused here. It looks like that log sample you provided is from a Splunk internal log on the indexer maybe? It's just giving you metrics on what is being indexed. I don't think you want to use that as your base for filtering. Do you have a sample of the actual log data you want to filter out. To apply the transforms you want, the regex has to match that data, not the log format of splunk's internal log. For example, if you search for the following, what do those raw events look like? index="main" sourcetype="syslog" host="rcmk"

maciep · ‎09-26-2015

Not sure if that's possible. Do you have the ability to change the format of that field to include the date you want? Or add other fields for month/day/time that you could use to build the timestamp? The TIME_FORMAT setting tells splunk how to identify the timestamp in the event, not what to set it to. Since you only have the year field, it obviously won't find the rest of the timestamp you have in there.

maciep · ‎09-18-2015

it depends on where you have the where. If it's after the stats, then you don't have a field called Comment1, you have a field called "latest(Comment1)". So maybe try putting it before the stats. Or try what somesoni2 mentioned to keep the original field names.

maciep · ‎09-18-2015

Look like you are applying that change to all data coming from that host. That includes splunk's internal log files. Are you sure that you're not just accidentally sending the _internal data to the winevent_dmz index now? Maybe along with the eventlog data? That would explain the splunkd sourcetype. Also, it might be easier to just do this from the frowarder on winserver01 in the inputs.conf. Just update the index there. If you have windows app installed, just be sure to make your changes in the local directory instead of default.

maciep · ‎09-18-2015

do the log entries actually have key-value pair in them like your regex does? Meaning, do the logs literally read "Host=rccrxxxx"? And if so, is that literally the first thing in the log (as opposed to a timestamp for example)? Have you tried just putting rccr as the REGEX value?

maciep · ‎09-18-2015

Yeah, where doesn't like wildcards, but you can use it with match function. .... | where match(comment_field,".")

maciep · ‎09-15-2015

which app are you running the scheduled search from and is that the same app where the lookup is being written? Sounds like it might be running from a different app and just creating an empty lookup there.

maciep · ‎08-28-2015

We're not using db connect 2.x yet. But in 1.x if you have the jdbc jar and know the connection settings, you should be able to create your own connection type. It looks like you should be able to do something similar in 2.x too: http://docs.splunk.com/Documentation/DBX/2.0.4/DeployDBX/Installdatabasedrivers#Install_drivers_for_other_databases http://docs.splunk.com/Documentation/DBX/2.0.4/DeployDBX/databasespec

maciep · ‎08-28-2015

Before my time here we had something kind of similar in active/inactive state. It was actually two syslog servers. Both servers would get the same data in the same folders/files but only one would have the forwarder running at any given time. The trick though was to put the fishbucket on a mount point and then symlink it on both servers from the normal fishbucket location. So the failover scenario was still manual - meaning we had to start up splunk on the backup server. But when it started, it was using the same fishbucket as primary so it knew where to start reading files from. I'm not sure how good of a solution that was but it could be an option for you. As long as the forwarders are reading from the same place and share a fishbucket, I guess it would work? In general though, we don't worry much about HA for forwarders. We have monitoring in place to start splunk if it stops and we get a daily report (from the Deployment Monitor app) of forwarders that haven't checked in to our deployment server. So typically we can address stopped forwarders before the data rolls. Hope that helps a little

maciep · ‎08-28-2015

Just messing around in my environment it looks like you can rex out the xml to another field and then use spath to find specific data in there. For example, this will put the xml for the task in a field called task_xml. Then pipe to spath and looking at that new field, grab the Action execution settings (putting it in a new field called action_command) index=wineventlog EventCode=4698 | rex "(?i)(?<task_xml>\<task[\s\S]+\<\/task\>)" | spath input=task_xml output=action_command path=Task.Actions.Exec.Command

maciep · ‎08-28-2015

Not sure how consistent that log format is, but something like this seems to work for me in a limited test env. I'm just using rex to grab the "*" portion of the event and throw it in a field called xml_field ... | rex "(?<xml_field>\<root\>[\s\S]+\<\/root\>)"

maciep · ‎08-20-2015

First off, are the dollar signs in your timestamp format typos or actually what you tried? They should be percent signs. For the second part I think the urldecode function should work. As an example | noop | stats count | eval blah="apps.baplc.com%2Ftravel%2Fgeneral%2Fpublic%2Fen" | eval meh =urldecode(blah)

maciep · ‎08-20-2015

I'm not sure I can provide the best advice but that does seem like a lot of indexes. I think generally speaking the two questions to ask when creating a new index are: Do I need to control access differently to this data? Will this data have different retention requirements? If the answer to those are no, then you don't necessarily need a new index. So if every application has to be limited to only those app owners/support folks, then having them separate might make sense. Or if you want to keep data from different applications longer/shorter period time, then it might make sense. Otherwise, it might not be necessary. That said, we do have some indexes for just applications. We have some for specific support teams. We have others for like infrastructures. Unfortunately or fortunately, we don't have any hard and fast rules. We try to use common sense as our guide posts, but that may not be the best approach. I would just caution to be careful of handcuffing yourself down the line. Make sure that whatever path you choose, you leave wiggle room to adjust for those unknown scenarios.

maciep · ‎08-18-2015

ah, I think I lose num_users after the second stats. So maybe this would work too? Or just eventstats both of them? ... | eventstats dc(userID) num_users | where is_successful=false | stats dc(userID) as num_failed_users max(num_users) as num_users by version | eval percent_failed_users = num_failed_users/num_users

maciep · ‎08-18-2015

well at least you found something but I'm not sure how to interpret those logs in a helpful way. If that's the actual dn for your account, I don't know why splunk and/or your bind account can't find it. If you have support, it may be worth opening a ticket at this point.

maciep · ‎08-17-2015

I thought when you said you wanted percent of users, that meant you needed a distinct count of all users as well (to get the percentage). That's why I was using eventstats first...

maciep · ‎08-16-2015

oh ok, I think I misunderstood the requirement. If you sort by B, then C then A, will the events be in an order where autoregress could work? Or are the timestamps random enough where you still can't rely on looking before/after the current event? Still not sure if I understand the data set well enough to give this a shot....

maciep · ‎08-16-2015

I think you want to use autoregress Something like: ... | autoregress p=1 FieldD as Next_FieldD | where FieldD=200 AND Next_FieldD!=500 And ... | sort FieldA | autoregress p=1 FieldD AS Prev_FieldD | where FieldD=500 AND Prev_FieldD=200 Not sure if you want to see those results in the same search or separate, but I think you can pipe to autoregress to get the next field, re-sort by FieldA, autoregress again to get the prev field, then pipe to where to filter on those two scenarios.

maciep · ‎08-15-2015

You should be able to use the regex command to filter on events that match a criteria. What does your base search look like? Is that full log entry one event in your env? Something like this should work, but if none of these suggestions are working for you, then we may need a little more context... [your base search] | regex "\"reportReferenceNumber\" : 0," | [stuff to do with the results]

maciep · ‎08-15-2015

hmm, can't you do this with the with eval in the stats command, instead of using a case? ... | bin span=5m _time | stats count(eval(time<300)) as A count(eval(time>=300 AND time < 600)) as B count(eval(time>=600 AND time < 1000)) as C count(eval(time>=1000 AND time < 3000)) as D count(eval(time>=3000)) as E by _time

maciep · ‎08-15-2015

Yep, it would be helpful to know which stanzas you're not getting logs from. Or if you're getting logs from all of the stanzas, but not every log you'd expect with a certain stanza if that makes sense. As mentioned by woodcock, if any of those files have large headers that are the same, then that might explain it as well.

maciep · ‎08-14-2015

with respect to that last apps question, the local folder on the the search heads are for changes made in the gui. So when you apply the bundle from your deployer, splunk runs through the whole precedence algorithm for the apps on the deployer - local vs default. The resulting files then will end up getting saved in just default folder under the app on the search heads themselves. If you then make changes in the splunk web for an app - create a search, update dashboard, create an extraction, whatever...those get saved to the local app folder on the search head and then they get replicated to the rest of the members. So now changes made in splunk web don't get overwritten when you redeploy from the deployer. But you can still have local/default folders in your app on the deployer. Hope that helps with that question at least. Not sure about alert config though...

Posts	826
Solutions	118
Karma Given	98
Karma Received	251
Member Since	‎09-18-2013

Online Status	Offline
Date Last Visited	‎10-22-2024 07:27 AM

Getting nested json values into Azure Log Analytic...

Why is distsearch.conf whitelisting all json files...

Any options to speed it up a falling behind JMS Me...

Is there a way to replace _raw at search time with...

Microsoft OMS TA - Any plans to update to the API ...

Best way to move half of a SHC?

How to create a custom streaming command with pyth...

Can a scheduled dashboard pdf delivery be sent to ...

Can we ignore timestamps for some of the inputs in...

How is the partial flag supposed to work with time...

Best way to use git for source version control wit...

Re: How do I create a custom dashboard plotting ti...

Re: How do I create an alert using a cron schedule...

Re: How to filter out syslog events from a group o...

Re: can I mix extracted and current timestamps whe...

Re: How do I remove rows from a table based on emp...

Re: How to change the target index for logs from c...

Re: How to filter out syslog events from a group o...

Re: How do I remove rows from a table based on emp...

Re: Why does my outputlookup search to truncate a ...

Re: How to connect Splunk to an Essbase database?

Re: Is there a way to configure high availability ...

Re: How to configure Splunk to extract XML fields ...

Re: How to extract XML data from mixed content int...

Re: How do I configure Splunk to recognize the non...

Re: What would be the best practice for creating i...

Re: How to search the percentage of occurrences of...

Re: After mapping groups to roles configuring Splu...

Re: How to search the percentage of occurrences of...

Re: associated query according to the context

Re: associated query according to the context

Re: How do I search for a field with the value 0 i...

Re: How to use eval and case results with bucket s...

Re: As a Splunk front end user, would I be able to...

Re: Search Head Clustering: Why is the Deployer no...

Join the Conversation