About maciep

maciep · ‎02-10-2016

Aside from making it CIM-compliant, it would also be helpful if we could have those index-time, search-time operations available in standalone TA too. ES is supposed to live on a search head by itself, so we don't want to have to install the entire bit9 app (with datamodels and saved searches) on ES. Just need the search time operations (props, transforms, lookups, etc) so that CIM mappings are available to ES.

maciep · ‎10-28-2015

Thanks for responding so quickly, Guilhem! I really appreciate it! I know very little about nmon myself, so I wouldn't have even thought the files were missing the performance data or timestamp. I'll review the nmon repository to see if I can find better ones to test with. If not, I'll engage our linux team that actually manages nmon to see if they an assist. Maybe all they need is the config data? And I will upgrade to the latest release. Not a problem at all. Thanks for confirming that I only need the index on on the indexer, no apps. If all goes well in our test environment, we'll migrate to production. In that environment we have a SHC, Indexer Cluster, Deployment Server and a Heavy Forwarder we can use for this process. With your documentation and a better understanding of how the app works in general, I don't think we'll have a problem with the deployment there. I'll follow up here once I have some more results. Thanks again!

maciep · ‎10-28-2015

I'm hoping to get some help with the NMON app, because I'm having trouble getting anything to show up in the nmon_data sourcetype. In our environment, we're in "cold" mode. Our linux team has a central repository for nmon files. They get copied from the servers to the central repository once a day. For now, I'm just trying to get his working in our Test splunk env. For that environment, we have a Search Head and an Indexer. The Search Head has access to the central repository, so it is the forwarder in this case as well. I have the nmon app deployed to the search head and created a local inputs.conf to monitor our share. I also disabled the default nmon monitor that pointed $SPLUNK_HOME/var/run/nmon.... I created the index manually on our indexer and also installed the PA-nmon app as well (although not sure if necessary since parsing should take place on the search head). I disabled all of the inputs in that app. Once I restart Splunk, I do see that the nmon files are processed (nmon_processing sourcetype), but no perfdata ever gets indexed. I verified that Splunk is watching the csv_repository directory. But I don't think anything ever shows up in that directory. I've tested the script manually by cat'ing a file directly to the script (as outlined in the nmon wiki). When running the python script, there was an error about "Encountered an Unexpected error while trying to analyse the ending period of this Nmon". So I updated nmon2csv.sh to launch the perl script instead. When testing the perl script directly, the output looks good (I think). But no nmon.csv files are created there either, only the config data is created. I'm not sure what I'm missing. I follow the script as best I can, but not well enough to determine in what scenario the nmon.csv file doesn't get created. Any help would be appreciated. I didn't want to clutter this initial post with all of the config files / script outputs / splunk logs / etc. But I can provide whatever might help resolve the issue.

maciep · ‎10-28-2015

Glad you were able to resolve and glad it was something silly...thanks for following up!

maciep · ‎10-26-2015

first off, do you need to set it to that on your universal forwarders. Can't you just configure your outputs.conf to send data to all of your indexers in the cluster? I thought the site0 functionality was just for search head affinity but could be wrong. That said, you could just deploy an app with your specific server.conf settings along with the uf install. So install install the uf, copy the app to etc/apps and restart splunk. And depending on how many forwarders you will be managing, you may benefit from implementing the splunk deployment server. Or you could just use puppet/sccm to manage configurations similarly.

maciep · ‎10-15-2015

This seems to work with a limited set of test data. But it's also late, so it could be way off too 🙂 Essentially, I'm trying split the event by the hashtags, and then expand those sections into separate events. So instead of 1 event above, you'd have 4. And then I rex the serial and bu for each of those new events (so each section of the original 1 event). Then across all events, i do a stats for the largest bu by _raw (even after the expand, all of those new events still have the same _raw). So now each event has a _raw, serial, bu and the max(bu) that represents the highest bu across the same original event. So finally, I filter on where the max field is the same as the bu field, since those will only match for those events that represent the largest bu per _raw. And that's it, just table the remaining serial and bu fields. Hopefully that makes sense (and is logically correct) index=* sourcetype="test:serial" | eval blah = _raw | makemv blah delim="#############################" | mvexpand blah | rex field=blah "Serial:\s+(?<serial>\S+)[^-]+-(?<bu>.+)" | eventstats max(bu) as max by _raw | where max=bu | table serial bu

maciep · ‎10-15-2015

I don't suppose you're in fast mode (instead of say smart mode)? I mean, from everything else you said I'm guessing not, but thought it worth asking. If I were you, I'd start narrowing down where the problem might be. For example, can you chart/timechart/table other fields. So is the problem with stats, or any command or a subset of commands etc. Also, maybe go back to the beginning of the search after the first pipe and just try to do a stats count by MEDIA_TYPE (or whatever). Does that work? If so, add more of the search until it doesn't work again. Kind of hard to go through all of those troubleshooting permutations here, but I think you get the idea.

maciep · ‎10-15-2015

can you group your results by any other field?

maciep · ‎10-15-2015

I don't think this is what you're looking for exactly, but this should give you an idea of the files splunk is tailing. index=_internal sourcetype=splunkd component="TailingProcessor" "adding watch"

maciep · ‎10-15-2015

if you remove the stats all together, does your result set have a field called MEDIA_TYPE? All uppercase with an underscore? It would seem just off hand that the field doesn't exist, so the by clause is null, so you don't see any results.

maciep · ‎10-15-2015

The change was made last night. All that I did was put the cluster in maintenance mode. I think if the master would have seen all cold data disappear from half or our indexers, it would have tried to clean things up. I'm not sure if that was necessary, but wasn't going to take a chance otherwise. Looking in the logs on the servers that lost their cold path, I see a lot of these messages. They start at the time of the change and they stop after the share was remounted. That was it. 10-14-2015 19:34:05.437 -0400 WARN DatabaseDirectoryManager - Directory='/[path to cold]/[some bucket]' does not exist. The directory representing this bucket might have just rolled.

maciep · ‎10-14-2015

yep, support would be able to help with that. That's probably the right place to go now. Once you get this resolved, just be sure to update this question with the final answer.

maciep · ‎10-14-2015

Nothing is jumping out at me here. Just to verify, you're lookups and evals from props actually work - you see them working in the web? What if you run btool for just those stanzas in transforms? splunk btool transforms list setnull --debug It might be worth just trying to create a new transforms.conf and typing everything in again. I know that's a long shot, but why not at this point.

maciep · ‎10-14-2015

It's kind of hard to give you advice without actually knowing how you will ultimately use the data. With a multi-value dropdown, you need to build your token correctly so that it makes sense when you substitute in your search. As an example, let's say I wanted to do something similar for a list of sourcetypes associated with the wineventlog index. So ultimately in my search, I want my token to be sourcetype="somesourcetype". Or int he case of the all option, it should be (sourcetype="something" OR sourcetype"something:else" OR ...) So I create one search for the list of sourcetypes, modify the field that will be the value of the token and create another field that will be the label in the dropdown. I think append to that search, a search that will do a similar thing, but will combine all of the sourcetypes into one field, format it the way I need it and then give it a label of "All". So for this example, this would be the search that populates the dropdown. The sourcetype field would be used for the value and the label field would be used for the label: | tstats count where index=wineventlog by sourcetype | fields - count | rex field=sourcetype "WinEventLog:(?<label>.+)" | eval sourcetype = "sourcetype=".sourcetype | append [ | tstats count where index=wineventlog by sourcetype | fields - count | eval sourcetype = "sourcetype=".sourcetype." OR" | mvcombine sourcetype | makemv sourcetype delim="\n" | eval sourcetype = "(".trim(sourcetype," OR").")" | eval label="All" ] So the results look like this (sorry for the formatting) sourcetype label sourcetype=WinEventLog:Application Application sourcetype=WinEventLog:Security Security sourcetype=WinEventLog:System System (sourcetype=WinEventLog:Application OR sourcetype=WinEventLog:Security OR sourcetype=WinEventLog:System) All

maciep · ‎10-13-2015

Yeah, I feel like there's something odd happening here. Where are you creating your props and transforms? Could you include them both in their entirety? And you have just the one server acting as the forwarder, indexer and search head? And all of the config is being done on that server?

maciep · ‎10-13-2015

So if you're still in the testing phase, maybe set REGEX to "." (no quotes). If everything gets discarded, then at least we know we made it to the stanza and it sent info to the null queue as expected. If not, then either the stanza in transforms is wrong or we never get there.

maciep · ‎10-13-2015

Ok, I think I understand now. I'm not sure if there is a way to inherently allow users to select all of the items in a multi-select dropdown (that would be nice). I can poke around the docs though to check further. You may be able to update the search that populates the IP multi-select to also return an "All" field with the relevant values for all of the IPs that were returned. But then you'd have to make sure that the way you use the token suffix/prefix settings still make sense. If you'd like, I think you can edit your question and include the dashboard source itself (in a code block). It might give me or others here a better idea of how it might work?

maciep · ‎10-12-2015

I think that is expected. This is an excerpt from the doc linked below (see the second item): For master nodes running Splunk Enterprise version 6.2 and above Peer nodes and search heads can run different versions from the master, subject to these restrictions: The peer nodes and search heads must run version 6.1 or above. The peer nodes and search heads must run the same or a lower version than the master node. The peer nodes and search heads within a site must all run the same version. http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/Systemrequirements#Splunk_Enterprise_version_compatibility The upgrade documentation is out there as well. I think it says you need to take the entire cluster down to do the upgrade, but that's a bit insane. There is another recent question out here somewhere about a similar downtime concern.

maciep · ‎10-12-2015

You don't need to have all of your extractions in one command. So why not just create a separate extraction for destination code that matches on the actual text "CDC_HB_RCV". Matybe something like this? CDC_HB_RCV\s\w\s(?<destination_code>\w) Or is not that consistent? Meaning, can the rcv events not have a dest code? And/Or can some send events have one?

maciep · ‎10-12-2015

It's kind of hard to follow your scenario. But in general, we usually handle that with a static "All" option with a value of * . So if a user chooses All from the list, then the token equals * . And so when you use that token later, it should match everything.

maciep · ‎10-11-2015

Thanks! I think we have plenty of disk space on our hot/warm mount to allow Splunk to keep running. If we decide to go that way, then I'll definitely follow up here with how it went.

maciep · ‎10-11-2015

Just a long shot, but did you copy and paste this search from somewhere else? Wondering if maybe the dash is like a long dash or something along those lines. If you typed it out yourself in the search bar, then that shouldn't be a problem.

maciep · ‎10-09-2015

I haven't used it but there is a function for eval/where called cidrmatch - seems to do just that. http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/CommonEvalFunctions cidrmatch("X",Y) This function returns true, when an IP addresse Y belongs to a particular subnet X. The function uses two string arguments: the first is the CIDR subnet; the second is the IP address to match. This example uses cidrmatch to set a field, isLocal, to "local" if the field ip matches the subnet, or "not local" if it does not: ... | eval isLocal=if(cidrmatch("123.132.32.0/25",ip), "local", "not local") This example uses cidrmatch as a filter: ... | where cidrmatch("123.132.32.0/25", ip)

maciep · ‎10-09-2015

I think your regex is probably fine. And I'm assuming once indexed you verified that the line break works - meaning, the entire second row is one event? If so, I'm guessing you're never getting to the stanza in transforms. And I'm wondering if it's because you need to double up on your backslashes in your source stanza. This is from the props.conf spec file on docs. **Considerations for Windows file paths:** When you specify Windows-based file paths as part of a [source::<source>] stanza, you must escape any backslashes contained within the specified file path. Example: [source::c:\\path_to\\file.txt] **[<spec>] stanza patterns:** And you could also probably just call your transforms from the sourcetype stanza too. Not sure if you needed to call it in a source stanza or if that was just from following the docs article you referenced.

maciep · ‎10-09-2015

I don't think there is a function for that, but maybe you can use the answer below to save yourself some work. I guess you could just flip the bits during the replace as well? http://answers.splunk.com/answers/222813/convert-an-hexadecimal-field-to-binary.html#answer-221785

Posts	826
Solutions	118
Karma Given	98
Karma Received	251
Member Since	‎09-18-2013

Online Status	Offline
Date Last Visited	‎10-22-2024 07:27 AM

Getting nested json values into Azure Log Analytic...

Why is distsearch.conf whitelisting all json files...

Any options to speed it up a falling behind JMS Me...

Is there a way to replace _raw at search time with...

Microsoft OMS TA - Any plans to update to the API ...

Best way to move half of a SHC?

How to create a custom streaming command with pyth...

Can a scheduled dashboard pdf delivery be sent to ...

Can we ignore timestamps for some of the inputs in...

How is the partial flag supposed to work with time...

Re: Are there plans to make the Bit9 Security Plat...

Re: How do implement the nmon app for cold data?

How do implement the nmon app for cold data?

Re: Discarding events using TRANSFORMS-null

Re: How to set the site during Universal Forwarder...

Re: Extracting field based on prior field extracte...

Re: Avg function doesn't output any value when use...

Re: Avg function doesn't output any value when use...

Re: Is there a REST API call or other method to ch...

Re: Avg function doesn't output any value when use...

Re: What happens if the cold mount becomes unavail...

Re: Discarding events using TRANSFORMS-null

Re: Discarding events using TRANSFORMS-null

Re: How can I add "select all" button in 1 of my i...

Re: Discarding events using TRANSFORMS-null

Re: Discarding events using TRANSFORMS-null

Re: How can I add "select all" button in 1 of my i...

Re: Search head 6.3 not compatible with cluster 6....

Re: Help extracting a field from an event

Re: How can I add "select all" button in 1 of my i...

Re: What happens if the cold mount becomes unavail...

Re: What's wrong with my rangemap?

Re: Check IPs on Blacklist in CIDR notaiton

Re: Discarding events using TRANSFORMS-null

Re: two's complement

Join the Conversation