About JDukeSplunk

JDukeSplunk · ‎12-20-2017

Nice try, but it fails completely. We have a subset of data where a few nodes were downed, and searching in that time range returns no results. If I lop off the |where statement and add the field "shouldAlert" to the table I see it, but it only contains the word "shouldAlert". I tried retooling the eval thinking that the != was throwing it. To this. | foreach *state [eval shouldAlert=if('<<FIELD>>'=="down","DOWN","UP")] Now shouldAlert appears once per line, and is showing as UP.

JDukeSplunk · ‎12-20-2017

I see where your going with this, but the search resulted in nasty errors and only returned the first three *state fields. Oddly enough, two of them were "active" 5 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors. [atlitpsplnk1_ind] Failed to parse templatized search for field '4DParticipant.node1.state' [atlitpsplnk1_ind] Failed to parse templatized search for field '4DParticipant.node2.state' [atlitpsplnk1_ind] Failed to parse templatized search for field '4DParticipant.node3.state' [atlitpsplnk1_ind] Failed to parse templatized search for field '4DParticipant.node4.state' [atlitpsplnk1_ind] Failed to parse templatized search for field '4DParticipant.shard1.state'

JDukeSplunk · ‎12-20-2017

I don't think rangemap likes decimals. Maybe the answer posted here using |eval case will be a suitable substitute. https://answers.splunk.com/answers/5659/rangemap-does-not-work-with-decimals.html

JDukeSplunk · ‎12-20-2017

We're pulling in a JSON from an API call. I'd like to setup an alert that only shows when field state is NOT active. The problem is that there is a stupid number of fields created by the input and they change names (but always end in *state). So I can't just list them in the table. Here's the search I muddled out. It works, the alert fires, and the renames lop off some of the noise. However it's not ideal. index=application sourcetype=json_no_timestamp DOWN OR RECOVERING OR RECOVERY_FAILED | rename cluster.collections.* as * | rename *shard1.replicas.core_node* as *node* | rename *shards.* as ** | table host *state It returns this. (This is an small slice of the screen). It would make the alert less cluttered if "active" nodes didn't show. It feels like this should be a subsearch / where, but I can't make that work with the wildcard *state. Here's an example of the _raw if you would like it. {"responseHeader":{"status":0,"QTime":4},"cluster":{"collections":{"CPTCodes":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"CPTCodes_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"CPTCodes_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":197,"configName":"CPTCodes"},"Q4LParticipants":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"Q4LParticipants_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"Q4LParticipants_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":568,"configName":"Q4LParticipants"},"nextgen":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"nextgen_shard1_replica1","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"},"core_node2":{"core":"nextgen_shard1_replica2","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":479,"configName":"nextgen"},"NamesDictionary":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node2":{"core":"NamesDictionary_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"},"core_node3":{"core":"NamesDictionary_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":377,"configName":"NamesDictionary"},"Providers":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"Providers_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"Providers_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":569,"configName":"Providers"},"Q4LClientData":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"Q4LClientData_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"Q4LClientData_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":415,"configName":"Q4LClientData"},"nextgencloud":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"nextgencloud_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"nextgencloud_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":494,"configName":"nextgencloud"},"workbaskets":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"workbaskets_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"workbaskets_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":341,"configName":"workbaskets"},"operators":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"operators_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"operators_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":479,"configName":"operators"},"4DParticipant":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"4DParticipant_shard1_replica1","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"},"core_node2":{"core":"4DParticipant_shard1_replica2","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":394,"configName":"4DParticipant"},"medications":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"medications_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"medications_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":564,"configName":"medications"},"SearchAllParticipants":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"SearchAllParticipants_shard1_replica1","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"},"core_node2":{"core":"SearchAllParticipants_shard1_replica2","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":635,"configName":"SearchAllParticipants"},"conditions":{"replicationFactor":"1","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"conditions_shard1_replica1","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"},"core_node2":{"core":"conditions_shard1_replica2","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":474,"configName":"conditions"},"Q4LProviders":{"replicationFactor":"2","shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{"core_node1":{"core":"Q4LProviders_shard1_replica1","base_url":"http://ATLAPDSOLR02:8983/solr","node_name":"ATLAPDSOLR02:8983_solr","state":"active","leader":"true"},"core_node2":{"core":"Q4LProviders_shard1_replica2","base_url":"http://ATLAPDSOLR02:9093/solr","node_name":"ATLAPDSOLR02:9093_solr","state":"down"}}}},"router":{"name":"compositeId"},"maxShardsPerNode":"1","autoAddReplicas":"false","znodeVersion":181,"configName":"Q4LProviders"}},"live_nodes":["ATLAPDSOLR02:8983_solr"]}}

JDukeSplunk · ‎09-05-2017

So I have a working example of this The PegaVerbose.csv looks like this PegaAlertNumber PegaAlertV PegaLevel PEGA0001 PEGA0001-HTTP interaction time exceeds limit Medium PEGA0002 PEGA0002-Commit operation time exceeds limit High PEGA0003 PEGA0003-Rollback operation time exceeds limit Medium PEGA0004 PEGA0004-Quantity of data received by database query exceeds limit High PEGA0005 PEGA0005-Query time exceeds limit Medium And the search I use is index=application sourcetype=Pega:Prod (PegaAlert=*) |lookup PegaVerbose.csv PegaAlertNumber as PegaAlert OUTPUT PegaAlertNumber PegaAlertV PegaLevel It matches up an event in the actual indexed data (the field named PegaAlert) with PegaAlertNumber in the csv, and then adds two new fields PegaAlertV PegaLevel that are available then for whatever you want. We also use this app for editing lookup files. https://splunkbase.splunk.com/app/1724/ Good luck.

JDukeSplunk · ‎08-29-2017

I would stop the UF on one system first, delete the app from the ../etc/apps folder, start the service and let the app redeploy. If that fails, I would upgrade or reinstall the UF software. Then lastly I would uninstall/reinstall. It stands to reason that the app is not the problem or it would not have deployed successfully to other hosts.. But maybe check the splunkd log these UF's for complaints about the app if it still fails. I can't guarantee anything. I'm just citing personal experience with picky UF's and deploying apps. We even had some linux UF's that wouldn't take the SPLUNK_TA_NIX app until the partial app was removed and the service restarted.

JDukeSplunk · ‎08-28-2017

I have seen this before on some versions of the UF. Some do not like the code in one of my apps. Aside from performing an upgrade of those UF's, you might isolate the failing hosts in a different server class with the same apps, remove the app from that server class. Then, either create an empty app (just a folder skeleton with maybe an empty /local/inputs.conf file and name it _force_splunk_recycle) and set that app to restart splunk, or have your custom app force a restart (but that might restart the service 1000 windows boxes) . Then deploy the app back to that server class along with the _force_splunk_recycle app you made. Basically this just removes the app, puts it back, and forces splunkd to restart on the UF's. Sometimes this works for me, sometimes I have to RDP and reinstall Splunk.

JDukeSplunk · ‎08-23-2017

These are my forwarders, and how they show up on my deployment server. Since their forwarder is not being reset with a "splunk clone-prep-clear-config" they are reporting back to the deployer with the name of the host they were cloned from. This makes it difficult to remove monitoring from a host, or change what apps are deployed. They report to the indexer fine, as their actual hostname.

JDukeSplunk · ‎08-22-2017

Shameless self bump.

JDukeSplunk · ‎08-21-2017

Our server team sometimes clones hosts without running "splunk clone-prep-clear-config". I recently found a handful of these simply because I knew that it had happened. They were all reporting back to the deployment server as the same name. After I had them run splunk clone-prep-clear-config and restart the service, all 9 of them appeared on the deployer. Is there a search that I can run to identify duplicate hosts/GUID's by IP(or something) on the deployment server?

JDukeSplunk · ‎08-16-2017

I've looked through the Splunk documentation for something that says "Once data is indexed, it cannot be edited". I can't find it. We are using Splunk to as evidence in a number of our audit items and we must prove that the data in Splunk is as accurate. I know this to be true, but I want a pdf I can link them to. Does anyone have a link that isn't back to answers.splunk.com?

JDukeSplunk · ‎08-11-2017

I don't know this app, but most things in Splunk are file based with .conf files driving them. I would try to find the file that this UI populates, and check the permissions on it. Or, just edit it manually. Maybe check in splunkhome/etc/APPNAME/system/default/something.conf for an example file, and save yours in splunkhome/etc/APPNAME/system/local/. Good luck

JDukeSplunk · ‎08-11-2017

When I've run into an issue with a log file posting future timestamps, (Wrong timezone, wrong timestamp parsing, etc) an "All Time" search will usually show them. Obviously, make the search as specific as you can to limit the results. I've not tried it, but the _indextime command might bear some fruit.

JDukeSplunk · ‎07-31-2017

Could you provide some sample data?

JDukeSplunk · ‎07-27-2017

I think you can basically take the whole /opt/splunk/etc/* folder from the old box and plop it down on the new one provided they are the same versions. That said, you will need to replace the host name in a few places. grep for your hostname

JDukeSplunk · ‎07-25-2017

So when you search this data you do not get the fields date_hour date_mday date_minute etc? Does adding this to your search add a new field named "indextime" ? | eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S")

JDukeSplunk · ‎07-24-2017

You will need to get the linux .tgz splunk file for your version, and extract it to a temporary location. Then using linux cp command to put the file default.xml where it needs to go. You tar command will look something like this. tar zxf splunk-xxxxxxx-Linux-x86_64.tgz -C /tmp/ OR.. Extract it using winrar on a Windows box, and use a tool like winscp to copy the files around that way.

JDukeSplunk · ‎07-24-2017

I am not versed enough in transforms to help you in this regard. Only from personal experience have I, when only a hand full of fields were needed to be renamed, simply used the GUI to alias them. With props and transforms, it's trial and error, and google searches. I typically import some sample data onto a development stand-alone splunk server and try it out there.

JDukeSplunk · ‎07-24-2017

Ok, so for this one you have edited a default file. As with so many things splunk, never edit /default/filename. Instead, edit /local/filename. So in your case, copy the default.xml file from /opt/splunk/etc/apps/search/default/data/ui/nav/default.xml to /opt/splunk/etc/apps/search/local/data/ui/nav/default.xml You may have to make the directories.. not sure. Then extract the contents of a Splunk install for your version, dig out the default.xml file, and put it back in /opt/splunk/etc/apps/search/default/data/ui/nav/default.xml Or copy it from another server that is not failing the integrity check. You may have to do some comparing on the files, to see if the xml file has changes worth keeping or if it's just an older version of the file.

JDukeSplunk · ‎07-21-2017

Go to https://(splunk)/en-US/manager/nest/data/inputs/nest and add new. This is where you put in your nest information.

JDukeSplunk · ‎07-21-2017

Nice! I tinkered with it a little to combine host and drive letter. I also reversed green/red so that lower percent free space will show up as critical. I also rounded off the decimal and instead of using an average, I am using "last" value. Since "last" is a more accurate representation of the servers current disk usage. sourcetype="Perfmon:Free Disk Space" counter="% Free Space" NOT instance=_total |eval host=host." ".instance | chart eval(round(last(Value)/1,0)) as FreeSpacePercent by host |sort - FreeSpacePercent | eval greenOK = if(FreeSpacePercent >= 85,FreeSpacePercent ,0) | eval yellowWarning = if(FreeSpacePercent > 55 AND FreeSpacePercent <=84,FreeSpacePercent ,0) | eval redCritical = if(FreeSpacePercent < 54,FreeSpacePercent ,0) | table host,redCritical,yellowWarning,greenOK

JDukeSplunk · ‎07-21-2017

Is this the app you are using? https://splunkbase.splunk.com/app/1729/ I run it on my home splunk instance, I'll check when I get there.

JDukeSplunk · ‎07-21-2017

So, I know you already have an answer for this, but here is a search we have that shows the splunk forwarder version, if its a "uf" or a full install, and what OS the host is. This can just be run on your search head, assuming the host you want to know about is reporting to your indexer. index=_internal source=*metrics.log* group=tcpin_connections | regex hostname!="\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" | eval sos_server=hostname | stats latest(sourceIp) AS IP latest(arch) AS cpu_arch latest(fwdType) AS forwarder_type latest(os) AS os_name latest(version) AS version by sos_server

JDukeSplunk · ‎07-21-2017

I use this app, and as I recall you have to provide it with your Nest.com login and password in the app's settings. Other than that, I don't remember having to do anything special.

JDukeSplunk · ‎07-21-2017

I thought you wanted a summary count for all your hosts. Anyway, try this, and combine with this article for colors. http://docs.splunk.com/Documentation/Splunk/6.1/Viz/BuildandeditdashboardswithSimplifiedXML#Specify_custom_colors_for_fields_in_charts Remove the "instance" if you want all disks. index=main sourcetype="Perfmon:Free Disk Space" counter="% Free Space" instance=C: | chart avg(Value) as PercentFree by host |sort - PercentFree

Posts	240
Solutions	24
Karma Given	19
Karma Received	61
Member Since	‎07-28-2015

Online Status	Offline
Date Last Visited	‎09-29-2022 02:03 PM

Tstats datamodel combine three sources by common f...

Some time questions.

Use lookup table to populate otherwise empty rows

Getting uptime of a process from PS

Field Extraction regex, stop at word or $

indexes.conf sanity question.

8.0.1 upgraded Heavy Forwarder- TcpOutputProc - Po...

Nest App, Google Migrated Account, 406 errors.

DB Connect, MSSQL Availability Group, Read Only In...

Need SEDCMD Help.

Re: Wildcard field used in table, return only fiel...

Re: Wildcard field used in table, return only fiel...

Re: How to group my search results with respect to...

Wildcard field used in table, return only fields w...

Re: Use lookup table values as a input for search ...

Re: Unable to push customized app from deployment ...

Re: Unable to push customized app from deployment ...

Re: Identify Cloned Hosts on Deployer?

Re: Identify Cloned Hosts on Deployer?

Identify Cloned Hosts on Deployer?

Index Fidelity for Audit Purposes, proof.

Re: Unable to configure Website Monitoring App

Re: What search can I use to identify logs with fu...

Re: Has anyone successfully parsed Exim logs into ...

Re: How to move splunk from one location to anoth...

Re: how to extract date from filename and add it w...

Re: Files Integrity Checker error ?

Re: How to write props & transforms to apply for f...

Re: Files Integrity Checker error ?

Re: How do you configure the nest thermostat and S...

Re: Bar color based on value on chart

Re: How do you configure the nest thermostat and S...

Re: How can I tell if a heavy forwarder or univers...

Re: How do you configure the nest thermostat and S...

Re: Bar color based on value on chart

Join the Conversation