About pgreer_splunk

pgreer_splunk · ‎05-12-2021

I haven't used map much in the past, so didn't think of it. Thanks for the reminder! Had to set the 'maxsearches' option so that I didn't hit the default limit of 10. And did a '| search blarg' in the map subsearch pipeline just to make sure the results set was cleared out before going on to the next one (not necessary, just a sanity check). Worked well, but NOT recommended if you have 1,000's of lookups all at a HUGE size - I/O spike!!! 🙂 Otherwise, works quite well for this limited sync use case. Thanks!

pgreer_splunk · ‎05-11-2021

I have a use case where there are over 50+ lookup files that I need to 'sync' between one app context and another. The idea is to: 1) read the lookup from context of App1 search bar 2) outputlookup to a lookup file named 'UPDATE_<lookupname>.csv' that resides in App2 context The idea is to have the 50+ lookup file names in a lookup named myLookupFiles with App1. Then pass that filename as a field into a macro. So the gist would be to: | inputlookup myLookupFiles | `mySyncMacro(myLookupFileNameField)` And the macro would thus then be something like: join type=left max=0 [| inputlookup $myLookupFileName$] | fields - myLookupFileNameField | outputlookup createinapp=true UPDATE_$myLookupFileName$ | search blarg Which, of course, doesn't work. Thoughts on a way to iterate across all 50+ file names when they are specified as values within a table to create the 50+ named lookup files with the name "UPDATE_<lookupname>.csv"?

pgreer_splunk · ‎08-07-2018

There have been others that have sent email alerts to recipient email addresses that are defined within each event within the results set. I'm struggling, however, on how to send a subset of the results set to a single email address, then another subset of the results to another. Here's the (obfuscated) use case. First I need to acquire a set of hosts that have had a certain condition within the past month. For hosts in one geographic location, I'd like to send an e-mail alert to administrator Bob. For hosts in another geographic location, I'd like to send an e-mail alert to administrator Shelly. For hosts in X, send to Y. Now, if there were just 2 or three admins, separate alerts could be lived with. But what if we have hundreds of locations and hundreds of admins? I'd like a single alert to then send each hosts groups by location to an admin that is defined in a field within the results set. However, I don't want to send each admin an e-mail for each host (each event) in that group, but instead the entire list of hosts in the geography within a single e-mail alert to that admin. I'm thinking this can only be facilitated with a custom alert action (python scripting), but was wondering if anyone out there has done something similar with the built in email alert functionality and/or via SPL with the sendmail command and/or other SPL tricks.

pgreer_splunk · ‎10-16-2017

That set me on the right track. Having issues ignoring certain events from being ingested, so still working that front, but the events are breaking as desired at a "" within the data stream from the API. Thanks!

pgreer_splunk · ‎10-14-2017

I'm needing to split a stream of data (from a REST API call) that is CSV data, variable line lengths at the initial set. The split should be when the stream of data has two double quotes together -> "" <- Example data is: "AmazonEC2","Asia Pacific (Sydney)","AWS Region","m3.2xlarge","Yes","General purpose","8","Intel Xeon E5-2670 v2 (Ivy Bridge/Sandy Bridge)","2.5 GHz","30 GiB","2 x 80 SSD","High","64-bit",,,,,,,,"Dedicated","Windows","No License required",,,,,,,,"APS2-DedicatedUsage:m3.2xlarge","RunInstances:0002",,,"26",,,,,,,,,,,,,,,"16",,"NA","Intel AVX; Intel Turbo","Amazon Elastic Compute Cloud""FGNPDK5ZFJP4S9NC","MZU6U2429S","FGNPDK5ZFJP4S9NC.MZU6U2429S.2TG2D8R56U","Reserved","Upfront Fee","2016-08-31",,,"Quantity","17896","USD","3yr","All Upfront","convertible","Compute Instance","AmazonEC2","US West (N. California)","AWS Region","c3.4xlarge","Yes","Compute optimized","16","Intel Xeon E5-2680 v2 (Ivy Bridge)","2.8 GHz","30 GiB","2 x 160 SSD","High","64-bit",,,,,,,,"Shared","RHEL","No License required",,,,,,,,"USW1-BoxUsage:c3.4xlarge","RunInstances:0010",,,"55","Yes",,,,,,,,,,,,,,"32",,"NA","Intel AVX; Intel Turbo","Amazon Elastic Compute Cloud""QD2X48Z37JG3VNFX","HU7G6KETJZ","QD2X48Z37JG3VNFX.HU7G6KETJZ.6YS6EN2CT7","Reserved","Windows with SQL Server Enterprise (Amazon VPC), r3.2xlarge reserved instance applied","2016-11-30","0","Inf","Hrs","1.9700000000","USD","1yr","Partial Upfront","standard","Compute Instance","AmazonEC2","Asia Pacific (Tokyo)","AWS Region","r3.2xlarge","Yes","Memory optimized","8","Intel Xeon E5-2670 v2 (Ivy Bridge)","2.5 GHz","61 GiB","1 x 160 SSD","High","64-bit",,,,,,,,"Dedicated","Windows","No License required",,,,,,,,"APN1-DedicatedUsage:r3.2xlarge","RunInstances:0102",,,"26","Yes",,,,,,,,,,,,,,"16",,"SQL Ent","Intel AVX; Intel Turbo","Amazon Elastic Compute Cloud""DCM8ZJ894B27CQ8G","4NA7Y494T4","DCM8ZJ894B27CQ8G.4NA7Y494T4.6YS6EN2CT7","Reserved","Linux/UNIX (Amazon VPC), g3.8xlarge reserved instance applied","2017-06-30","0","Inf","Hrs","2.1400000000","USD","1yr","No Upfront","standard","Compute Instance","AmazonEC2","US West (N. California)","AWS Region","g3.8xlarge","Yes","GPU instance","32","Intel Xeon E5-2686 v4 (Broadwell)","2.3 GHz","244 GiB","EBS only","10 Gigabit","64-bit",,,,,,,,"Shared","Linux","No License required",,,,,,,,"USW1-BoxUsage:g3.8xlarge","RunInstances",,"7000 Mbps","0","Yes","2",,,,,,,,,,"Yes","Yes","Yes","64",,"NA","Intel AVX, Intel AVX2, Intel Turbo","Amazon Elastic Compute Cloud""EX33FD39CKVCKNYQ","MZU6U2429S","EX33FD39CKVCKNYQ.MZU6U2429S.2TG2D8R56U","Reserved","Upfront Fee","2017-04-30",,,"Quantity","14074","USD","3yr","All Upfront","convertible","Compute Instance","AmazonEC2","US West (N. California)","AWS Region","m4.4xlarge","Yes","General purpose","16","Intel Xeon E5-2676 v3 (Haswell)","2.4 GHz","64 GiB","EBS only","High","64-bit",,,,,,,,"Dedicated","Linux","No License required",,,,,,,,"USW1-DedicatedUsage:m4.4xlarge","RunInstances",,"2000 Mbps","53.5","Yes",,,,,,,,,,,,,,"32",,"NA","Intel AVX; Intel AVX2; Intel Turbo","Amazon Elastic Compute Cloud""QGQ2W8XX4J2CGD82","4NA7Y494T4","QGQ2W8XX4J2CGD82.4NA7Y494T4.6YS6EN2CT7","Reserved","Red Hat Enterprise Linux (Amazon VPC), m4.xlarge reserved instance applied","2017-04-30","0","Inf","Hrs","0.2154000000","USD","1yr","No Upfront","standard","Compute Instance","AmazonEC2","Asia Pacific (Singapore)","AWS Region","m4.xlarge","Yes","General purpose","4","Intel Xeon E5-2676 v3 (Haswell)","2.4 GHz","16 GiB","EBS only","High","64-bit",,,,,,,,"Shared","RHEL","No License required",,,,,,,,"APS1-BoxUsage:m4.xlarge","RunInstances:0010",,"750 Mbps","13","Yes",,,,,,,,,,,,,,"8",,"NA","Intel AVX; Intel AVX2; Intel Turbo","Amazon Elastic Compute Cloud""DZS3NEJDE8E98442","4NA7Y494T4","DZS3NEJDE8E98442.4NA7Y494T4.6YS6EN2CT7","Reserved","Windows with SQL Server Standard (Amazon VPC), i3.4xlarge reserved instance applied","2017-06-30","0","Inf","Hrs","3.5970000000","USD","1yr","No Upfront","standard","Compute Instance","AmazonEC2","EU (Ireland)","AWS Region","i3.4xlarge","Yes","Storage optimized","16","Intel Xeon E5-2686 v4 (Broadwell)","2.3 GHz","122 GiB","2 x 1.9 NVMe SSD","Up to 10 Gigabit","64-bit",,,,,,,,"Shared","Windows","No License required",,,,,,,,"EU-BoxUsage:i3.4xlarge","RunInstances:0006",,"3500 Mbps","99","Yes",,,,,,,,,,,,,,"32",,"SQL Std","Intel AVX, Intel AVX2, Intel Turbo","Amazon Elastic Compute Cloud"

pgreer_splunk · ‎09-25-2017

Having a bit of a struggle. AWS has a pricing API available at: AWS JSON Pricing API URL Because of how the JSON is formatted, it looks like a custom handler is needed. Snipped of the JSON is: { "formatVersion" : "v1.0", "disclaimer" : "This pricing list is for informational purposes only. All prices are subject to the additional terms included in the pricing pages on http://aws.amazon.com. All Free Tier prices are also subject to the terms included at https://aws.amazon.com/free/", "offerCode" : "AmazonEC2", "version" : "20170921013650", "publicationDate" : "2017-09-21T01:36:50Z", "products" : { "76V3SF2FJC3ZR3GH" : { "sku" : "76V3SF2FJC3ZR3GH", "productFamily" : "Compute Instance", "attributes" : { "servicecode" : "AmazonEC2", "location" : "Asia Pacific (Mumbai)", "locationType" : "AWS Region", "instanceType" : "d2.4xlarge", "currentGeneration" : "Yes", "instanceFamily" : "Storage optimized", "vcpu" : "16", "physicalProcessor" : "Intel Xeon E5-2676v3 (Haswell)", "clockSpeed" : "2.4 GHz", "memory" : "122 GiB", "storage" : "12 x 2000 HDD", "networkPerformance" : "High", "processorArchitecture" : "64-bit", "tenancy" : "Host", "operatingSystem" : "Windows", "licenseModel" : "No License required", "usagetype" : "APS3-HostBoxUsage:d2.4xlarge", "operation" : "RunInstances:0002", "ecu" : "56", "enhancedNetworkingSupported" : "Yes", "normalizationSizeFactor" : "32", "preInstalledSw" : "NA", "processorFeatures" : "Intel AVX; Intel AVX2; Intel Turbo", "servicename" : "Amazon Elastic Compute Cloud" } }, "G2N9F3PVUVK8ZTGP" : { "sku" : "G2N9F3PVUVK8ZTGP", "productFamily" : "Compute Instance", "attributes" : { "servicecode" : "AmazonEC2", "location" : "Asia Pacific (Seoul)", "locationType" : "AWS Region", "instanceType" : "i2.xlarge", "currentGeneration" : "No", "instanceFamily" : "Storage optimized", "vcpu" : "4", "physicalProcessor" : "Intel Xeon E5-2670 v2 (Ivy Bridge)", "clockSpeed" : "2.5 GHz", "memory" : "30.5 GiB", "storage" : "1 x 800 SSD", "networkPerformance" : "Moderate", "processorArchitecture" : "64-bit", "tenancy" : "Host", "operatingSystem" : "Windows", "licenseModel" : "No License required", "usagetype" : "APN2-HostBoxUsage:i2.xlarge", "operation" : "RunInstances:0102", "ecu" : "14", "enhancedNetworkingSupported" : "Yes", "normalizationSizeFactor" : "8", "preInstalledSw" : "SQL Ent", "processorFeatures" : "Intel AVX; Intel Turbo", "servicename" : "Amazon Elastic Compute Cloud" } .... I added a handler to responsehandlers.py as: #handler for AWS pricing API call, split and print out all product stanzas class AWSPriceHandler: def __init__(self,**args): pass def __call__(self, response_object,raw_response_output,response_type,req_args,endpoint): if response_type == "json": output = json.loads(raw_response_output) for product in output["products"]: print_xml_stream(json.dumps(product)) else: print_xml_stream(raw_response_output) But that isn't working for me. It doesn't ingest any events at all when referring to that handler. Take the handler out, and the ingest is a single event that gets truncated at the line length limit. Any suggestions?

pgreer_splunk · ‎07-19-2017

Yes, custom apps can contain Java code, but to be vetted for Cloud installation the java source code needs to be available for inspection. More on Cloud App coding/vetting requirements are found here There is a JRE - not certain at the moment as to what JRE version is within the latest Cloud version (6.5.x) that is available...

pgreer_splunk · ‎02-15-2017

Integration with SAML IdP's is vastly improved in 6.5. One has to seldom modify the authentication.conf file directly if the integration is performed correctly for the IdP through the SAML configuration UI. There are many more IdPs that we've integrated with since 6.5. In 6.4 we only "supported" Okta, Ping, ADFS and Azure. There are blogs for each of those four that step you through how to perform the integration, specific to each IdP, in 6.4. Such as ADFS integration here Okta integration here Ping integration here and Azure integration here . Most likely one of those would get you what you need for whichever IdP you're trying to integrate with now. NOTE: In addition to those four, we have customers (in Splunk Cloud) that have integrated with Google, OneLogin, IBM Tivoli Identity Manager, SecureAuth, CA Siteminder and many others. And most likely many of those and others by on-prem customers. There is a team within our support organization that is well schooled on SAML integrations and are ready and willing to help you with your setup. I suggest you open a support ticket with Splunk, note the SAML IdP you're integrating with, and you'll get all the support you need. If it's an IdP we've not integrated with in the past, the support team will work with you to get it up and running so we'll have the internal knowledge to know how to get it working for the next customer.

pgreer_splunk · ‎02-02-2017

I'm not fully understanding your question - however, what can be done is to simply pass such data within your search results (which is passed into the python script within the JSON payload). Thus anything that can be calculated and captured within a field in your search can be parsed out of the JSON payload and used within your python script. For instance, for a customer e-mail notification alert as an example, you can have the search populate some fields named 'replyTo', 'recipient', 'subject', 'numberOfEvents' - then within the python script parse the JSON payload for the those specific fields and perform actions upon them.

pgreer_splunk · ‎02-02-2017

done! Glad it worked out for you.

pgreer_splunk · ‎02-02-2017

That seems to have done it. So if reading it correctly, the regex says "store into "myField1" one or more characters (in the string) that are NOT whitespace and NOT ':'. Which gobbles up anything until the next match between the groupings, which is zero or more of ':', then store into "myField2" zero or more non-whitespace characters". That seems to have done it for me. Thanks!

pgreer_splunk · ‎02-02-2017

Try playing around with the charting.chart.columnSpacing setting for the bar chart. For example, I opened up the source for a bar chart with overlay that I had set up. By adding the 'charting.chart.columnSpacing' option and setting it really high (100 in my case) it crunched down the bards to nearly a single pixel depending on how wide I have my browser window scaled within my desktop's resolution. .... <option name="charting.legend.labelStyle.overflowMode">ellipsisMiddle</option> <option name="charting.legend.placement">right</option> <option name="charting.chart.columnSpacing">100</option> </chart> .... A similar question was asked on answers as well, here

pgreer_splunk · ‎02-01-2017

I have a field that has a pattern where there is a first portion of the string that I'd like to capture into one field, then an optional second field that follows a ':' For example: field1 field1:field2 field1 ... I've tried (?<myfield1>.*):(?<myfield2>.*) but that only matches if the original field has the "field1:field2". How should I set up the regex so that it always captures the 'field1' into 'myfield1' and will also capture 'field2' into 'myfield2' if it exists after a ':'?

pgreer_splunk · ‎01-26-2017

After posting this (and editing it for clarity) I believe I've found the culprit, the macro expands to have the characters '<' and '>' which do not work all that well in XML and should instead be coded as < > So the question is: if the macro has characters such as '>' and '<' - they work outside of the join (such as in the base search itself, or in a subsearch of a search command) - but they do NOT work within the join.... Is there a way to have the macro expand in a manner that it will be properly encoded within the XML within the join command?

pgreer_splunk · ‎01-26-2017

I've come to find out that one cannot use macros within join statements in dashboards have base searches (driving multiple/all panels in the dashboard). For example, the following code doesn't work: <dashboard> <label>mylabel</label> <search id="base"> <query>index=some_index sourcetype="mysourcetype" earliest=-30d@d | `mymacro` | search [search index=customer_index | `mymacro` | table customer | dedup customer] | stats count by field1 field2 field3 | lookup mylookup customer OUTPUT customer_name as "Customer" | join customer [ search index=some_index earliest=-30d@d sourcetype="mysourcetype" | **`mymacro`** | rex "(?<core>\w*)\s*(?<version>(\d|\.)*)\s*(?<build>.*)" | fields customer version] </query> <earliest>-30d@d</earliest> <latest>now</latest> </search> <row> <panel> <title>Total Customers</title> <single> <search base="base"> <query>| table customer | dedup customer | stats count</query> </search> </single> </panel> </row> </dashboard> but if I take the macro out of the join statement, it will work: <dashboard> <label>mylabel</label> <search id="base"> <query>index=some_index sourcetype="mysourcetype" earliest=-30d@d | `mymacro` | search [search index=customer_index | `mymacro` | table customer | dedup customer] | stats count by field1 field2 field3 | lookup mylookup customer OUTPUT customer_name as "Customer" | join customer [ search index=some_index earliest=-30d@d sourcetype="mysourcetype" | rex field=host "(.*)\.(?<StackId>[^\.]+).splunkcloud.com" | eval customer = StackId | rex "(?<core>\w*)\s*(?<version>(\d|\.)*)\s*(?<build>.*)" | fields customer version] </query> <earliest>-30d@d</earliest> <latest>now</latest> </search> <row> <panel> <title>Total Customers</title> <single> <search base="base"> <query>| table customer | dedup customer | stats count</query> </search> </single> </panel> </row> </dashboard> where the macro mymacro expands to: rex field=host "(.*)\.(?<StackId>[^\.]+).splunkcloud.com" | eval customer = StackId Has anyone seen this before? Occurs in 6.4.x and 6.5.x - and if so is there a way around it?

pgreer_splunk · ‎01-25-2017

As @lguinn stated, you need data in Splunk (indexes) first before you can search on/across it. For Windows filesystem related data, maybe this will help? https://docs.splunk.com/Documentation/Splunk/6.5.1/Data/MonitorfilesystemchangesonWindows

pgreer_splunk · ‎01-25-2017

Your regex does match (for your example) just the log message: 12.156.22.149 - - [09/Dec/2016:20:20:44 -0500] \"-\" 408 - \"-\" \"-\" \"-\" as tested at https://regex101.com/ So the regex seems valid. Maybe it's the transform format. From the documentation https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Transformsconf it says in the regex section: If the REGEX extracts both the field name and its corresponding field value, you can use the following special capturing groups if you want to skip specifying the mapping in FORMAT: KEY , VAL . For example, the following are equivalent: Using FORMAT: * REGEX = ([a-z]+)=([a-z]+) * FORMAT = $1::$2 Without using FORMAT * REGEX = (?<_KEY_1>[a-z]+)=(?<_VAL_1>[a-z]+) So maybe the regex should be?: [setparsing] REGEX = "message": "(?<_raw>.*")"

pgreer_splunk · ‎01-25-2017

Yes, but the cloud instance needs to be at 6.5.x.

pgreer_splunk · ‎01-25-2017

Depending on the version you're on it differs a bit. With 6.5.x you can enter Settings->Access Controls->Authentication Method then click on the 'SAML' radio button option, then click on the 'Configure Splunk to use SAML' link that appears. Once in the SAML Groups page, click on the green 'SAML Configuratin' button in the upper right corner. In the sub-window that pops up there is a button to download the metadata. In versions prior to 6.5.x (6.4.x, 6.3.x) go to your search head URL at http(s)://:port/saml/spmetadata NOTE: That URL above will actually work for 6.5.x as well. In previous versions you had to copy/paste the XML from the page to a file, in 6.5.x that URL triggers a download/save of the XML file.

pgreer_splunk · ‎01-12-2017

Splunk Cloud (managed service) is nearing PCI and HIPAA compliance, we're getting close. Keep an eye out on http://www.splunk.com/en_us/products/splunk-cloud.html for more information in the future on compliance. There are many customers that utilize Cloud as well as On-Prem independently and some as well that perform hybrid (searching across on-prem and cloud). For PCI and HIPAA compliance for on-prem, that is up to the customer. The Splunk core software is by itself not something that fits a compliance criteria as there is much more to compliance than just the software package (processes, procedures, reporting, auditing, security, roles and responsibilities, etc. etc. etc.).

pgreer_splunk · ‎01-12-2017

If that trial was created outside of 15 days prior to when you posted your question (12/25/16) then the trial had expired. Cloud trials are good for 15 days. You can convert them to a POC, then later if you purchase, to production. If it was past 15 days, the instance is gone and you'll need to start from scratch again, or look into starting up a POC with your local Splunk or partner representative.

pgreer_splunk · ‎01-12-2017

As @hunters states, in LDAP yes, they can be part of multiple groups and if those multiple groups 'map' to multiple roles in Splunk the user will 'map' to multiple roles. The subject of your question has 'SAML' and the tags include 'SAML' as well as 'PingFederate'. So, if you are asking about SAML, here's a little additional info: In SAML, PingFederate (if configured as appropriate for SAML integration with Splunk on-prem or cloud) will send an XML assertion to Splunk with an attribute named 'role'. This 'role' attribute will contain a list of all LDAP groups that a user's account is within. In the SAML mappings setup in 6.5.0, you will then configure what role (or roles) the LDAP group maps to. If there are more than one 'group->role' mappings for the user's 'role' list, then once the user is authenticated in via SAML they will 'map' to multiple Splunk roles. For more information on configuring Ping for SAML (cloud specific but nearly identical for on-prem), see the blog: Ping SAML Blog

pgreer_splunk · ‎12-15-2016

Was a case created in your customer portal in Splunk? Or were e-mails just sent?

pgreer_splunk · ‎12-15-2016

What Identity Provider are you using?

pgreer_splunk · ‎12-05-2016

With Splunk 6.5.x yes. CA Siteminder (SAML SSO) has been configured (for Splunk Cloud and thus as well will work for on-prem). Siteminder is not "supported" for 6.4.x or earlier. The two factor auth is facilitated on the CA side prior to sending the SAML assertion to the Splunk instance, so the two factor piece is separate from integration with CA Siteminder via SAML.

Posts	97
Solutions	16
Karma Given	4
Karma Received	56
Member Since	‎01-05-2016

Online Status	Offline
Date Last Visited	‎04-23-2024 07:44 PM

Passing a lookup file name as a parameter in a mac...

Email alert to recipient within results set, multi...

Line splitting at a regular expression for ""

REST modular input JSON custom handler for AWS Pri...

Regex to split field, optional second field

Dashboard base search cannot use macros

Can the REST call to delete a user from Splunk be ...

Re: Passing a lookup file name as a parameter in a...

Passing a lookup file name as a parameter in a mac...

Email alert to recipient within results set, multi...

Re: Line splitting at a regular expression for ""

Line splitting at a regular expression for ""

REST modular input JSON custom handler for AWS Pri...

Re: Does Splunk Cloud support apps / ERPs with Jav...

Re: Problem with SSO SAML (Splunk 6.5)

Re: How can I get some additional alert details in...

Re: How to adjust chart column width?

Re: Regex to split field, optional second field

Re: How to adjust chart column width?

Regex to split field, optional second field

Re: Dashboard base search cannot use macros

Dashboard base search cannot use macros

Re: How to search and display all files or folders...

Re: How to index only the data payload from json s...

Re: Does Splunk Cloud support DUO two factor authe...

Re: Splunk with SAML for authentication: How to ob...

Re: What are the security differences between Splu...

Re: Why does Splunk Cloud Instance URL return "Err...

Re: SAML: If a user belongs to several groups in L...

Re: splunk cloud with ldap

Re: How to configure Splunk SAML SSO on Windows?

Re: Does Splunk support CA Siteminder Two Factor A...