Fairly certain you cannot do what you're referring to as it would involve changing the internal workings of how Splunk uses the management port to communicate between other Splunk components. Since the slave is outside the organization, and based on your question in the title of your post, your best option would likely be encrypting that communication. You can secure the communication between the slaves and license master by configuring SSL in server.conf. http://docs.splunk.com/Documentation/Splunk/6.6.0/Security/AboutsecuringyourSplunkconfigurationwithSSL ... View more

Hmm, the Splunk Add on for Windows contains field extractions for Windows-based XML logs in both the props and transforms. Are you using this app? https://splunkbase.splunk.com/app/742/ ... View more

You might explore creating a field alias for those fields under settings > fields > field alias Splunk references field aliases as a first step under "Make your fields CIM-compliant" in the Common Information Model Add-on Manual. There are step by step instructions for various tasks: http://docs.splunk.com/Documentation/CIM/4.8.0/User/UsetheCIMtonormalizedataatsearchtime Also, search time field extractions are recommended over index time: https://answers.splunk.com/answers/5817/search-time-versus-index-time-field-extractions.html https://answers.splunk.com/answers/2535/search-time-vs-index-time-field-extraction.html ... View more

Hello Sundar, There are detailed instructions for creating indexes via Splunk Web, under "Use Splunk Web", located here: https://docs.splunk.com/Documentation/Splunk/6.5.3/Indexer/Setupmultipleindexes ... View more

You might try renderXml=true in your inputs.conf file beneath the stanza for that sourcetype, under the app you're working with. http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf ... View more

You're very welcome! I went ahead and turned my reply into another post so I could insert pictures for you. ... View more

I had to convert this comment into another answer (so I could insert pictures) to address your other questions. I am assuming you have admin rights to your Splunk. 1.) For the first part, if you are presented with a value of spaces between the brackets, you could simply add a | search yourfield!=" " at the end of your search, which filters out any value with a space in it. 2.) Yes, you can turn this into an extracted field without using props.conf and/or transforms.conf by doing it entirely in the GUI... and for the second part (making a multivalue field split) you can create a calculated field. Go to Settings > Fields > Field Extractions > New If you don't have a specific app you're working in, leave that as its default value. Insert a name for your extraction, the sourcetype where the data resides, and insert the regex from the search excluding the double quotes, like this: Save it. When you save it, you'll be taken back to a section where you can search through other field extractions. Search the name of the field extraction you just created. You will need to change the permissions on it for other apps and/or for other people to see/use it. Under the sharing column, click permissions. Where it says "Object should appear in", either change it to "This App" or "Global". If you choose This App, the object will only be available to use in the current app (which is probably the Search app). If you choose Global, it will become a global object available in all apps. Use your discretion as to which roles you'd like to assign read or write permissions to. I usually select read for everyone, and admin for write, but I don't know what kind of environment you're working in. Once you save it, now you'll need to go to Settings > Fields > Calculated fields Click New, follow the similar format to the field extractions, and fill in the values. This time, you're going to insert the eval expression split(yourfield,",") Click save, make sure you change your permissions of the calculated field to the same permissions you gave your field extraction. From here, to apply your extractions and make them visible for use, you can either: Restart your search head or type into the search bar |extract reload=true and then in your address bar of your browser do a debug refresh SearchHeadURL:8000/en-US/debug/refresh and press refresh button - wait for it to complete. I usually do both because for whatever reason one or the other doesn't always work by itself. What this does is force a refresh on splunkd resources. Now, you should be able go to where your data resides and simply type in index=yourIndex sourcetype=yourSourcetype and see yourfield under "interesting fields" Lastly, as with this topic and any other question, make sure to accept the answer(s) so others know the question was answered 🙂 Hope that helps! ... View more

EDIT: I just noticed your log from your original post has a space between the closed parenthesis and the open bracket, but the log you posted in your comment does not. If your log does NOT have a space between the ) and [ try this: | rex field=_raw "\)\[(?P<yourfield>.[^\]]*)" If your log DOES have one or more spaces between the ) and [ try this: | rex field=_raw "\)\s+\[(?P<yourfield>.[^\]]*)" If your logs switch between having and not having spaces between the ) and [ try this: | rex field=_raw "(\)\s+|\))\[(?P<yourfield>.[^\]]*)" ... View more

Try this: | rex field=_raw "\[(?P<yourfield>.[^\]]*)" | makemv delim="," yourfield Worked for me: ... View more

As in the contents of the PPT or PDF? No, not without some type of intermediary. Splunk docs might help you with what Splunk can monitor and index: http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/WhatSplunkcanmonitor http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Monitorfilesanddirectories ... View more

Oh well, next best thing is to investigate what you can disable and if you're still getting more noise, I'd look into setting up a nullqueue in your props.conf and tranforms.conf. Happy to help! ... View more

So, if I understand correctly, you have a centralized log server collecting audit logs from other servers aggregating in a single file, and you'd like Splunk to identify the host name of each device? If the audit logs coming from the other servers have a host name in each of the raw events, you can specify the host_regex in the inputs.conf file for that index or sourcetype where Splunk will extract the hostname from in the raw event. Another option, since you mentioned these audit logs are from other servers, you could consider bringing some structure to your syslogs and establish syslog rules to separate each of those server logs into their own file or directory for monitoring. I touched a little about organizing log sources in syslog and monitoring them in Splunk in another post in case you're interested: https://answers.splunk.com/answers/504420/forward-syslogs-with-correct-sourcetypes.html#answer-504450 ... View more

Alright... I THINK I understand what you're trying to do. Maybe. I think you're trying to join two searches based on a common field. If that's the case, try something like this: YourSearch Workflow="*tsr*" OR "go_choix_1*" | eval ACResponse=mvjoin(ACResponse,";") | search ACResponse!="*200*" | join OCId type=outer [ AnotherSearch Workflow="debordement_*" | stats count by OCId] | stats list(ACResponse) AS ACResponse, dc(ACResponse) AS ACResponse_dcount by OCId You're doing a few different things with the stats functions in your searches, so I'm not sure exactly what your expected output is. You might need to explain a little more. Otherwise, I hope that helps. ... View more

If you're collecting too much data, I'd explore maybe tuning down the noise at its source (ie, the boxes themselves) instead of trying to filter everything out with Splunk. In the case of using the vmware oriented Splunk apps, it appears those apps collect a lot of logs and do a lot of performance and monitoring oriented functions (should you choose to enable them). It sounds like many of these extra monitoring capabilities are enabled. I'm not sure what you're situation is as far as who set this up for you, but you can always go into each of the inputs.conf files in the apps and change the disabled = 0 to disabled = 1 , which will disable the monitoring stanzas for that particular log source. Since it appears you're only interested in collecting shell.log, auth.log, hostd.log, you may explore just adding your own monitoring stanzas for those log locations and disable everything else, like this: [monitor:///var/log/shell.log] disabled = 0 [monitor:///var/log/auth.log] disabled = 0 [monitor:///var/log/hostd.log] disabled = 0 ... View more

Have you looked at this post? https://answers.splunk.com/answers/7899/splunkweb-fails-to-start-timeout-when-binding-to-port.html ... View more

Do you mean like this? your base search | stats count by dest_port, protoid ... View more

A few things I noticed from your picture. I'm purely speculating because I don't know what those underlying searches look like. Did you check the date and time of that event in comparison to the time you were searching for it? Did you check whether the TZ and/or timestamp settings are correct for both the host you used the Eicar test file on, and the indexer? I looked at the time chart dates and times in your picture, and compared that to the time you posted. It looks as if that event might have a time stamp in the future. Not sure what time zone you're in, but I've run into an issue before where an appliance had the incorrect system time set for events, and events were showing up in Splunk in the future. I also noticed something similar with my case that you also did; the events show up in a time chart, but not in a single value count or any adhoc search that specified a particular time range that wasn't all time. If that isn't the case, I could take a look at the searches for you? ... View more

For the extractions you wanted from your original post, these should work: | rex field=_raw "\"cvpEditAction\"\s\:\s\"(?P<cvpEditAction>.[^\"]*)" | rex field=_raw "\"cvpEditAllowedAmount\"\s\:\s(?P<cvpEditAllowedAmount>.[^\,]*)" As far as helping you understand how to build these, and for future instances: I'd do some reading on regular expressions and how to use them in Splunk. Learning regular expressions is actually easier than it looks. I started with this site: https://regexone.com/. Then, I browsed some examples here on Splunk Answers and practiced on my own. Might help you too. Looking over the log sample you provided, it appears you can use similar rex patterns for the other fields you want. It looks as if each field name (the key) has a double quote in the begging and end, and the field value (the key value) either has a double quote or a comma at the end of it. With that in mind, you can use the patterns I provided to interchange and extract other fields out of similar logs. There are several other ways to extract the fields, this is just the way I chose to do it because it is the way I learned. I'm not a regex master... yet. Take note of the end part for each regex I provided to you. | rex field=_raw "\"cvpEditAction\"\s\:\s\"(?P<cvpEditAction>.[^\"]*)" .[^\"]*) basically says, extract everything up to a double quote. and | rex field=_raw "\"cvpEditAllowedAmount\"\s\:\s(?P<cvpEditAllowedAmount>.[^\,]*)" .[^\,]*) basically says extract everything up to the comma. For example, let’s say you want to extract the value of "cvpEditAddedLineCount" in your log sample. Notice the value of "cvpEditAddedLineCount" is 0, . A comma appears at the end of your value. So you want to extract everything AFTER "cvpEditAddedLineCount" : but BEFORE the comma. Modify the regex I provided for excluding the comma, like this: | rex field=_raw "\"cvpEditAddedLineCount\"\s\:\s(?P<cvpEditAddedLineCount>.[^\,]*)" Notice the only thing I changed was the text for the field name (the key). From cvpEditAllowedAmount to cvpEditAddedLineCount . Everything else stayed the same. From | rex field=_raw "\"cvpEditAllowedAmount\"\s\:\s(?P<cvpEditAllowedAmount>.[^\,]*)" to | rex field=_raw "\"cvpEditAddedLineCount\"\s\:\s(?P<cvpEditAddedLineCount>.[^\,]*)" Likewise, Say you want to extract the value of "cvpEditClaimNumber" from your sample log. Notice the value, "OK1-0201705480170030C-00" has double quotes wrapped around it. Modify the regex with the quote exclusion, from: | rex field=_raw "\"cvpEditClaimNumber\"\s\:\s\"(?P<cvpEditClaimNumber>.[^\"]*)" to | rex field=_raw "\"cvpEditClaimNumber\"\s\:\s\"(?P<cvpEditClaimNumber>.[^\"]*)" If I’m not making any sense, let me know so I can clarify further. Now, I’m assuming at some point you might want to extract the other fields from that log. Cluttering up your search with 30+ rex commands might get a bit cumbersome. There are a few ways to have Splunk extract fields without specifying them directly in the search. 1.) Via the GUI under Settings > Fields > Field Extractions 2.) Via props.conf or props.conf AND transforms.conf Via the GUI under Settings > Fields > Field Extractions, I used one of your fields as an example If you intend on sharing this field extraction with other sourcetypes or with other people, you’ll need to edit the permissions of the field extraction after you save it. Another way you can have Splunk extract fields for you is by directly editing props.conf of your app on your search head. In this case, I'm using the search app, which, for me is located at /opt/splunk/etc/apps/search/local. To extract the "cvpEditAction", you can add a stanza in the props.conf file that looks something like this: [yourSourcetype] EXTRACT-cvpEditAction = \"cvpEditAction\"\s\:\s\"(?P<cvpEditAction>.[^\"]*) You can also use a combination of props.conf and transforms.conf, but for simple field extractions like this, I prefer keeping them consolidated under props.conf. You don't necessary have to restart your search head after adding a field extraction either (although you can if you want). Simply type |extract reload=true into your search bar. You can read more about extracting fields here: http://docs.splunk.com/Documentation/Splunk/6.5.2/Knowledge/Aboutfields http://docs.splunk.com/Documentation/Splunk/latest/Search/Extractfieldswithsearchcommands https://docs.splunk.com/Documentation/Splunk/6.5.2/Scenarios/Extractfields and about props.conf and transforms.conf here: https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Propsconf http://docs.splunk.com/Documentation/Splunk/latest/admin/transformsconf and of course, practice makes perfect! https://regex101.com/ http://regexr.com/ Hope that helps. ... View more

Can you post the original raw log and your expected output after extraction? ... View more

From time to time I run into an issue extracting fields from _raw with Windows logs containing new lines, but sometimes what I'm looking for is in the Message field bunched together too. If that's the case for you. Give this a try: | rex field=Message "(IP\:\s+|\d+\,)(?P<ClientIP>.[^\s]*)" | makemv delim="," ClientIP This should extract the IPs and separate them into their own values. If you don't want the IPs as their own values, remove the makemv portion of it. Also, someone posted a log the other day from an ADFS log where the Client IP had a . at the end of the second IP. If that appears in your events, give this a try: | rex field=Message "(IP\:\s+|\d+\,)(?P<ClientIP>.*)." | makemv delim="," ClientIP For the username, give one of these a try: | rex field=Message "message\:\s(?P<username>.*)-" or | rex field=Message "message\:\s(?P<username>.[\s]*)" or | rex field=Message "message\:\s(?P<username>.[\-]*)" If none of that works, would you mind providing a sanitized sample of the entire log? ... View more

I recently went through the Architecting and Deploying Splunk course and the instructor touched on this subject. There are multiple correct answers and multiple wrong answers, but it really depends on your environment, your preferences, use case, and the volume of data per index. If the reasoning surrounds data retention and you aren’t worried about things like correlation or complexity with multiple index/sourcetype search queries, then go for it. More indexes might help with search performance in some circumstances. When you divide similar data types into different indexes, you will need to specify each index before you search to see a positive difference in search performance. Specifying which index to look at during search time should help with search performance. If you create an index with the thought of organization and optimal search performance, in my opinion, I'd stick with separating the each of those log sources into individual sourcetypes. For example, I'm using a syslog server in my environment to forward logs from various appliances in my infrastructure. I created rules in my syslog.conf file to organize the data into individual folder structures, and a universal forwarder sits on the server to monitor each directory and forward the events to their respective sourcetype while keeping the index of syslog. My data is organized in a way that makes sense to me (and really, that's what matters in your case since you are the one managing it). I can simply specify sourcetype=cisco or sourcetype=juniper to search different logs instead of specifying a different index. This also helps me when I'm correlating data. There's really no drastic performance benefit to either technique from what I've experienced so far, but then again, I haven't experimented with creating a massive repository of indexes. I touched on the syslog stuff here: https://answers.splunk.com/answers/504420/forward-syslogs-with-correct-sourcetypes.html#answer-504450 On the other hand, I do have some special cases where I forward data from the syslog appliance to its own index due to the volume of data generated from the appliance. In my case, my proxy logs are massive, so I keep those separate from my syslog index. Again, I think it really comes down to preference and use case. Every index you create means that Splunk has to look in more buckets for the tsidx files relating to your search terms. If you need to find a search term over a 30 day period in all indexes, the more indexes you have means Splunk must look in each one of those indexes, and the speed of the returned results of your search is commensurate on the hardware specs of your server (especially disk type), how many indexes you have, the volume of those indexes, the complexity of your search, and the time range you’re searching. If you intend on searching multiple indexes with large volumes over long periods of time, creating multiple indexes might create additional latency during search time. On the other hand, if you’re searching specific indexes with smaller volumes over smaller periods of time, your searches should be much faster than if you were looking in one index with multiple sourcetypes. There is also the added element of difficulty correlating searches. In your case, you want to create indexes for nginx, celery, supervisord, nodejs, custom application, etc. If for any reason you need to correlate results from those indexes, you might need to create a search query containing several boolean operators, multiple subsearches, transactions, joins and/or appends. This will impact search performance and may cause difficulty getting accurate, fast, meaningful results. There will also probably be an issue if you have a team of people looking at dashboards containing multiple panels with complex search queries like that. Bear in mind that each search ties up a core of the processor and will not release that core until the search finishes. Then again, if you have no use case for making searches and correlating each of the indexes/sourcetypes, then I wouldn’t worry too much about it. I also encountered a customer who did something similar by creating several indexes. They had around 150 indexes and created over 50 dashboards with 5 or more panels per dashboard consisting of tables and visualizations. While the panels were all meaningful representations of their data, it overwhelmed analysts and admins very quickly, created confusion, searches and panels executed slowly, and they often missed important events occurring in their infrastructure. Just some food for thought. ... View more

This documentation might help answer some of the questions you have regarding monitoring files and directories: http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Monitorfilesanddirectorieswithinputs.conf If your logs are generated from a script, try reading about scripted inputs: http://docs.splunk.com/Documentation/Splunk/6.5.2/AdvancedDev/ScriptSetup I think in your case, you might take a look at Example 2 from the documentation pertaining to monitoring files and directories: To load anything in /apache/ that ends in .log. [monitor:///apache/*.log] or for Windows [monitor://C:\path\to\your\stuff\*.log] Splunk will monitor any new additions or changes to the files you set up on a monitoring stanza on. The retention of your data depends on a number of factors. Most of the time, this depends on how much storage you have and Splunk will automatically remove data from indexes should storage be an issue. You can set a retirement date for your data by either size or age (time), and even do this per index if you care more about the retention of specific data. This document explains a bit about how Splunk data "ages" and "rolls" to different buckets depending on your settings: https://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Setaretirementandarchivingpolicy If you want to determine how long your data is hanging around, you could try this: | dbinspect [ eventcount summarize="false" index=* | dedup index | fields index] | stats min(startEpoch) AS startEpoch,min(modTime) AS modTime by index,splunk_server | convert ctime(startEpoch) AS startEpoch | rename modTime AS "Oldest Bucket",startEpoch AS "Earliest Event Time" ... View more

Try this: index=Application sourcetype=Servers | search [|inputlookup yourLookup | fields + host | table host] | stats count by host ... View more

Splunk supports a variety of languages. Try taking a look at some of the documentation pertaining to scripted inputs, modular inputs, and sdks. http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ScriptedInputsIntro http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/Getdatafromscriptedinputs http://docs.splunk.com/Documentation/Splunk/6.5.2/AdvancedDev/ModInputsIntro http://dev.splunk.com/sdks ... View more