You are missing the new line in the split command as shown in my suggestion - try using the command exactly as I suggested ... View more

| rex field=source "info_(?<timestamp>\d{8}_\d{6})\.csv" | eval _time=strptime(timestamp,"%Y%m%d_%H%M%S") ... View more

Change your global time zone to be your local time zone e.g. EST. To calculate differences in times you need to parse the strings to epoch format | eval epoch_timestamp=strptime(c_timestamp,"%FT%T.%6N%z") | eval local_timestamp=strftime(epoch_timestamp,"%F %T.%6N %Z") | eval epoch_mod=strptime(c_mod,"%FT%T.%6N%z") | eval local_mod=strftime(epoch_mod,"%F %T.%6N %Z") | eval diff=epoch_mod-epoch_timestamp ... View more

  | rex "(?<head1>[^,]*),(?<head2>[^,]*),(?<head3>[^,]*),(?<head4>[^,]*),(?<head5>[^,]*),(?<head6>[^,]*),(?<head7>[^,]*),(?<head8>[^,]*),(?<head9>[^,]*),(?<head10>[^,]*),(?<head11>[^,]*),(?<head12>[^,]*)" The fields will be null so you could use fillnull to give them values e.g. | fillnull value="N/A"   ... View more

@ravir_jbp , Did you try fillnull https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Fillnull OR  replace in case its a literal value NULL https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Replace    ... View more

only data with cl1 is getting replaced. I also have data with cl3 which needs to be replaced by ACD85. There is no possibility @ITWhisperer's search should give this half of replacement.  But first, your search is very inefficient: The third line starting with search should be accomplished in the first line so fewer events are computed.  Secondly, using regex on rigidly formatted data (CSV) is a waste and prone to errors.  This is what I suggest, using exactly what @ITWhisperer proposed.   index=csv sourcetype="miscprocess:csv" source="D:\\automation\\miscprocess\\output_acd.csv" ("\cl3\" OR "\cl1\") | eval filename = split(_raw, ",") | eval filesize = mvindex(filename, 1), filelocation = mvindex(filename, 2) | eval filename = mvindex(filename, 0) | eval filelocation=if(like(filelocation,"%\cl1%"),"ACD55","ACD85")   Also important: Play with the following emulation and compare with your real data:   | makeresults | fields - _* | eval data=split("012624.1230,13253.10546875,E:\totalview\ftp\acd\cl1\backup_modified\012624.1230 012624.1230,2236.3291015625,E:\totalview\ftp\acd\cl3\backup\012624.1230 012624.1200,13338.828125,E:\totalview\ftp\acd\cl1\backup_modified\012624.1200 012624.1200,2172.1640625,E:\totalview\ftp\acd\cl3\backup\012624.1200 012624.1130,13292.32421875,E:\totalview\ftp\acd\cl1\backup_modified\012624.1130 012624.1130,2231.9658203125,E:\totalview\ftp\acd\cl3\backup\012624.1130 012624.1100,13438.65234375,E:\totalview\ftp\acd\cl1\backup_modified\012624.1100", " ") | mvexpand data | rename data AS _raw | search (\\cl1\\ OR \\cl3\\) ``` the above emulates index=csv sourcetype="miscprocess:csv" source="D:\\automation\\miscprocess\\output_acd.csv" ("\cl3\" OR "\cl1\") ``` | eval filename = split(_raw, ",") | eval filesize = mvindex(filename, 1), filelocation = mvindex(filename, 2) | eval filename = mvindex(filename, 0) | eval filelocation=if(like(filelocation,"%\cl1%"),"ACD55","ACD85")   The output is _raw filelocation filename filesize 012624.1230,13253.10546875,E:\totalview\ftp\acd\cl1\backup_modified\012624.1230 ACD55 012624.1230 13253.10546875 012624.1230,2236.3291015625,E:\totalview\ftp\acd\cl3\backup\012624.1230 ACD85 012624.1230 2236.3291015625 012624.1200,13338.828125,E:\totalview\ftp\acd\cl1\backup_modified\012624.1200 ACD55 012624.1200 13338.828125 012624.1200,2172.1640625,E:\totalview\ftp\acd\cl3\backup\012624.1200 ACD85 012624.1200 2172.1640625 012624.1130,13292.32421875,E:\totalview\ftp\acd\cl1\backup_modified\012624.1130 ACD55 012624.1130 13292.32421875 012624.1130,2231.9658203125,E:\totalview\ftp\acd\cl3\backup\012624.1130 ACD85 012624.1130 2231.9658203125 012624.1100,13438.65234375,E:\totalview\ftp\acd\cl1\backup_modified\012624.1100 ACD55 012624.1100 13438.65234375 As you see, there is no such "partial replacement".  You will need to illustrate and explain any discrepancy between real data and this mock data if you don't get the same results. ... View more

If you actually have tabs separating your fields (instead of commas), the issue is that you have used + (at least 1 occurrence) rather than * (zero or more occurrences) ^(?P<ACD>\w+\.\d+)\t(?P<ATTEMPTS>[^\t]+)\t(?P<FAIL_REASON>[^\t]*)\t(?P<INTERVAL_FILE>[^\t]+)\t(?P<STATUS>\w+)\t(?P<START>[^\t]+)\t(?P<FINISH>[^\t]+)\t(?P<INGEST_TIME>.+) ... View more

| eval curActStart=strftime(strptime(curActStart,"%F %T.%S")+(60*60*5)+(60*30),"%F %T.%S") ... View more

Hi @ravir_jbp ... for the data already logged into splunk, do you want to use Splunk Search query and get some results? (and maybe do you want to create dashboard/alert/report)  or do you want to onboard/ingest some csv files, but the field extraction not working as expected, please suggest, thanks.  ... View more

| rename 'old field name' as "new field name" or, change the field values before the timechart | eval connectionType=case(connectionType=="old field value 1","new field value 1", connectionType=="old field value 2", "new field value 2", true(), connectionType) ... View more

As I noted in https://community.splunk.com/t5/Splunk-Search/Date-time-formatting-variables-not-producing-result-I-expected/m-p/666477#M228639, the letter "Z" signifies a standard time zone and you should NOT simply remove it.  Instead, Splunk should process it as a timezone token before you render the end result in any string format you wanted.  In other words, | eval stime=strftime(strptime(stime,"%FT%T%Z"),"%F %T") | eval etime=strftime(strptime(etime,"%FT%T%Z"),"%F %T") | eval orgstime=strftime(strptime(orgstime,"%FT%T%Z"),"%F %T") | eval orgetime=strftime(strptime(orgetime,"%FT%T%Z"),"%F %T")   ... View more

I would be cautious to anchor regex as closely as the data is regular.  Something like   | rex field=source "\\\t4\\\(apch\\\node|logs)\\\(?<node>[^-\\\\]+)"   This should give node source node06 E:\view\int\t4\apch\node\node06\log\server.log node06 E:\view\int\t4\apch\node\node06\log\run.log node03 E:\view\int\t4\apch\node\node03\log\server.log node01 E:\view\int\t4\apch\node\node01\log\server.log node01 E:\view\int\t4\apch\node\node01\log\run.log core02 E:\view\int\t4\logs\core02-core.log web37 E:\view\int\t4\logs\web37-wfmws.log core01 E:\view\int\t4\logs\core01-core.log You can play with the emulation @ITWhisperer offered and compare with real data.   | makeresults format=csv data="source E:\view\int\t4\apch\node\node06\log\server.log E:\view\int\t4\apch\node\node06\log\run.log E:\view\int\t4\apch\node\node03\log\server.log E:\view\int\t4\apch\node\node01\log\server.log E:\view\int\t4\apch\node\node01\log\run.log E:\view\int\t4\logs\core02-core.log E:\view\int\t4\logs\web37-wfmws.log E:\view\int\t4\logs\core01-core.log" ``` data emulation above ```       ... View more

@ITWhisperer  IT worked !!! thank you so much  ... View more

Hi @ravir_jbp, sorry but there's some misunderstandings: at first I spoke about "sourcetype", not "source", they are two different things: sourcetype is usually defined in the input stanza where you run the Powershell script and it's the way that Splunk uses to define specifications and parsing rules of a Data Flow, for this reason, you should set the sourcetype in the input.conf stanza containing the run of the script to exactly define the specifications of your data. Then, in props.conf you have to add the option "INDEXED_EXTRACTIONS = csv" that it isn't possible to use in the search (that you defined "query") and that permits to you to automatically extract all the fields. As I said, you should search in the YouTube Splunk Channel some video that describes how to ingest and parse csv data, these are some examples: https://www.youtube.com/watch?v=fKoAB6n_ivs&pp=ygUKc3BsdW5rIGNzdg%3D%3D https://www.youtube.com/watch?v=3kx0OGKy_XU&pp=ygUKc3BsdW5rIGNzdg%3D%3D https://www.youtube.com/watch?v=SfEDJj7Jgpg&pp=ugMICgJpdBABGAHKBQpzcGx1bmsgY3N2 but you can find more others. About sourcetype configuration, you have to create the props.conf file containing all the information to correctly parse you Data Flow, starting from Timestamp definition, Timestamp format and "INDEXED_EXTRACTIONS = csv". Ciao. Giuseppe  ... View more

| regex Employee_Id=="^[A-Za-z0-9\.\%\,\@\*\&\^\$\#\@\!\)\(]+$" | regex Employee_Id!="^\d{9}$" ... View more

| rex field=source "/(?<user>[^_/]+)[^/]+\.log$" ... View more

If there is a possibility that the field does not exist in the event, the extraction should be split across multiple commands: | rex "Endpoint '(?<endpoint>\w+)' received message ''(?<received_message>[^']+)'" | rex "AttributeThisDN[^\"]+\"(?<AttributeThisDN>[^\"]+)\"" | rex "AttributeAgentID[^\"]+\"(?<AttributeAgentID>[^\"]+)\"" | rex "AttributeAgentWorkMode[^\[]+\[[^\[]+\[(?<AttributeAgentWorkMode>[^\]]+)\]" | rex "ReasonCode[^\"]+\"(?<ReasonCode>[^\"]+)\"" ... View more

Hi @ravir_jbp, sorry! try this https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Spath Ciao. Giuseppe ... View more

Hi @ravir_jbp, good for you, see next time! Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated 😉 ... View more

Create a couple of reports which look at events in the previous 10  minutes (or 5 minutes if appropriate), one which searches for the first string and the other searches for the second string. If you get no results, trigger the alert with the send email action. ... View more

Why use a regular expression when Splunk has a command specifically for this data format?  Check out the multikv command at https://docs.splunk.com/Documentation/Splunk/8.2.4/SearchReference/Multikv ... View more

  \[[^\]]+\]\s+\w+\s(?<service>[^\/]+)\/((\w+\-)*?|)(?<hostname>\w+)(\-(?<port>\d+)|)?\s(\d+\/){4}(?<response>\d+)\s+   ... View more

Your example doesn't have the outer JSON field, so I added that in order to extract the array elements as a multivaliue field which could expanded (mvexpand) - you may be able to extract this more easily from your real data. Your epoch times are in milliseconds not seconds as normal, so I converted those before calculating the duration and formatting for readability | makeresults | eval _raw="[{\"hostname\":\"BBBBBBBBB\",\"contentSourceName\":\"Authored\",\"contentSourceType\":\"Authored\",\"incremental\":true,\"skipCrawl\":false,\"isBulk\":false,\"startTime\":1616802303335,\"endTime\":1616802355772,\"reportStatus\":\"Success\",\"documentsFound\":20,\"documentsFailed\":0,\"documentsSucceeded\":16,\"documentsFiltered\":0,\"documentsUnchanged\":0,\"contentProcessed\":16,\"contentAdded\":0,\"contentUpdated\":0,\"contentDeleted\":0,\"pdfSlices\":0,\"pdfDocCount\":0,\"exceptionCount\":0,\"generalExceptionCount\":0,\"warningCount\":0,\"processorFailureCount\":0,\"generalizedFailureCount\":0,\"heritrixErrorCount\":0,\"duplicateItemCount\":0,\"duplicateReportRelativeFilename\":null,\"jobId\":-1}, {\"hostname\":\"AAAAAAAA\",\"contentSourceName\":\"Authored\",\"contentSourceType\":\"Authored\",\"incremental\":true,\"skipCrawl\":false,\"isBulk\":false,\"startTime\":1616801520297,\"endTime\":1616801578765,\"reportStatus\":\"Success\",\"documentsFound\":40,\"documentsFailed\":0,\"documentsSucceeded\":28,\"documentsFiltered\":0,\"documentsUnchanged\":0,\"contentProcessed\":28,\"contentAdded\":0,\"contentUpdated\":0,\"contentDeleted\":0,\"pdfSlices\":0,\"pdfDocCount\":0,\"exceptionCount\":0,\"generalExceptionCount\":0,\"warningCount\":0,\"processorFailureCount\":0,\"generalizedFailureCount\":0,\"heritrixErrorCount\":0,\"duplicateItemCount\":0,\"duplicateReportRelativeFilename\":null,\"jobId\":-1}, {\"hostname\":\"ZZZZZZZZZ\",\"contentSourceName\":\"Authored\",\"contentSourceType\":\"Authored\",\"incremental\":true,\"skipCrawl\":false,\"isBulk\":false,\"startTime\":1616797920257,\"endTime\":1616797999256,\"reportStatus\":\"Success\",\"documentsFound\":104,\"documentsFailed\":0,\"documentsSucceeded\":59,\"documentsFiltered\":0,\"documentsUnchanged\":0,\"contentProcessed\":59,\"contentAdded\":0,\"contentUpdated\":0,\"contentDeleted\":0,\"pdfSlices\":0,\"pdfDocCount\":0,\"exceptionCount\":0,\"generalExceptionCount\":0,\"warningCount\":0,\"processorFailureCount\":0,\"generalizedFailureCount\":0,\"heritrixErrorCount\":0,\"duplicateItemCount\":0,\"duplicateReportRelativeFilename\":null,\"jobId\":-1}, {\"hostname\":\"YYYYYYYY\",\"contentSourceName\":\"Authored\",\"contentSourceType\":\"Authored\",\"incremental\":true,\"skipCrawl\":false,\"isBulk\":false,\"startTime\":1616794883261,\"endTime\":1616795120383,\"reportStatus\":\"Success\",\"documentsFound\":236,\"documentsFailed\":3,\"documentsSucceeded\":121,\"documentsFiltered\":0,\"documentsUnchanged\":0,\"contentProcessed\":121,\"contentAdded\":0,\"contentUpdated\":0,\"contentDeleted\":0,\"pdfSlices\":0,\"pdfDocCount\":0,\"exceptionCount\":0,\"generalExceptionCount\":0,\"warningCount\":0,\"processorFailureCount\":3,\"generalizedFailureCount\":3,\"heritrixErrorCount\":0,\"duplicateItemCount\":0,\"duplicateReportRelativeFilename\":null,\"jobId\":-1}, {\"hostname\":\"XXXXXXXX\",\"contentSourceName\":\"Authored\",\"contentSourceType\":\"Authored\",\"incremental\":true,\"skipCrawl\":false,\"isBulk\":false,\"startTime\":1616742071025,\"endTime\":1616794342113,\"reportStatus\":\"Success\",\"documentsFound\":83004,\"documentsFailed\":640,\"documentsSucceeded\":81533,\"documentsFiltered\":0,\"documentsUnchanged\":0,\"contentProcessed\":81528,\"contentAdded\":0,\"contentUpdated\":0,\"contentDeleted\":0,\"pdfSlices\":0,\"pdfDocCount\":0,\"exceptionCount\":0,\"generalExceptionCount\":0,\"warningCount\":0,\"processorFailureCount\":640,\"generalizedFailureCount\":640,\"heritrixErrorCount\":0,\"duplicateItemCount\":0,\"duplicateReportRelativeFilename\":null,\"jobId\":-1}] ================================" | eval _raw="{\"events\":".rtrim(_raw,"=")."}" | spath path=events{} output=events | mvexpand events | eval _raw=events | fields _raw | spath | eval startTime=round(startTime/1000,3) | eval endTime=round(endTime/1000,3) | eval duration=tostring(endTime-startTime,"duration") | fields hostname contentSourceName incremental startTime endTime duration reportStatus documentsFound documentsFailed | fields - _* | fieldformat startTime=strftime(startTime,"%Y-%m-%d %H:%M:%S.%Q") | fieldformat endTime=strftime(endTime,"%Y-%m-%d %H:%M:%S.%Q") ... View more

Hi @ravir_jbp, Great to hear it helped you, I added port field as well; rex "\[[^\]]+\]\s\w+\s(?<service>[^\/]+)\/\w+\-(?<hostname>\w+)\-(?<port>\d+)\s\d+(?:\/\d+){4}\s(?<status_code>\d+)\s(?<response_time>\d+)" | table _time service hostname port status_code response_time ... View more

Hi @manjunathmeti    IT worked!!! thank you so much for all your help.  ... View more