Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Are you a member of the Splunk Community?
Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Find Answers
- :
- About siu
siu
Loves-to-Learn Everything
Member since:
02-16-2023
01-20-2025
Community Statistics
| Posts | 28 |
| Solutions | 0 |
| Karma Given | 0 |
| Karma Received | 0 |
| Member Since | 02-16-2023 |
Activity Feed
- Posted Re: Joining searches on common columns on Splunk Search. 01-03-2025 12:49 AM
- Posted Re: Joining searches on common columns on Splunk Search. 01-02-2025 08:20 PM
- Posted Re: Joining searches on common columns on Splunk Search. 01-01-2025 05:42 PM
- Posted Re: Joining searches on common columns on Splunk Search. 12-31-2024 07:02 PM
- Posted Re: Joining searches on common columns on Splunk Search. 12-30-2024 06:05 PM
- Posted Re: Joining searches on common columns on Splunk Search. 12-29-2024 06:03 PM
- Posted Re: Joining searches on common columns on Splunk Search. 12-29-2024 12:51 AM
- Posted Re: Joining searches on common columns on Splunk Search. 12-28-2024 06:16 PM
- Posted Re: Joining searches on common columns on Splunk Search. 12-28-2024 06:10 PM
- Posted Re: Joining searches on common columns on Splunk Search. 12-28-2024 08:20 AM
- Posted Re: Joining searches on common columns on Splunk Search. 12-28-2024 08:03 AM
- Posted Re: Joining searches on common columns on Splunk Search. 12-27-2024 08:27 PM
- Posted Re: Joining searches on common columns on Splunk Search. 12-27-2024 06:08 PM
- Posted Re: Joining searches on common columns on Splunk Search. 12-27-2024 08:04 AM
- Posted Re: Joining searches on common columns on Splunk Search. 12-27-2024 07:12 AM
- Posted Re: Joining searches on common columns on Splunk Search. 12-27-2024 07:00 AM
- Posted Re: Joining searches on common columns on Splunk Search. 12-27-2024 03:30 AM
- Posted Re: Joining searches on common columns on Splunk Search. 12-27-2024 02:24 AM
- Posted Joining searches on common columns on Splunk Search. 12-27-2024 02:23 AM
- Posted Re: Splunk Custom Search Commands: How can I write the generator.py script so that it calls the script.py? on Splunk Search. 04-28-2023 05:09 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 0 | |||
| 0 |
01-03-2025
12:49 AM
Sure, also all the results are in a single row, ideally I would want them in separate Wahts important to note is that I did get all the columns in the fields command but they are empty (the ones i stated , eg. NB onwards) Time Event 03/01/2025 16:05:37.609 2025-01-03 16:05:37.609, system="murex", id="646523556", sky_id="646523556", trade_id="32248978", event_id="100120362", mx_status="live", operation="nooperation", action="modification", tradebooking_sgp="2025/01/02 02:01:23.0000", eventtime_sgp="2025/01/02 02:01:21.3800", sky_to_mq_latency="-1.-620", portfolio_name="test_oprtoflio", portfolio_entity="test_entity", trade_type="VanillaSwap" 03/01/2025 11:05:39.000 32248978;LIVE;0.00001000;AUD;IRD;CD;;test_prtoflio;CAMBOOYAPTSYDAU
... View more
01-02-2025
08:20 PM
These fields are missing TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO
... View more
01-01-2025
05:42 PM
Thanks! Albeit abit slow and unresponsive i got some results NB action event_id mx_status operation portfolio_entity portfolio_name sky_id trade_type tradebooking_sgp 0 0 LIVE sgp usa usaeod ABC Korea ABC Panema ... ... A USD AOU ... ... ... 12345678 VanillaSwap ... ... YYYY/MM/DD HH:MM:SS ...
... View more
12-31-2024
07:02 PM
Thanks, as per my first post I'd like to join 2 searches together on NB and retain all columns . I am able to retain all columns but some rows are filled are some are not (but NB is definitely matching in both searches) index=sky sourcetype=sky_trade_murex_timestamp
| rex field=_raw "trade_id=\"(?<trade_id>\d+)\""
| rex field=_raw "mx_status=\"(?<mx_status>[^\"]+)\""
| rex field=_raw "sky_id=\"(?<sky_id>\d+)\""
| rex field=_raw "event_id=\"(?<event_id>\d+)\""
| rex field=_raw "operation=\"(?<operation>[^\"]+)\""
| rex field=_raw "action=\"(?<action>[^\"]+)\""
| rex field=_raw "tradebooking_sgp=\"(?<tradebooking_sgp>[^\"]+)\""
| rex field=_raw "portfolio_name=\"(?<portfolio_name>[^\"]+)\""
| rex field=_raw "portfolio_entity=\"(?<portfolio_entity>[^\"]+)\""
| rex field=_raw "trade_type=\"(?<trade_type>[^\"]+)\""
| rename trade_id as NB
| table sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type index=sky sourcetype=mx_to_sky
| rex field=_raw "(?<NB>\d+);(?<TRN_STATUS>[^;]+);(?<NOMINAL>[^;]+);(?<CURRENCY>[^;]+);(?<TRN_FMLY>[^;]+);(?<TRN_GRP>[^;]+);(?<TRN_TYPE>[^;]*);(?<BPFOLIO>[^;]*);(?<SPFOLIO>[^;]*)"
| eval NB = tostring(trim(NB))
| table TRN_STATUS, NB, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO
... View more
12-29-2024
06:03 PM
Thanks guys! index=sky sourcetype=sky_trade_murex_timestamp OR sourcetype=mx_to_sky
``` Parse sky_trade_murex_timestamp events (note that trade_id is put directly into the NB field) ```
| rex field=_raw "trade_id=\"(?<NB>\d+)\""
| rex field=_raw "mx_status=\"(?<mx_status>[^\"]+)\""
| rex field=_raw "sky_id=\"(?<sky_id>\d+)\""
| rex field=_raw "event_id=\"(?<event_id>\d+)\""
| rex field=_raw "operation=\"(?<operation>[^\"]+)\""
| rex field=_raw "action=\"(?<action>[^\"]+)\""
| rex field=_raw "tradebooking_sgp=\"(?<tradebooking_sgp>[^\"]+)\""
| rex field=_raw "portfolio_name=\"(?<portfolio_name>[^\"]+)\""
| rex field=_raw "portfolio_entity=\"(?<portfolio_entity>[^\"]+)\""
| rex field=_raw "trade_type=\"(?<trade_type>[^\"]+)\""
``` Parse mx_to_sky events ```
| rex field=_raw "(?<NB>[^;]+);(?<TRN_STATUS>[^;]+);(?<NOMINAL>[^;]+);(?<CURRENCY>[^;]+);(?<TRN_FMLY>[^;]+);(?<TRN_GRP>[^;]+);(?<TRN_TYPE>[^;]*);(?<BPFOLIO>[^;]*);(?<SPFOLIO>[^;]*)"
``` Reduce to just the fields of interest ```
| fields sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type, TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO
``` "Join" events by NB using stats ```
| head 1000
| stats values(*) as * by NB
| fillnull
| head 1
| transpose 0 Now its this I get a 1000 events for an hour range estimated and results as shown column row1 NB 0 action 0 event_id 0 mx_status live operation sydeod portfolio_entity SG KOREA USA ... portfolio_name AUD APT ... sky_id 673821 ... trade_type VanillaSwap tradebooking_sgp 2024/12/26 00:06:34.3572 ...
... View more
12-29-2024
12:51 AM
index=sky sourcetype=sky_trade_murex_timestamp OR sourcetype=mx_to_sky
``` Parse sky_trade_murex_timestamp events (note that trade_id is put directly into the NB field) ```
| rex field=_raw "trade_id=\"(?<NB>\d+)\""
| rex field=_raw "mx_status=\"(?<mx_status>[^\"]+)\""
| rex field=_raw "sky_id=\"(?<sky_id>\d+)\""
| rex field=_raw "event_id=\"(?<event_id>\d+)\""
| rex field=_raw "operation=\"(?<operation>[^\"]+)\""
| rex field=_raw "action=\"(?<action>[^\"]+)\""
| rex field=_raw "tradebooking_sgp=\"(?<tradebooking_sgp>[^\"]+)\""
| rex field=_raw "portfolio_name=\"(?<portfolio_name>[^\"]+)\""
| rex field=_raw "portfolio_entity=\"(?<portfolio_entity>[^\"]+)\""
| rex field=_raw "trade_type=\"(?<trade_type>[^\"]+)\""
``` Parse mx_to_sky events ```
| rex field=_raw "(?<NB>[^;]+);(?<TRN_STATUS>[^;]+);(?<NOMINAL>[^;]+);(?<CURRENCY>[^;]+);(?<TRN_FMLY>[^;]+);(?<TRN_GRP>[^;]+);(?<TRN_TYPE>[^;]*);(?<BPFOLIO>[^;]*);(?<SPFOLIO>[^;]*)"
``` Reduce to just the fields of interest ```
| fields sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type, TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO
``` "Join" events by NB using stats ```
| dedup NB
| stats values(*) as * by NB
... View more
12-28-2024
06:16 PM
OH also i dont get these in the results TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO I only see the first half of the fields command I included.
... View more
12-28-2024
06:10 PM
thanks! Yep i tried that for just an hour range. I got one single row of data with everything in there it seems. I also couldnt scroll the page to confirm as page became unresponsive
... View more
12-28-2024
08:20 AM
im gonna try this thanks!! I think i got something like all the results into one row and performance is very bad as there are many events i did not manage to get a proper search result
... View more
12-28-2024
08:03 AM
ah yes i removed those and the search continued. But it was so laggy as there were many events, i did not get a proper search result without it hanging
... View more
12-27-2024
08:27 PM
Hi all, this i get about Running4,003,400 of 4,003,400 events matched in aday | join type=left NB
[ search index=sky sourcetype=mx_to_sky
| rex field=_raw "(?<NB>[^;]+);(?<TRN_STATUS>[^;]+);(?<NOMINAL>[^;]+);(?<CURRENCY>[^;]+);(?<TRN_FMLY>[^;]+);(?<TRN_GRP>[^;]+);(?<TRN_TYPE>[^;]*);(?<BPFOLIO>[^;]*);(?<SPFOLIO>[^;]*)"
| table TRN_STATUS, NB, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO]
| table sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type
| table sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type, TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO index=sky sourcetype=mx_to_sky
| rex field=_raw "(?<NB>[^;]+);(?<TRN_STATUS>[^;]+);(?<NOMINAL>[^;]+);(?<CURRENCY>[^;]+);(?<TRN_FMLY>[^;]+);(?<TRN_GRP>[^;]+);(?<TRN_TYPE>[^;]*);(?<BPFOLIO>[^;]*);(?<SPFOLIO>[^;]*)"
| table TRN_STATUS, NB, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO This below i get about 1,065,810 events in a day
... View more
12-27-2024
06:08 PM
Thank you both, I need for >50k/>10k events I am thinking of using appendcols but they are not able to join like this. Any other work around? both searches amass quite a huge number of events >50k >10k and I need to search for today which would be alot.
... View more
12-27-2024
08:04 AM
Its quite random but here is one 2024-12-27 23:05:09.917, system="murex", id="645437844", sky_id="645437844", trade_id="31791027", event_id="100038914", mx_status="live", operation="nooperation", action="fixing", tradebooking_sgp="2024/12/27 08:42:39.0000", eventtime_sgp="2024/12/27 08:42:33.6400", sky_to_mq_latency="-5.-360", portfolio_name="A ZZ AUD LTR", portfolio_entity="ANZBG MELB", trade_type="BasisSwap" 31791027;LIVE;17500000.00000000;AUD;IRD;IRS;;A ZZ AUD LTR;X_GCM_IRD_AUCNZ index=sky sourcetype=sky_trade_murex_timestamp
| rex field=_raw "trade_id=\"(?<trade_id>\d+)\""
| rex field=_raw "mx_status=\"(?<mx_status>[^\"]+)\""
| rex field=_raw "sky_id=\"(?<sky_id>\d+)\""
| rex field=_raw "event_id=\"(?<event_id>\d+)\""
| rex field=_raw "operation=\"(?<operation>[^\"]+)\""
| rex field=_raw "action=\"(?<action>[^\"]+)\""
| rex field=_raw "tradebooking_sgp=\"(?<tradebooking_sgp>[^\"]+)\""
| rex field=_raw "portfolio_name=\"(?<portfolio_name>[^\"]+)\""
| rex field=_raw "portfolio_entity=\"(?<portfolio_entity>[^\"]+)\""
| rex field=_raw "trade_type=\"(?<trade_type>[^\"]+)\""
| eval trade_id = replace(trade_id, "\"", "")
| rename trade_id as NB
| table sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type
| join type=left NB
[ search index=sky sourcetype=mx_to_sky
| rex field=_raw "(?<NB>[^;]+);(?<TRN_STATUS>[^;]+);(?<NOMINAL>[^;]+);(?<CURRENCY>[^;]+);(?<TRN_FMLY>[^;]+);(?<TRN_GRP>[^;]+);(?<TRN_TYPE>[^;]*);(?<BPFOLIO>[^;]*);(?<SPFOLIO>[^;]*)"
| table TRN_STATUS, NB, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO]
| table sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type
| table sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type, TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO For that NB i get all columns and empty for TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO
... View more
12-27-2024
07:12 AM
Thanks!! I get this error though Error in 'multisearch' command: Multisearch subsearches might only contain purely streaming operations (subsearch 1 contains a non-streaming command).
... View more
12-27-2024
07:00 AM
No problem. But why is it that some rows are populated and some are not then? i.e some match and some don't match. I renamed trade_id as NB then left=join NB so it should join 2 of them together but why does it not work for some rows or columns although clearly it matches in both searches?
... View more
12-27-2024
03:30 AM
Sorry how do i rectify it? 32265376;DEAD;3887.00000000;XAU;CURR;FXD;FXD;CM TR GLD AUS;X_CMTR XAU SWAP Did u mean this
... View more
12-27-2024
02:24 AM
And the problem is these columns are empty for some and populated for some. For those empty, I clearly checked the NB is matching in both searches TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO
... View more
12-27-2024
02:23 AM
HI query joining 2 searches on left join. Its matching some rows and not matching some rows although the column where I join on is clearly seen in both searches. index=sky sourcetype=sky_trade_murex_timestamp
| rex field=_raw "trade_id=\"(?<trade_id>\d+)\""
| rex field=_raw "mx_status=\"(?<mx_status>[^\"]+)\""
| rex field=_raw "sky_id=\"(?<sky_id>\d+)\""
| rex field=_raw "event_id=\"(?<event_id>\d+)\""
| rex field=_raw "operation=\"(?<operation>[^\"]+)\""
| rex field=_raw "action=\"(?<action>[^\"]+)\""
| rex field=_raw "tradebooking_sgp=\"(?<tradebooking_sgp>[^\"]+)\""
| rex field=_raw "portfolio_name=\"(?<portfolio_name>[^\"]+)\""
| rex field=_raw "portfolio_entity=\"(?<portfolio_entity>[^\"]+)\""
| rex field=_raw "trade_type=\"(?<trade_type>[^\"]+)\""
| rename trade_id as NB
| dedup NB
| eval NB = tostring(trim(NB))
| table sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type
| join type=left NB
[ search index=sky sourcetype=mx_to_sky
| rex field=_raw "(?<NB>\d+);(?<TRN_STATUS>[^;]+);(?<NOMINAL>[^;]+);(?<CURRENCY>[^;]+);(?<TRN_FMLY>[^;]+);(?<TRN_GRP>[^;]+);(?<TRN_TYPE>[^;]*);(?<BPFOLIO>[^;]*);(?<SPFOLIO>[^;]*)"
| eval NB = tostring(trim(NB))
| table TRN_STATUS, NB, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO]
| table sky_id, NB, event_id, mx_status, operation, action, tradebooking_sgp, portfolio_name, portfolio_entity, trade_type, TRN_STATUS, NOMINAL, CURRENCY, TRN_FMLY, TRN_GRP, TRN_TYPE, BPFOLIO, SPFOLIO This above is my source code And the raw data is Time Event
27/12/2024
17:05:39.000
32265376;DEAD;3887.00000000;XAU;CURR;FXD;FXD;CM TR GLD AUS;X_CMTR XAU SWAP
host = APPSG002SIN0117source = D:\SkyNet\data\mx_trade_report\MX2_TRADE_STATUS_20241227_200037.csvsourcetype = mx_to_sky Time Event
27/12/2024
18:05:36.651
2024-12-27 18:05:36.651, system="murex", id="645131777", sky_id="645131777", trade_id="32265483", event_id="100023788", mx_status="DEAD", operation="NETTING", action="insertion", tradebooking_sgp="2024/12/26 01:02:01.0000", eventtime_sgp="2024/12/26 01:01:51.7630", sky_to_mq_latency="-9.-237", portfolio_name="I CREDIT INC", portfolio_entity="ANZSEC INC", trade_type="BondTrade"
host = APPSG002SIN0032source = sky_trade_murex_timestamp sourcetype = sky_trade_murex_timestamp
... View more
04-05-2023
06:22 PM
ahh I see thanks guys, For my custom generating search command would this work? #!/usr/bin/env python
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import \
dispatch, GeneratingCommand, Configuration, Option, validators
@Configuration()
class %(command.title())Command(GeneratingCommand):
filename = Option(require=True)
def generate(self):
filename = self.filename
# Put your event code here
# To connect with Splunk, use the instantiated service object which is created using the server-uri and
# other meta details and can be accessed as shown below
# Example:-
# service = self.service
pass
dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__) After I have this then I would put the python script which needs the file path argument in commands.conf And then run the search command using | mycustomsearchgeneratingcommand filename="D:\Temp\Temp.txt" And if that generates the csv file output then I will test it with user input, creating a field in the dashboard and using as suggested by you guys | mycustomsearchgeneratingcommand filename=$field_token$
... View more
04-05-2023
07:44 AM
Yes it helps but when I have that path=... I want it to be a user input in the dashboard. How can I Do so?
... View more
04-03-2023
07:25 AM
So I have a python script called Analysis.py And normally I would run it locally like this Analysis.py <filepath>, so as an example Analysis.py D:/Temp/temp.txt And what this python script does is it generates a csv file. What I would like to do is a dashboard in splunk which does visualization on this csv file, eg. like a line chart or some bar graphs. However this python script runs with a <filepath> argument. And also, this dashboard would accept a custom input but the user who will input the <filepath> argument and the dashboard will show the results accordingly, visualized in a line chart for example. How can I write a splunk custom search command such that I can create this dashboard
... View more
Labels
- Labels:
-
stats
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
01-20-2025
07:22 AM
|
