Tag Archives: debug

Formula debugger… More addictive than heroin

Hum, probably not the best title in the world. But that’s the truth: for any Murex support person, the formula debugger has become very hard to live without.

What is Formula debugger?

I’ll have to assume that you are from another planet . The formula debugger is THE tool to use whether you are actually working (so useful) or just pretending (looks very serious and on the scale of Lazy Fridays Hangover I am Coming to Work but I am Still Asleep (LFHICWISA), it scores a 9 (only beaten by the Excel screensaver)).

As the formula debugger name could have led you to believe: It is used to debug formula. When you turn it on, whenever you are using functions which calls the pretrade, you will see a pop up with all the formulas executed as well as their resolution.

How does it work?

To turn it on, it is very simple, from your enduser session, do Help-Monitor-Formula debugger. You will get a pop-up stating: Formula debugging is now on.  (do the same steps to turn it off)

Now, very simply price a trade from ePad, or whatever you want to do, and a pop up will come up showing you the different formulas executed.

Even better (it’s like a double rainbow) if shows you the resolution of formulas. For example (first line is my formula, second one is the resolution details always starting with //)

IF fLFHICWISA>9. THEN sResults:=’Better than Excel Screensaver’

// IF 8.5>9. statement result is FALSE

You also get the content of the variables when they are assigned:

sBonanza:=legGetString(pLeg,”Counterpart”)

// legGetString(pointer,”Counterpart”) returned “HSBC”

And of course you also get the name of the formula being executed so you can check the formula you have been working on is properly being used by the pricing.

And the whole text is exportable to Notepad or you can also search through it.

Any more tips?

You are getting quite demanding, but I actually have 2 more things:

One is the obvious Clear button, to make sure that you only have as debugging output what you need to focus on (between 2 tests, just clear the window)

And the other one is to couple with the action button wf:XML refresh in the strategies/notpads (right click on the folder,Properties, Actions. Then you can add a button wf:XML refresh. Usually it is the last one in the list.

This button lets you reload all the changes you made in the formulas in a CONFIG session without having to exit eTradepad and re-enter.

Time for you to get cracking on new strategies and formulas!

Murex logs and nothing to do with lognormal

When you work with Murex, it is likely that you will encounter issues. (if you don’t you’re indeed very long sighted or you should definitely buy some lottery ticket).

Anyway, it is very likely that you will soon end up in the logs directory of the app directory and then what… You have the launcher logs, process pids, many folders, etc… Knowing which logs to open or check is quite daunting but fortunately Unix/Linux has so many tools to do the search for you that you’re in for a treat.

When browsing the logs, you have  2 best friends : a more experienced consultant (if he has a PAC background, you definitely need him as a friend) and… Google! Google is my go-to whenever I have question as to how to find something based on any criteria.

Example:

Last night I was running our EOD script and wanted to check the answer files which had terminated successfully. Quick google search (“how to find files containing string unix”) and I got: fi

find / -type f -exec grep -l "text-to-find-here" {} \;

Transformed it into:

find . -name '*answer*' -exec grep -l "Successfully" {} \;

Well, it worked but then I realize that I actually only cared about the ones which failed and that was getting too painful to do it myself (I was also getting tired which does not help). Another quick google search and

find .  -name '*answer*' -exec  grep  -H -E -o -c  "Successfully"  {} \; | grep 0

And it gave me the list of logs which failed making it much easier to investigate the failing ones.

So that was my example for end of day. But you could also find the files which were modified in the last 5 minutes, narrowing down a lot the number of items to browse through.

find . -cmin -5

And then you can do another search for OutOfMemory or similar if the filename/filepath is not enough to tell you which logs you should check.

And I would recommend to have a go through the logs quickly. If indeed you can’t find anything, of course you can call in a friend as there is no 50-50 or ask the audience sort of joker. My problem with calling in a friend, is that it gets always more tempting as a quick way to solve a problem and you don’t build up your skills as much.

What’s more frustrating than having a problem, calling Murex and checking the very first file: <servicenamelogfile>.log and get : user XXXX not authorized to log in? (or any similar error which could have been fixed in 2 minutes once you have the error cause).

What about you dear reader? Any other command/tips you would recommend?

Murex assistance or handover, how much information does one need?

I had that conversation last week: in order to provide Murex assistance, how much information is required?

Murex consultants (from Murex) providing support usually don’t need design documents, they can be useful sometimes but most of the time they will intervene on single specific problems. Access to an environment is ample information and they will dig in to retrieve whatever information is necessary. If you ask them as to why a crash is occurring, a task stopped working or a formula is not triggered, they do not need to look at the full picture. They can only focus on the single issue and all the information will be at hand (remember that Murex gives usually easy access to all the information used to produce a result: rate curve will give all pricing details, formula debugger will show code execution as well as variable values.

The other reason is that usually Murex consultants are supporting multiple customers and reading through the design document would take a significant amount of time while yielding no direct advantage to answer questions.

On the other hand, internal support team members  need to read the design documents if they haven’t been involved in the project. The design document will effectively describe the big picture and explain how everything fit together.

But thinking it over about the time I was supporting different customers at Murex, we would either ask the customers when something seemed not right. Sometimes it would indeed be a design decision and it would not make sense to go against it.
An example: few weeks ago I wrote about rate propagation and basically KMQA/IS is the most logical choice. But you will find customers using KZQC, especially in the Fx world, and this is perfectly normal.

Of course, there is a real added value for the Murex consultants to know about the design choices. This knowledge could come from reading the design document or more often from frequent support and proximity to the customer. And I was indeed more efficient in my efforts when I clearly knew why such a configuration was selected.

Does this mean that Murex consultants should read and go through design documents? I don’t think so, except if they are supporting only that customer (or very few customers). Their job is to provide information about the software, help you get most of it and solve your issues. But the internal support team can provide them assistance and guidance as to why a solution would not be suitable in the customer’s context. Good support is always at the encounter of Murex knowledge and customer configuration knowledge.

Murex bugs: Funny stories

For this Friday, I’ve decided to write about the bad and ugly: Murex bugs!

All software contains bugs (even good old Notepad). I won’t go into details about how to debug them or anything like that. Today, I’ll be interested more in the exotic ones. The ones that as a customer, you report knowing that:

  1. No one will believe you
  2. They can’t be reproduced
  3. They always happen when you’re about to go on weekend

As a Murex consultant:

  1. No one will believe you
  2. Your bug will tank in severity and priority
  3. You’re in for a debugging session

The problems with these bugs is that while rare they do happen and are hard to pinpoint.

I’ll start with my number one in weirdness:

Exotic Murex bug number 1: The 1 week curse of bermudan swaption.

At my desk and got a call: simulation is crashing for the IRO exotic book. Alright, usual checks and impossible to trace down what is causing the issue. On-site visit, can reproduce the crash and as all debugging methods failed, had to break down the portfolio till I isolated 1 deal. A simple bermudan swaption expiring in 2025 (at the time we we were in 2005? 2006?). Long story short: the root cause of the crash was the expiry date. If it was falling within 1 week in 2025 it was crashing! Any bermudan that would mature during that week would crash.

The problem was a memory corruption. It was not happening on every server OS. And in the end, a debug binary showed the developer where to fix his code.
Problem solved but when good luck explaining to a trader he should not have bermudan maturing during that week till it gets patched!

Number 2 in my experience of funny Murex bugs:

Exotic Murex bug number 2: Computer says no

When trying to open trade number 100, trade number 99 gets opened. Trade number 99 opens normally. The two trades were not linked (nice try for those who thought it could have been the case). Whatever we were doing, we could not get to open trade 100 from the browse. If we were querying trade 100 alone, then it would open fine.
All traces were showing that we were querying trade 99. I had given up when a PAC guy came up with an idea (bless him for being around that day!). The issue was the java version of the client being too old and a java bug was causing the focus to go on another trade.
Why was it happening only on that trade? I will never know but the problem did disappear with the correct java version.

Last one for today:

Exotic Murex bug number 3: When consolidated simulation is right and detailed wrong

When you start working on Murex, there are some Murexian laws you learn quickly one of them being that: Always trust the detailed simulation over the consolidated one.

We got a call from a trader complaining that his detailed simulation was wrong but the consolidated one was correct. Back to rule number 1 above, we did not believe him at all but we went to have a look with him.

And truth was: detailed was indeed incorrect and conso was right!

The issue was that the workflow was playing with the trades entered, somehow the trader was losing access rights to the transaction and were no longer loaded in a detailed simulation. But the warehouse was correctly aggregating the trade and showing the right position.

A quick fix in the workflow status and updating the existing trades fixed the issue and let us move on.

 

Just to keep things in perspective, such strange bugs occurs less than 1% of the time. That’s what keeps them exotic and every Murex expert after few years will always have couple of similar stories to tell.

And exotic or not remember, as any bug you need to:

Do your part: smash bugs!
Do your part: smash bugs!

Murex performance – the chicken and egg story

Murex performance is often in the spotlight: how quickly can Murex do XXX or create YYY. (Replace XXX and YYY with your choice of tasks)? The problem is that the list of requirements between 2 customers varies and results in very different timings.

So to take out the main question first (if you’re the sort to prefer short answer): Can you get good performances out of Murex? Absolutely!

How you’re going to achieve depends on few things (which makes answering the question how long does it take to do something impossible to answer):

  • Hardware is the first one to come to mind. With great hardware comes great performance. Well, not really, you also need to have it tuned right but yes it is a major factor
  • Requirements. This one tends to be overlooked: “I want to get real theta for my whole portfolio over the next 10 days, along with a spot shock and at any time rewrite the spot levels. And it needs to be fast!” (you have similar questions with trade input, reports, etc…). Of course, if you ask for time consuming tasks (or put it many consistency checks), you will slow down the processes.
  • Maintenance. If all works fine on day 1 but not 10 days later, clearly there’s some maintenance that was not done properly
  • Software. I put this one last as it is very rarely the software the issue. Very rarely (feels good to repeat it)

For most of this issues, the PAC team is the go-to team. They can size the hardware you need based on your system usage, advise you on maintenance procedures and debug if something runs too slow.

In general if you believe that a process is taking too long given the configuration (inserting a deal takes 5 minutes, report still running after 1h, etc…) you need to do the following.
If it is an isolated occurrence, it could well be a lock either at DB level or at system level. For locks at DB level (rare but it happens), check with your dbas, also check if no heavy process is currently running. For locks at software level, Murex has you covered with ipmonit. Login to ipmonit from the monit tool, and you can access a lock report showing you all the locks put in by the system (for example if someone is editing a trade, it is locked to avoid 2 modifications at the same time). Check the documentation for ipmonit as screenshots are very helpful when navigating the screens.

If it happens all the time, then it is unlikely to be a lock and you need to generate performance traces. The first ones are generated with /TIMER slash command. This slash command will generate mxtiming files into your log directory (you can put the slash command if required in the launchers for services). The mxtiming file will show the time spent on CPU and while waiting for the DB. If time spent on DB is too high, indexes could be missing on the tables. So you need to run a DB traces (shameless link to my older post for how to). These DB traces can be sent to Murex and they will give you the number of logically read on each table. A number too high indicates (likely) that a table is unindexed. Indexing that table should improve performance.

If the system is slow, the reason lies either in the hardware or the configuration. Rarely the problem is due to a bug.

There are also cases where Murex develops a new feature to speed up a process that is known to be always slow due to the sheer amount of computing/data crunching it requires. Parallelization or pre-crunching are the 2 big methods to do so. But this applies when you start to have a volume: inserting a single deal should always be fast!

Comments, experiences are welcome!

Murex database – Hack your problems away!

Alright, today let’s crack open this black box that is the Murex database! While all of you know that Murex doesn’t publish its database organization, sometimes there is no choice than go directly where the data is.

My rule of thumb is that if one can avoid it, going direct to the database should be avoided. Any problem caused while browsing will have impacts and cause problems in the environment. For reporting, dynamic tables or viewer reports are your friends. For filtering, list of fields is actually quite exhaustive. In many cases, you will find all the information you need without opening any single SQL client. But sometimes, for some filters (back to RQWHERE post!), for some reporting or for some DB cleaning, you’ll need to go through the database.

Working with Murex database is the same as working with any other trading system database: backup, test in test environments, test again, backup and it should work. The problem is that sometimes some fields are not very clear as to what their roles are and when trying to populate lines (insertion or update), this could turn out to be a real problem. Murex consultants are then the best suited to help you out, especially if you’re not sure your request is safe. In case of migrations, again, Murex consultants should be the ones to provide you with the right scripts, only write yours when you’re absolutely confident of what you’re doing.

Now from a Murex consultant point of view, it is not always easy either to determine what fields have what roles. But the first step is to understand what the other party is trying to do. Maybe SQL is not the best way forward and there could be an easier solution?
Then you can check what other people have done. It is rare to have a problem with only 1 customer that has not been encountered by somebody else.

I learned SQL while working at Murex and many times it actually sped up processes tremendously:

– Inserting in bulk some data (or duplicating records)

– Cleaning up unwanted data. Especially logs (or market data, much much faster)

– Building my own extractions when doing reconciliation reporting

But it also happened that my scripts did not work as expected (and lucky I had a backup and was doing it on a test environment): updates/delete without a correct where condition. I once removed all records from the transaction header!

If you’re working on a limited set of tables and you don’t want to call upon the DBAs to do the backup, then you can should use the following tools: Help-Monitor-DPI info-Transfer from RDB to DBF. You will need an authorization code to proceed but then you can transfer the table from the database to a file in the application server file system. The step Transfer from DBF to RDB does just the opposite. So it gives you the flexibility to backup any table you want from the database to the file system and bring it back whenever required.
Note that you can use jokers in the name of the table you wish to transfer and you should not put _DBF but .dbf.

And you? What’s your relationship with SQL? Comments and experiences below if you wish!

Database traces

Database tracing is very handy when you’re trying to grasp what is causing an error and you suspect something wrong in the configuration (an incorrectly setup security, market data, etc…) or you’re looking to build an SQL query and need to have a bit of a better understanding of the data model (note that I do not encourage you to use SQL, reporting and dynamic tables should be your first stop but sometimes it is not enough).

In Murex there are couple of database traces and some are more useful than others.

The first one I used is the one coming up through the GUI. Every request pops on the screen and by pressing escape gets executed. It’s simple to use, simple to turn on and don’t need to retrieve the file from the application server. Sadly, it sometimes causes crashes due to the popup windows interfering with the application or sometimes you simply have too many requests for it to be useful.
To turn it on, you need to go to Help-Monitor-DBX Info-Request . You can then enter criteria to filter the requests: From the action (update, delete, insert, select) to a string search (e.g. TRN_HDR or BOND001).
When I began in the Murex world, I loved that tool as it helped me put together the pieces of the puzzle.

The second one is to activate the traces dumped into a log file. To do so, go to Help-Monitor-DBX Info-RDB Statistics. You can choose what level of debugging you need. The basic one is good enough if you need to look at the queries executed. The full verbose (apart from the plan) one is necessary if you’re trying to track missing DB Indices or need to understand which requests are costing too much time. That trace will by default dump files into logs/mxsession/mx but the default path can be modified within the launcher.
One can also turn on that tracing with a slash command: /RDXSTATISTICS:<prefix>:<trace level>

You might end up using the information given from the traces for:

– Building a report

– Building a RQWHERE filter

– Narrowing down an issue

– Building a SQL procedure (useful when upgrading/reconciliation but always check with Murex if it’s all good)

DB traces are not the silver bullet and sometimes won’t give you the information you’re after. Also in case of investigating a crash, keep in mind that the last request might not be the one responsible for the crash and you might need to go back up in the logs to find the root cause.

 

If you have questions, funny stories or feel like it, please comment below!

Debugging crashes

Alright, this post will be very much focused on how to debug issues and will be low level.

Murex does crash and number of times, working for Murex we did not get all the information we needed to pinpoint precisely the problem.

The system crashes… what to do?

The first thing is to gather information from the user encountering the crash and understand what was the user doing, if it happened regularly, etc… One important thing to check, if the crash is recent: was anything changed recently (change of configuration or setting for example).

The second step is to reproduce the crash on your side (hopefully without the user). If the crash is too random or cannot be reproduced at will, the best is then to gather the core files and share them with Murex to obtain more information as to what the session was doing at the time.

Assuming the crash can be reproduced, the next step is to narrow it down. This step will actually speed greatly the turnaround from Murex side on the problem. And it might also lead to a workaround to the crash. Depending as to where the crash is happening, there could be multiple way of proceeding. I’ll cover the most common case: crash happening while loading multiple transactions.

If you’re lucky, the transaction scan watching might give you the answer. Transaction scan watching basically logs in when a transaction is opened for valuation and then logs it out. So each transaction should appear an even number of times. If one does appear an odd amount of time (and it’s the last one on the list) you might then have a winner. To confirm that the transaction is indeed the culprit, just try your process (accounting entries generation, simulation) on that single trade and see if the crash occurs.

To use transaction scan watching, go to help-monitor, SPB Information and choose transaction scan watching (Zap). This will delete the existing log (transaction scan watching is only used for debugging, you do not endanger anything by zapping it out, except of course if someone else is doing debugging at the same time!). Then choose activate and run your process again. Once it crashes, with a new session choose Transaction scan watching (Display) to check the log.

If the transaction scan watching fails to focus on a single trade, it will give you trades which have already been processed and might also give you a trend: all IRS have passed, it failed while doing loans. Be careful with this info as the way of scanning transactions is not always clear.
Without the transaction scan watching, to narrow it down, you then need to break it down by portfolios or deal type (usually the knowledge of what was changed recently helps as you can focus on what you suspect being wrong first). Don’t hesitate to use filters on trades inserted today or which had events performed on them during the day.

Once you have the trade, check it out (check also the static data and market data it uses) to understand where is the issue coming from. If you can fix it, it’s a win .
If you can’t, you can then test that the issue occurs with a new trade for which you picked the same pattern. And in that case, congrats you’ve found the bug and can raise it as such.