Friday funnies, Info

15 years in this industry – a Murex retrospective

Sometimes I look back at how things have changed since I started in this industry. It’s always interesting to see how things have changed as it can hint at to where things will head next. So join me on this Murex retrospective.

I started working for Murex in 2000: Matrix, internet bubble, Y2K bubble burst, Nirvana (hum, no, they were already gone). I was quite impressed with my mentor at the time as she knew everything about Murex and finance. Murex was about to release its first Java based version (the first of mxg2000 generation).

Banks were not expecting so much from vendors: they had the financial knowledge and all they expected was a software that could do what they wanted to do.

Over the years, this has changed quite a lot. They still have expertise domains but often will use vendors as a source of information as to what is being done on the market and what they should be doing. Let’s drill into some specifics:

Rate curves

Back in 2000, people were satisfied with 1 curve per currency without much focus being put on basis curves and risk.
Then came the USD basis curves and later the single currency basis curves with a major focus on OIS curves. Now you can add to the mix funding curves and from 1 to 2 curves per currency, you have now 5-6 curves for each currency.

Volatility

Volatility also changed over the years. Started quite simply with an ATM curve and smile. But complexity started to kick in. Normal volatility came first as it offered a more stable measure of vega without any noise from prices. It accelerated when interest rates started to become negative and one had to use either shifted lognormal or normal models.
Then the volatility models became more popular and a must have: SABR for instance where traders start to measure their risk against the parameters.

The changes also occurred in Murex when trying to cater for a demand of always more flexibility: workflows, viewers, eTradepad, system architecture.

What’s next?

Can we expect the same trend to keep going: an always more flexible software and an industry always more complex?

I’m not sure. In regards to the industry, the move towards clearing for derivatives might push the banks to investigate more complex products that would fall outside clearing. But would there be a market for them, especially if more and more derivatives are cleared and benefit from a better price?
Or would the market data and pricing models increase in complexity in order to offer a price that is always closer to where the market actually is?

Regarding the software, it could always offer more flexibility but with flexibility comes complexity. Software complexity has a cost in terms of upgrade, configuration and maintenance. So past a certain point, the benefits are basically not worth the cost. Have we reached that point yet, I’m not sure at all?

And you? What do you think about the future?
And how was your learning experience: daunting? Challenging? Or a walk in the park?

Tips

Spring cleaning, Purge the database

Spring cleaning, I know I’m a month early but purge is an important task and sometimes you need to make sure it is adapted to your environment and needs.

Purging the database will let you keep the database growth under control and ensure that you get the maximum performance out of the system. But there’s often a fear that purging will result in data loss and quickly you find yourself with massive retention periods, 7 years for trade,  2 years of daily market data and all logs.

The first to keep in mind: Murex is a production system for trading and processing, it is not a data repository system. You need to keep it running in top shape as to maximize the benefits you get from the system. If you need to retain some data stored in Murex, export and store it on your own system. It is much cheaper and more appropriate.
This might sound obvious but when talking about purge, regulation is often the first topic that comes and it blocks any further discussion as long as a solution for storing all the data to be purged has not been implemented.

Once everyone is convinced of the importance of the purge, there are multiple items to purge by importance:

– Documents and their entries (usually ranking at number 1 in DB usage)

– Market data (normally ranking at number 2)

– Trades

– Logs

– Static data

– The forgotten ones: view, layouts, filters

Documents

Purging Mxmlexchange is actually quite straight forward and is done through scripts provided by Murex. Just be very careful with the scripts and ensure that proper testing is done on test environments before deploying to production.

But if you test it properly and only purge the intermediate documents, it is quite straight forward without surprises

Market data

Market data is made of 2 parts. The visible side of the iceberg where you purge market data for dates you no longer need (good practice tends to let people keep the month end only for older dates and daily market data for few months (1-3 depending on your aggressiveness). This can be done through the GUI if you want, quite straight forward.

But there’s also a second part of market data  purge which helps a lot: expired instruments (read Bonds and listed options mainly). By default, Murex automatically copies all market data entries from today to tomorrow as part of EOD. This automatic copy means you also have entries for expired listed options (ETOs), futures or bonds which keep being rolled. It might not sound like much but ETOs can quickly snowball especially if you trade very short dated ones such as intradays and overnight. Here, Murex can provide you with a script to clean them out. Symptom for this second one are tables such as MP*_GLOB and MP*_PRIC being large in size.

Trades

Trade purging makes sense especially when you do volume trading. The trade purging is done through the GUI (very important) and in such a fashion that all purged positions are getting aggregated to avoid any jump in cash balances.

The trade purge occurs in 2 steps: a logical one, where the trade is no longer read for reports and simulation but is still present in the database. All its contributions are stored and aggregated with other purged deals. It can be undone if required
The physical purge will effectively remove the trade from the system, you cannot anymore query it and it cannot be reversed.

Position and cash balances testing needs to be performed after each purge step. After the logical purge, it is the most important as Murex will no longer evaluate the trade but read directly its stored contribution. After the physical purge could almost be skipped as it does not affect anymore the aggregated results, it is simply removing the unused trade records.

Trade purging depends on the trade complexity, simple spot forwards can (and should) be purged much more aggressively than more structured deals

Logs and audit

Murex will give you the scripts for these, purge as requested and make a copy if you feel the need upfront. They don’t consume much space but clean logs make browsing through them a lot easier!

Static data

I am actually an advocate against purging static data. Murex often references static data under the purged deal contributions or in other places and removing them, will remove that link for Murex. One could always try to fix all the problems which ensure out of it but in my opinion, it is simply not worth it. The amount of problems generated (and which could come later during or after an upgrade) is not worth the small amount of DB it occupies.

Filter, layouts, views, etc…

These items should not be purged per se but should be kept under control. Restraining users from creating, duplicating is probably the way to go.
To clean them up would probably have not much of an impact on the database but you risk that an EOD report or a process would fail. Except if you have kept a very precise list of which items are used by what process (and if you did, kudos!), you probably have to leave them where they are or start a massive campaign identifying and decommissioning the unwanted ones.

 

In summary, if you concentrate on the top 4 items of this list, your DB should grow as expected when the hardware was planned with Murex and performances will remain optimal. Just keep an eye on the DB usage by table and if something grow too quickly, Murex will always be happy to sort you out!

If I forgot something or if you feel like to add something, please feel free to!

Friday funnies

Murex bugs: Funny stories

For this Friday, I’ve decided to write about the bad and ugly: Murex bugs!

All software contains bugs (even good old Notepad). I won’t go into details about how to debug them or anything like that. Today, I’ll be interested more in the exotic ones. The ones that as a customer, you report knowing that:

  1. No one will believe you
  2. They can’t be reproduced
  3. They always happen when you’re about to go on weekend

As a Murex consultant:

  1. No one will believe you
  2. Your bug will tank in severity and priority
  3. You’re in for a debugging session

The problems with these bugs is that while rare they do happen and are hard to pinpoint.

I’ll start with my number one in weirdness:

Exotic Murex bug number 1: The 1 week curse of bermudan swaption.

At my desk and got a call: simulation is crashing for the IRO exotic book. Alright, usual checks and impossible to trace down what is causing the issue. On-site visit, can reproduce the crash and as all debugging methods failed, had to break down the portfolio till I isolated 1 deal. A simple bermudan swaption expiring in 2025 (at the time we we were in 2005? 2006?). Long story short: the root cause of the crash was the expiry date. If it was falling within 1 week in 2025 it was crashing! Any bermudan that would mature during that week would crash.

The problem was a memory corruption. It was not happening on every server OS. And in the end, a debug binary showed the developer where to fix his code.
Problem solved but when good luck explaining to a trader he should not have bermudan maturing during that week till it gets patched!

Number 2 in my experience of funny Murex bugs:

Exotic Murex bug number 2: Computer says no

When trying to open trade number 100, trade number 99 gets opened. Trade number 99 opens normally. The two trades were not linked (nice try for those who thought it could have been the case). Whatever we were doing, we could not get to open trade 100 from the browse. If we were querying trade 100 alone, then it would open fine.
All traces were showing that we were querying trade 99. I had given up when a PAC guy came up with an idea (bless him for being around that day!). The issue was the java version of the client being too old and a java bug was causing the focus to go on another trade.
Why was it happening only on that trade? I will never know but the problem did disappear with the correct java version.

Last one for today:

Exotic Murex bug number 3: When consolidated simulation is right and detailed wrong

When you start working on Murex, there are some Murexian laws you learn quickly one of them being that: Always trust the detailed simulation over the consolidated one.

We got a call from a trader complaining that his detailed simulation was wrong but the consolidated one was correct. Back to rule number 1 above, we did not believe him at all but we went to have a look with him.

And truth was: detailed was indeed incorrect and conso was right!

The issue was that the workflow was playing with the trades entered, somehow the trader was losing access rights to the transaction and were no longer loaded in a detailed simulation. But the warehouse was correctly aggregating the trade and showing the right position.

A quick fix in the workflow status and updating the existing trades fixed the issue and let us move on.

 

Just to keep things in perspective, such strange bugs occurs less than 1% of the time. That’s what keeps them exotic and every Murex expert after few years will always have couple of similar stories to tell.

And exotic or not remember, as any bug you need to:

Do your part: smash bugs!
Do your part: smash bugs!
Tips

Murex viewer – My 10 tips

Murex viewer, I still remember when it started being available in the very early production releases (2003 I think?). Concept seemed fantastic but it still needed to mature.

The concept is simple: rather than showing hardcoded/static screens to show results and/or to perform actions what about letting the users build their own and re-arrange them as wanted. The viewer was born with functions very similar to pivot tables. The source data being everything Murex static data and all computed data. Add filters, calculated fields, conditional formatting, linked views and you end up with a very rich environment.

Icing on the cake, Murex viewer comes with some preconfigured layouts simplifying everyone’s work as one only needs to modify as required. It is also a showcase of examples where one can see what can be done and how to try to harmonize the views together.

But with such a powerful tool, you won’t master it on day 1 (not on day 2 either). So let me give you 10 tips to become a pro with the viewer:

  1. Use the search function! The size of the data dictionary is staggering. Worse, sometimes fields appear multiple times and do not always display the same value (reason is usually that some fields are asset specifics). Ctrl-F and F3 are your friends. And remember that the search function works only one column of the data dictionary at the time
  2. Don’t over commit. Often I got calls from support teams unable to achieve what they wanted in the Murex viewer. Some functions and/or data might not always be available. Except if you’re 100% sure it works, don’t promise you will deliver it to the enduser
  3. GMPs are your friends when you need external data! This ones always come sooner or later when the traders want to access extra data/inputs. For instance, they wish to set a date limit for short dated bonds. This date could be entered through GMPs (rather than hardcoded). Best of all, when using simulation, the GMPs will be available in the loaded market data for easy access/input
  4. Viewer is not eTradepad! Typical example of being carried away, the viewer is a tool for displaying data and not a calculation engine. It is easy to be mistaken when opening the formula editor as it looks like the eTradepad formula editor but they are different!
  5. Be careful with your colors. This might sound obvious but actually I tend to find it the hardest part of the Murex viewer. When I was preparing demos, I usually ended up spending more time on the colors to use rather than on the data itself.
  6. Endusers should not be able to modify views. This is actually quite important. One would believe that giving everyone the right to create their own would alleviate the support team workload. Quite the contrary. The number of views will grow out of control and supporting users using different views is a nightmare.
  7. An extension to the point above: Try to limit the number of views in the system. As much as possible try to rationalize the views. For support it is much easier when the number of views is low. For upgrades, it is also less work.
  8. Try to limit the number of views on screen. Murex has a client-server infrastructure and most calculations are done one the server side. But the rendering of the screen is done on the client side. If you start to have many views with a large amount of data, you might end up with the ugly out of memory java error.
  9. Item formulas are great but don’t abuse. Items formula lets you use calculated fields as line breakdowns. I often ended up using them when I needed to aggregate 2 different breakdown fields.
  10. New functions. That’s the best part about the viewer. Each new Murex release comes with new features be it graphical or within the viewer, always inquire about the novelties are they’re usually great eye candy!

So here are my 10 tips about the Murex viewer. What about you dear reader? Any other ones?

News

A great article: video games and the economy

Sadly, this article is in French but it’s about the new Greek finance minister. He basically worked for Steam (video game producer also well known for their digital marketplace for video games) for 2 years and was given the chance to have a look at an economy where all data is known. No more statistics, all the information is available and known.

It has been a while since an article struck me as much as that one so I felt like sharing here it today. And for you non French readers, google translate is your friend.

Link to Le monde article (article was free to read for me at the time of posting)

Comments/debate very welcome 🙂

Info, Tips

Sybase vs Oracle

1 Comment

This is the question one often hears when the decision has been to go with Murex: Sybase vs Oracle. Which one is better? Which one do you recommend, etc…

To first repeat what has been said numerous times: Murex works very well with either and if you need to use one or another due to bank policy or any reason, you can’t get wrong. Murex will deliver results and everything will be A-OK.

But there are differences and both have pros and cons. Historically Murex only supported Sybase and many customers feel that they will get better support from Murex if they go with Sybase. Oracle is quite well known at Murex nowadays and there is no change in the quality of support regarding Oracle. PAC team especially is knowledgeable on both fronts and can provide configuration recommendations for both systems.

Even in performance, that’s not where the difference is really going to lay (many people would disagree here and give reason to go for one or another). I feel the difference is pretty in the actual usage of each: they each work slightly differently. Not from a Murex front end of course, to the end user, Sybase or Oracle does not make any difference, system looks the same, functions work the same way. It is really when you start using SQL where you can see differences.

I graduated from SQL school with Sybase as a teacher, so I do know more about Sybase than Oracle.
Sybase wise, identifiers are directly attributed (the good old M_IDENTITY). When writing SQL, no need to take care of that field, it handles itself. With Oracle, it’s a different story, one need to call the sequence (TABLENAME_DBFS) to retrieve the latest number in order to update it. A bit more painful.
SQL clients with Oracle are for some reason always more of a pain especially if you mix direct commands and stored procedures. I used SQL developer and not seing the results of my stored procedures is a pain. I also use a lot SQuirreL. The later works great for everything EXCEPT the initial connection to the Oracle servers. When the server is distant, the initial load of tables took couple of minutes (started at 15 minutes and went down to 2-3 minutes once the link to other offices got upgraded).  Oracle also was a pain with the username/password for each schema. Not too sure why it was like that but while in Sybase one can easily switch from one DB to another with the same user, the way it is configured to work with Murex Oracle forces to log out/log back in (or log in multiple times to each schema).
But I had my fair share of issues with Sybase. DB corruption happened quite a few times (I suspect it happens also with Oracle but I did not experience it firsthand). The worse DB corruption was when receiving a dump from a customer which contained a trigger (triggers are not your friends). That trigger was attached to a different user id which we did not have when we loaded the dump. So we had to reset the user id for that trigger before deleting it. When updating that user id, it caused a DB corruption which could only be solved when stopping/restarting the server. There were other cases but nothing repeating as easily as that one.

I’d be interested to hear from Oracle experts to give me all good sides of Oracle as from my point of view, I usually found Sybase easier to work with and often wasted few hours trying to adapt a stored procedure that I wrote in Sybase to work with Oracle. Usually PAC team were the ones able to set me straight and get the procedure up and running.

Info, Tips

Murex performance – the chicken and egg story

Murex performance is often in the spotlight: how quickly can Murex do XXX or create YYY. (Replace XXX and YYY with your choice of tasks)? The problem is that the list of requirements between 2 customers varies and results in very different timings.

So to take out the main question first (if you’re the sort to prefer short answer): Can you get good performances out of Murex? Absolutely!

How you’re going to achieve depends on few things (which makes answering the question how long does it take to do something impossible to answer):

  • Hardware is the first one to come to mind. With great hardware comes great performance. Well, not really, you also need to have it tuned right but yes it is a major factor
  • Requirements. This one tends to be overlooked: “I want to get real theta for my whole portfolio over the next 10 days, along with a spot shock and at any time rewrite the spot levels. And it needs to be fast!” (you have similar questions with trade input, reports, etc…). Of course, if you ask for time consuming tasks (or put it many consistency checks), you will slow down the processes.
  • Maintenance. If all works fine on day 1 but not 10 days later, clearly there’s some maintenance that was not done properly
  • Software. I put this one last as it is very rarely the software the issue. Very rarely (feels good to repeat it)

For most of this issues, the PAC team is the go-to team. They can size the hardware you need based on your system usage, advise you on maintenance procedures and debug if something runs too slow.

In general if you believe that a process is taking too long given the configuration (inserting a deal takes 5 minutes, report still running after 1h, etc…) you need to do the following.
If it is an isolated occurrence, it could well be a lock either at DB level or at system level. For locks at DB level (rare but it happens), check with your dbas, also check if no heavy process is currently running. For locks at software level, Murex has you covered with ipmonit. Login to ipmonit from the monit tool, and you can access a lock report showing you all the locks put in by the system (for example if someone is editing a trade, it is locked to avoid 2 modifications at the same time). Check the documentation for ipmonit as screenshots are very helpful when navigating the screens.

If it happens all the time, then it is unlikely to be a lock and you need to generate performance traces. The first ones are generated with /TIMER slash command. This slash command will generate mxtiming files into your log directory (you can put the slash command if required in the launchers for services). The mxtiming file will show the time spent on CPU and while waiting for the DB. If time spent on DB is too high, indexes could be missing on the tables. So you need to run a DB traces (shameless link to my older post for how to). These DB traces can be sent to Murex and they will give you the number of logically read on each table. A number too high indicates (likely) that a table is unindexed. Indexing that table should improve performance.

If the system is slow, the reason lies either in the hardware or the configuration. Rarely the problem is due to a bug.

There are also cases where Murex develops a new feature to speed up a process that is known to be always slow due to the sheer amount of computing/data crunching it requires. Parallelization or pre-crunching are the 2 big methods to do so. But this applies when you start to have a volume: inserting a single deal should always be fast!

Comments, experiences are welcome!

Info, Tips

Murex database – Hack your problems away!

2 Comments

Alright, today let’s crack open this black box that is the Murex database! While all of you know that Murex doesn’t publish its database organization, sometimes there is no choice than go directly where the data is.

My rule of thumb is that if one can avoid it, going direct to the database should be avoided. Any problem caused while browsing will have impacts and cause problems in the environment. For reporting, dynamic tables or viewer reports are your friends. For filtering, list of fields is actually quite exhaustive. In many cases, you will find all the information you need without opening any single SQL client. But sometimes, for some filters (back to RQWHERE post!), for some reporting or for some DB cleaning, you’ll need to go through the database.

Working with Murex database is the same as working with any other trading system database: backup, test in test environments, test again, backup and it should work. The problem is that sometimes some fields are not very clear as to what their roles are and when trying to populate lines (insertion or update), this could turn out to be a real problem. Murex consultants are then the best suited to help you out, especially if you’re not sure your request is safe. In case of migrations, again, Murex consultants should be the ones to provide you with the right scripts, only write yours when you’re absolutely confident of what you’re doing.

Now from a Murex consultant point of view, it is not always easy either to determine what fields have what roles. But the first step is to understand what the other party is trying to do. Maybe SQL is not the best way forward and there could be an easier solution?
Then you can check what other people have done. It is rare to have a problem with only 1 customer that has not been encountered by somebody else.

I learned SQL while working at Murex and many times it actually sped up processes tremendously:

– Inserting in bulk some data (or duplicating records)

– Cleaning up unwanted data. Especially logs (or market data, much much faster)

– Building my own extractions when doing reconciliation reporting

But it also happened that my scripts did not work as expected (and lucky I had a backup and was doing it on a test environment): updates/delete without a correct where condition. I once removed all records from the transaction header!

If you’re working on a limited set of tables and you don’t want to call upon the DBAs to do the backup, then you can should use the following tools: Help-Monitor-DPI info-Transfer from RDB to DBF. You will need an authorization code to proceed but then you can transfer the table from the database to a file in the application server file system. The step Transfer from DBF to RDB does just the opposite. So it gives you the flexibility to backup any table you want from the database to the file system and bring it back whenever required.
Note that you can use jokers in the name of the table you wish to transfer and you should not put _DBF but .dbf.

And you? What’s your relationship with SQL? Comments and experiences below if you wish!

Friday funnies

End of day – The joy of night shifts!

The terror of everyone working in this industry: end of day and especially their crashes. Because end of days happen at nights, most of the time everything is automated. Reports, market data copy, accounting generation are all these lovely activities which happen during the night and which are supposed to be a well-oiled mechanic.

Murex has come a long way in terms of End of day issues (EOD for the acronym lovers). When I started, the big fear was the future rollout date as the system would often crash and one would need to sort it out manually, undo what was done and redo manually. Fortunately in 15 years,  if the EODs have become more complex due to stronger requirements, the system has proven to be also a lot more reliable and resilient.

I won’t cover how to debug these issues (that was another post earlier, the search function is your friend!), but the funny cases (well, looking at them now is funny) that I had encounter.

Calling my wife. A customer had my wife number (I had given it to them when my phone was broken) but few years after, they were still calling her for EOD issues. Quite a classic, when it’s 11pm and my wife is awoken with questions she can’t understand.

Missing cutoff times. Such a big stress the first times. GL entries need to be posted by 12am or… Your first times, guesses are that it is the end of the world, bank will go bust, another black Thursday will happen. So when an issue happens, massive panic and how to get everything rolling before the cutoff. Later you learn that the world will keep going if the data to the ledger is late. Not ideal, but not a cause for WW3.

Leveraging of someone’s visit. This happened actually quite a few times. When visiting a customer and the day is closing, I got a call from the office asking to help someone for a small problem. I then spent few hours (finishing just before midnight) to solve that “small” problem that was end of day related. Other variations include saying bye to everyone and being asked if I could help on this tiny matter.

This being said, that’s an aspect of the job I like (and I hate as well). You need to be always on your toes and it’s actually good fun to be at the customer that late: very few people in (and most lights turned off) and quite a relaxed atmosphere.

And you? How much did you enjoy your night shifts?

Tips

Murex go live – High times fun times

Everyone who worked on Murex has been exposed to Murex go live. This is a crucial moment where many different things can happen:

  • Legacy system replacement by Murex
  • Upgrade (or major version migration)
  • New functions launch

Regardless of the reasons for a go live, they are moments of stress, pressure and (hopefully) joy. But most of them (especially the first ones and the big ones) will leave lasting memories.

A go live happens usually on weekends. Once New York finishes trading, the EOD runs and depending on timings, migration might or not start yet. The idea is to have a fallback environment, all ready to go in case the go live won’t happen.

Migration, configuration, checks and regression testings. They happen more or less consecutively with (and that’s mandatory) at least one thing going wrong. Then there’s the rush to get everything all ok before the endusers coming in to approve the results (if required) and finally the go/no-go decision.

The first times one goes through this exercise is highly stressful. But with experience, one starts to learn and stress much much less. During one of my first Murex go live weekends, I met someone really relaxed. He had a few go lives under his belt and was able to take a step back and advise what to do. I remember being very stressed and stumped on a problem but he took him 2s to suggest a report that would help me. You indeed need small hands during these events but you also need knowledgeable people who can keep a cool head.

More recently, I was only on call for the migration/config part (the privilege of experience and seniority) and onsite when the endusers were coming in. I have to admit that I missed the long Saturday nights sitting in front of the computer to get it all working. And catching some Z’s early on Sunday morning to be back for the endusers. I think that’s the downside of experience, you get less thrills.

And you, dear reader, what’s your experience with Murex go live? Got some great stories or some horror ones you want to share?