Tag Archives: end of day

Tips

Murex logs and nothing to do with lognormal

When you work with Murex, it is likely that you will encounter issues. (if you don’t you’re indeed very long sighted or you should definitely buy some lottery ticket).

Anyway, it is very likely that you will soon end up in the logs directory of the app directory and then what… You have the launcher logs, process pids, many folders, etc… Knowing which logs to open or check is quite daunting but fortunately Unix/Linux has so many tools to do the search for you that you’re in for a treat.

When browsing the logs, you have  2 best friends : a more experienced consultant (if he has a PAC background, you definitely need him as a friend) and… Google! Google is my go-to whenever I have question as to how to find something based on any criteria.

Example:

Last night I was running our EOD script and wanted to check the answer files which had terminated successfully. Quick google search (“how to find files containing string unix”) and I got: fi

find / -type f -exec grep -l "text-to-find-here" {} \;

Transformed it into:

find . -name '*answer*' -exec grep -l "Successfully" {} \;

Well, it worked but then I realize that I actually only cared about the ones which failed and that was getting too painful to do it myself (I was also getting tired which does not help). Another quick google search and

find .  -name '*answer*' -exec  grep  -H -E -o -c  "Successfully"  {} \; | grep 0

And it gave me the list of logs which failed making it much easier to investigate the failing ones.

So that was my example for end of day. But you could also find the files which were modified in the last 5 minutes, narrowing down a lot the number of items to browse through.

find . -cmin -5

And then you can do another search for OutOfMemory or similar if the filename/filepath is not enough to tell you which logs you should check.

And I would recommend to have a go through the logs quickly. If indeed you can’t find anything, of course you can call in a friend as there is no 50-50 or ask the audience sort of joker. My problem with calling in a friend, is that it gets always more tempting as a quick way to solve a problem and you don’t build up your skills as much.

What’s more frustrating than having a problem, calling Murex and checking the very first file: <servicenamelogfile>.log and get : user XXXX not authorized to log in? (or any similar error which could have been fixed in 2 minutes once you have the error cause).

What about you dear reader? Any other command/tips you would recommend?

Friday funnies

End of day – The joy of night shifts!

The terror of everyone working in this industry: end of day and especially their crashes. Because end of days happen at nights, most of the time everything is automated. Reports, market data copy, accounting generation are all these lovely activities which happen during the night and which are supposed to be a well-oiled mechanic.

Murex has come a long way in terms of End of day issues (EOD for the acronym lovers). When I started, the big fear was the future rollout date as the system would often crash and one would need to sort it out manually, undo what was done and redo manually. Fortunately in 15 years,  if the EODs have become more complex due to stronger requirements, the system has proven to be also a lot more reliable and resilient.

I won’t cover how to debug these issues (that was another post earlier, the search function is your friend!), but the funny cases (well, looking at them now is funny) that I had encounter.

Calling my wife. A customer had my wife number (I had given it to them when my phone was broken) but few years after, they were still calling her for EOD issues. Quite a classic, when it’s 11pm and my wife is awoken with questions she can’t understand.

Missing cutoff times. Such a big stress the first times. GL entries need to be posted by 12am or… Your first times, guesses are that it is the end of the world, bank will go bust, another black Thursday will happen. So when an issue happens, massive panic and how to get everything rolling before the cutoff. Later you learn that the world will keep going if the data to the ledger is late. Not ideal, but not a cause for WW3.

Leveraging of someone’s visit. This happened actually quite a few times. When visiting a customer and the day is closing, I got a call from the office asking to help someone for a small problem. I then spent few hours (finishing just before midnight) to solve that “small” problem that was end of day related. Other variations include saying bye to everyone and being asked if I could help on this tiny matter.

This being said, that’s an aspect of the job I like (and I hate as well). You need to be always on your toes and it’s actually good fun to be at the customer that late: very few people in (and most lights turned off) and quite a relaxed atmosphere.

And you? How much did you enjoy your night shifts?

Tips

Debugging crashes

1 Comment

Alright, this post will be very much focused on how to debug issues and will be low level.

Murex does crash and number of times, working for Murex we did not get all the information we needed to pinpoint precisely the problem.

The system crashes… what to do?

The first thing is to gather information from the user encountering the crash and understand what was the user doing, if it happened regularly, etc… One important thing to check, if the crash is recent: was anything changed recently (change of configuration or setting for example).

The second step is to reproduce the crash on your side (hopefully without the user). If the crash is too random or cannot be reproduced at will, the best is then to gather the core files and share them with Murex to obtain more information as to what the session was doing at the time.

Assuming the crash can be reproduced, the next step is to narrow it down. This step will actually speed greatly the turnaround from Murex side on the problem. And it might also lead to a workaround to the crash. Depending as to where the crash is happening, there could be multiple way of proceeding. I’ll cover the most common case: crash happening while loading multiple transactions.

If you’re lucky, the transaction scan watching might give you the answer. Transaction scan watching basically logs in when a transaction is opened for valuation and then logs it out. So each transaction should appear an even number of times. If one does appear an odd amount of time (and it’s the last one on the list) you might then have a winner. To confirm that the transaction is indeed the culprit, just try your process (accounting entries generation, simulation) on that single trade and see if the crash occurs.

To use transaction scan watching, go to help-monitor, SPB Information and choose transaction scan watching (Zap). This will delete the existing log (transaction scan watching is only used for debugging, you do not endanger anything by zapping it out, except of course if someone else is doing debugging at the same time!). Then choose activate and run your process again. Once it crashes, with a new session choose Transaction scan watching (Display) to check the log.

If the transaction scan watching fails to focus on a single trade, it will give you trades which have already been processed and might also give you a trend: all IRS have passed, it failed while doing loans. Be careful with this info as the way of scanning transactions is not always clear.
Without the transaction scan watching, to narrow it down, you then need to break it down by portfolios or deal type (usually the knowledge of what was changed recently helps as you can focus on what you suspect being wrong first). Don’t hesitate to use filters on trades inserted today or which had events performed on them during the day.

Once you have the trade, check it out (check also the static data and market data it uses) to understand where is the issue coming from. If you can fix it, it’s a win .
If you can’t, you can then test that the issue occurs with a new trade for which you picked the same pattern. And in that case, congrats you’ve found the bug and can raise it as such.