Yves here. Based simply on Clive carefully drawing out the implications of public information and a wee bit of testing on his own TSB account, it is becoming more and more apparent that the IT disaster at TSB is not only vastly worse than the bank maintains, but was completely self-inflicted. It’s as if they decided to play Russian roulette with a gun with a bullet in every chamber.
Clive’s forensics demonstrate:
Basic customer information is not mapping to his account. TSB does recognize that he had a credit card and apparently has its credit line correct as well. But it has no transaction history, sees it as a new card when it isn’t.
New transactions are not being associated with his account either.
The systems problems aren’t simply an “online banking problem” and go well beyond that. TSB and its parent Sabadell have been telling customers that the “engine” is fine, they just can’t get those pesky, flaky apps to connect to that humming engine properly. Clive went to the branch and they couldn’t find his new transaction either.
Twitter complaints indicate that some (many?) customers who have mortgages similarly have the TSB system, when they finally get in, showing a zero balance. That says the problems with finding historical data are affecting a yet undetermined number of customers for at least two major products, and the ones that almost certainly account for most of TSBs credit risk and profits.
In a best case scenario, TSB has a data mapping problem that it presumably can eventually sort out, but the cost and time involved could be very high. In a worst case, the data is corrupted or lost. And if that has happened on anything more than a trivial scale, I’m not sure how the bank recovers from that. Similarly, if the bank is making payments but they are floating in some sort of data purgatory not posted to account, what does it take to clean that up?
Contrast that with the continued delusion back at Sabadell, :
The bank’s boss, Paul Pester, said TSB will waive £10m in overdraft fees and pay extra interest on current accounts. He has hired a new team of IT experts from IBM who have been told the problems must be fixed by Saturday.
Does Pester also tell pigs to fly?
Update 6:00 AM, courtesy Richard Smith. Well, at least TSB does appear not to have lost track of how much cash is in checking and savings accounts. But otherwise, as he notes: “Going great in the branches, I see.”
just went into branch and have withdrawn all my money. Have no confidence to sort this! Branch had put up barrage of desks and chairs and security guy to block access to cashier and letting people in one at a time.
— Victoria Mason (@VicMas4)
So much for the cheery claims in the press that most customers won’t leave. The body language of the barricade says TSB is not so optimistic. Shorter, per Lambert; “Ooh, a bank run. What fun.”
By Clive, a UK-based IT professional
Watching the unfolding TSB IT debacle (see here and here for a catch-up if you’re late to this party) it has been depressingly predictable but no less disheartening to see the obfuscation, sophistry and media messaging war which the bank’s management has been waging. All evidence — such as the — on how to best handle such crises is that honesty and action is a far better approach than trying to spin your way out of trouble.
I happen to have a TSB product — a credit card which I use as a back-of-the wallet emergency card, just in case I need an alternative means of payment and there’s a problem with my regular cards. So I couldn’t resist using it as a test product for how well, or rather badly, TSB’s IT system recovery is going.
By way of background, there’s three cornerstones to the problems which TSB created for itself as part of its migration to a new banking platform system (and its customers). One was the migration event which needed two days to get all the legacy data over to TSB and run the import jobs — this was obvious from messages which TSB sent to its customers, such as myself, in the run-up to the switch over to the new system.
When I read the communications from TSB, there were a couple of stand-out facts which had me shaking my head. The first was that TSB were moving their entire customer base over to the new system in one go. This required a two day (the weekend of 21-22 April) complete system outage which implied it was only just about feasible to do the transfer and the data load in that timescale. This meant that there was no possibility of pre-live testing. And that there would be a single Critical Success Factor to the migration which was simply measured as getting all the data over to the new TSB system and the imports completing. Whether that data was the right data for the fields being populated in the TSB system and whether fields were mandatory or optional in terms of getting data inputted into them before the start of the on-line day on Monday 23rd was never apparently considered. There are now big gaps in the historic data on the TSB platform – I’ll cover this more below.
Once reports of the botched migration got widespread press attention, I decided to run a test of my own, to see exactly what was the truth of the situation and how that compared to what TSB was trying to convince its customers, the regulators and politicians was the current position. Having given TSB a day’s grace period to resolve what could have been fairly-easily rectified teething troubles, another day to live up to its promise of then fixing the issues it had admitted to I picked day 3 (Wednesday April 25th) to run a transaction through TSB’s new IT system. I used a large UK retailor with a fairly stable and mature EPoS infrastructure in order to rule out any possibility of acquirer-specific issues. I also, wisely as it turned out, generated a small-ticket value transaction (less than £20).
At the merchant, I presented my TSB credit card, which had a nil balance on it as the account is always settled in full at the end of the billing month. When I presented the card, the cashier asked if I could use “less” (NFC chip enabled) as there was a queue and it would be a quicker verification method than the traditional Chip and PIN, so I obliged and tried to wave my card at the card reader. After a few seconds delay, the EPoS terminal advised “Contactless Not Possible Use PIN” which was very odd. Only very rarely should less transactions be rejected — sometimes the card issuer or card scheme will make a random request for a PIN as a fraud prevention measure, or there’s a consecutive less usage limit (such as if you try to do more than 5 or 10 less transactions in succession). The only time apart from those narrow situations is when it is a newly issued card and it is the first occasion that card has been used, where you must initiate a PIN-verified transaction first before less becomes enabled. But that wasn’t the situation with this card — I’d done several less transactions in the past few months. So that told me something was amiss. However, after entering my PIN, the transaction seemed to go through – the receipt from the merchant and the EPoS terminal showed the amount I spent had been authorised.
The next day (Thursday April 26th) I used TSB’s online servicing channel to look at the account and see if my transaction had been posted.
I was here that I discovered the second fundamental problem which you have — the new TSB system itself. It is simply not production ready. There’s loads of cosmetic issues such as ridiculously old date ranges:
Then there’s more serious things like navigation journeys which do incorrect actions (e.g. “Ways to Pay” which should allow you to settle your credit card balance but instead is where users end up being shunted into prompts for credit limit increases):
And that then causes knock-on problems because it triggers functionality which isn’t valid but the system is to dumb to check first – in this case, the TSB system thinks the account is too new to be eligible for a limit review but it nevertheless triggered it as an option anyway.
So, returning to the point about missing data, that shows the third major defect and even deeper issues – the credit card I looked at is a minimum of two years old from the date of account opening, however the way the data was imported into the TSB system, it is being treated as a new-to-brand card dating, presumably, from less than a week ago. Once I saw what data was held about my credit card account, the reason for the less transaction failing was obvious — TSB’s new system is treating my two-year old (minimum) credit card account as newly-opened. My card has at least a year or more of archived statements, but when I tried to view these, the TSB system just hung – so there’s enough data been imported into the new TSB platform to tell it there’s multi-year statement history available, but not the statement history itself. This is a UK Financial Conduct Authority (FCA) regulatory breach – you can’t cut customers off from accessing their account records.
Interestingly, while the transaction amount has been earmarked from the account (it shows an available balance which is less than the credit limit on the card by the amount of the transaction) the transaction details are not yet posted to the account when I attempt to view an account statement – this should have happened overnight and the details of the transaction showing today. So while payments are “working” on the TSB system, as in, not being declined, they aren’t working end-to-end in terms of posting to the customers’ accounts from the back-end through to the front end account servicing layer.
As a follow-up, I went into my local TSB bank branch, advised the teller that I’d made a payment but it wasn’t showing on my online statement of account. I said that I had a retailer receipt, but wanted to know when or if this transaction would be posted to my account. And here I got to the most worryingly of all part of this story — the teller checked on the branch system (I specifically asked them to make a thorough check, including any pending transactions) and pronounced they could see no record of my credit card payment at all. I did ask if I should the retailor to make another payment if this one wouldn’t be going through, the teller advised me that customers should the payee and make a payment if needed. TSB simply could not confirm if the payment would be valid or not. Only time will tell if this transaction does hit my account, in the right amount with the right details.
As it was only my testing of TSB’s systems, I didn’t have an urgent need to confirm this. Imagine if this transaction had been genuinely important to either the person making, or the person receiving, the payment. Or both.
Finally, going back to the new IT system, there’s unforgivable application logic and design bugs. Just for fun, I tried to increase the limit on my credit card account. Even though I don’t have a mortgage or pay rent, the application insisted I populate this field with a non-zero value:
I tried entering nominal sums such as £10 into the field (which would be a misstatement on an application for credit, which is technically fraud) and even larger more typical sums such as £500 or £1000 a month (this would have been fraudulent too). But these were rejected with a message “there is a problem with the information given” – without telling me what the problem was. So TSB is going to get hit on the revenue side, because the sales process is trashed as a result of system issues.
During the course of my testing the TSB online system, I got thrown out several times to the “Logon” screen. This suggests that there is very short session timeouts being set to allow people to have a few minutes access on their accounts then after very small periods of inactivity, they got bumped off to make capacity available. Either that, or you kept landing on the “dead” side of the load balancer and getting thrown off the system as a result – TSB has admitted they are only up on one leg of their 50/50 load balanced system. Response times were also poor – a minute or so to navigate from page to page was not uncommon.
For TSB to say or allude to these IT system problems as being “fixed” is, as my experience and evidence above shows, at the most very, very generous reading of the facts, is a partial view of the truth. At worst, it is downright lying.
Overarching all of this debacle is the complete unsuitability of CEO Paul Pester to manage situations such as this. Infamous for a hard driving heavy on the machismo management style (one senior trade union official I spoke to who didn’t want to be identified said that Pester frequently bragged about competing in endurance sports such as triathlons to demean less physically fit colleagues), Pester is suggestive of a hypermasculinity personality disorder. The corporate culture is aggressive and reported as being frequently intimidating. TSB – and Pester in particular – is zealously anti-union with well-documented attempts to . Blaming, unrealistically demanding and overly-critical management styles are guaranteed to make this sort of bad situation even worse.
No wonder no-one wanted to tell Pester the truth about how unprepared the new TSB platform was to host live customer accounts. We’re all — TSB’s customers, retail stores or similar payees and bullied, harassed and browbeaten TSB branch or call centre staff — acting as (sometimes unpaid) TSB IT testers and defect fixers as a result.
Postscript: After I had checked this post in for Yves to edit and schedule, I made a final attempt at logging in to TSB’s Internet Banking Service before turning in for the evening at around 9:15 PM UK time just on the chance my mysterious test transaction had appeared. The site was even flakier than it had been earlier in the day. When I did finally manage to log in after numerous attempts, all my account statement would show was –
An error has occurred
We’re sorry, but we’re experiencing some technical issues with the service. Please call us on 0345 835 3846. If you’re calling from abroad you can call us on +44 203 284 1581. Lines are open 24 hours a day, 7 days a week
This is now day 6. With no end in sight.