Some rather personal experiences of the fiasco:
– Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid extensions block.
– This was introduced at the time of the switchover, and until that point the IT system used looked identical between Lloyds, TSB and Halifax / BOS systems (I have accounts with some of those)
– The online browser-based system was telemetry and JS heavy, replacing a far leaner page
– I was unable to log in during the time of the fiasco, mostly due to 403 errors or timeouts. Often the page would just hang as an async request wasn't answered.
– Once I did manage to log in, I was amazed to see another person's account details (!!!), replete with (their) name and statement.
– I was unable to use online banking to pay bills or check my balance – I could see someone else's account in detail but was too honest to do anything with that knowledge. I can't remember if my card stopped working but I was effectively forced to make other arrangements for quite an extended period of time.
I remember one of those banks using the "leaner" page also had heavy telemetry turned on at some point. I type very fast, so I noticed that when I was entering my user id, it was lagging heavily. Then I turned on developer tools only to see that they were logging all keystrokes to analytics. Including username and password. At first I thought I got a virus or something, but these appeared to be legit scripts from the bank. So I decided to not use that bank account for a while. I wonder why would they turn something like that on.
If you're in the US I know for a fact the regulators listen to and review complaints.
Abusive relationships can trap you, personal or business.
Paper did not have those incompatibility problems...
However, from the BBC article I conclude that even customers with a default browser could not necessarily use their account
Edit: Forgotten not added.
I think you’d get blank looks if you asked that question followed by generic we use modern blah and new improvement next blah
So you have extensions that literally break normal browser behaviour and you are blaming them somehow? CORS is part of browser security and should be respected.
Not saying that TSB aren't clearly a shitshow but maybe just disable the extension for that site.
Are you patting yourself on the back for not commiting fraud?
money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and excessive after-the-fact incident reporting, this all puts a strain on the delivery organization’s ability to untangle the complexity before ‘go live’.
2018: "Timeline of trouble: how the TSB IT meltdown unfolded". [2]
It's probably more complicated than that, but perhaps not much more complicated.
[1] https://www.itpro.co.uk/197982/lloyds-tsb-cuts-more-uk-it-jo... [2] https://www.theguardian.com/business/2018/jun/06/timeline-of...
Here's the thing I've learned over the years: Never touch offshored code. Always go for a complete re-write. Don't add features to it, don't refactor it, don't extend it. Just re-write. In my experience it's the best approach.
I know guys who made it their whole business to go and completely re-write projects from scratch after offshoring efforts failed.
There was also a tightening of posted worker regulations, so that banks couldn't ship workers from overseas as a source of cheap talented workforce.
The confusion caused by ordering a member of the general public not to request a bean from a bean factory in a destroy method implementation still makes me laugh, even now.
Brilliant. Reminisce with a screenshot here:
https://twitter.com/thejackthomson_/status/98856435451268710...
Devs fixed it by returning 200's with stack traces.
And as page was ESI stitched together on Varnish, when they fucked up there wasn't just a stacktrace, but a bunch of different ones in various parts of the page.
> The Migration Programme experienced delays from the outset and fell behind the IMP timings. While progress had been made, on 20 September 2017 the firm decided that the Migration Programme would have to be re-planned. However, nine days after it had resolved to re-plan, and before it had concluded its re-planning exercise, TSB publicly announced it would now migrate in Q1 2018.
This is a shameless indulgence in incompetence and recklessness. They didn't even bother to test large swaths of the transitional data or have a fallback plan if things went wrong.
Most likely their customers will now simply leave and the bank will be shut down.
At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable.
I never claimed to have worked with big data.
If something really urgent had come up I could have done what I needed via telephone banking or in a branch, but it was a huge pain in the arse because of a single point of failure.
Just let me use a Yubikey as my second factor damnit.
How about a website where we pledged to open an account and deposit £X into savings, or switch current account, if they offered Webauthn/FIDO?
>SABIS was TSB’s principal outsourced provider
>SABIS relied extensively on 85 third parties (TSB’s fourth parties) to deliver the systems required for the migration and the operation of the platform, which required it to act as a service aggregator.
It amazes me the sheer complexity of a retail bank software system and I suspect most of it is due to legacy systems, legal requirements and lack of regular spring cleaning.
https://news.ycombinator.com/item?id=16910947
Others
https://hn.algolia.com/?dateEnd=1600300800&dateRange=custom&...
I received compensation for the harm caused and got all of my money back.
In fact it was all quite boring and simply the result of the sheer incompetence of the bank’s leadership in running bank IT.
My understanding was that they’re a law firm, perhaps they’ve also branched into IT consultancy?
1. https://www.natwestgroup.com/news-and-insights/feature-conte...
2. https://www.rics.org/uk/about-rics/corporate-governance/inde...
3. https://www.legalbusiness.co.uk/blogs/metoo-latest-bakers-ap...
4. https://www.unicef.org.uk/press-releases/unicef-uk-confirms-...
5. https://www.civilsociety.co.uk/news/unicef-appoints-differen...
And that's exactly how such IT disasters begin.
If I'm asked for an estimate to do X and I say I can deliver in three weeks, and my boss says customer needs it for golive in three days, I'll try to find some way of making that work. Perhaps they can live without a full solution for the first few weeks, instead requiring only a subset of the requirements in that period. Or, if I insist it just cannot be done, I'll tell the boss who'll try to push back golive if it's important enough.
There needs to be respect for each other and the project though.
https://www.computerweekly.com/news/252474170/TSB-programme-...
It's what is and isn't in the data - often a lot of junk in my experience if the source system is a legacy system that has evolved
what meaning that data has within a completely different system
what the demands are on the completeness of that data is in the new system
how to deal with exceptions
and whether that data can ever be frozen, or whether it is still online (as in the case of banking transactions)
This is unlikely to be simply a technical problem of ETLing tables, changing date ranges from inclusive to exclusive and mapping some address fields.
Of course the size of the data after a certain point does make a big difference to risk planning and business continuity planning. It's not possible to rollback and try again within the migration window should a catastrophic issue occur, and it's not possible to simply run some bulk updates to fix issues during the go-live validation.
It is noted though in this project that the data migration itself was not found to contribute to the failure.
If you have a simple db structure with a few tables and very clear data/index rules then billions and billions of records is pretty easy. Your indexes cut out 99% of the work and everything runs smooth and efficient.
But then you can have eldritch horrors where your stored procedures look like seedy detective novels where you chase join after join and have scary high memory requirements on execution.
Personally I would say it is a lot. No other number mentioned in the article even approaches a billion. Billions are big. It might be not be true big data big, but it is still a lot of customer records for a migration project (depending on what exactly they were trying to do within the migration) and it does illustrate why they had so many issues, because there was a lot to deal with.
(Of course this is just speculation. I have no insider knowledge.)
As a sibling comment states a screw up of this magnitude is never simply a technology issue — it requires bad management at many levels.
Still need to write all the procedures, test it, then do it on live system again.
It might make prototyping easier (...or harder) but that's about it
The other problem (let's just say devs were not... that great with architecture) was that they were getting exceptions in the logs without attached URL and other metadata so they kinda wanted to get that exception directly on the webpage.
I'll paint you a picture of how shoddy some stuff was: they were using a templating language to generate JSONs (and had many bugs in it too) for years, instead of just natively encoding some JSON in Java, coz of party shoddy architecture, partly years-long war between frontend and backed dept. They migrated to Git in... 2018 I think ? No CI/CD of any sort till recently.
End result (of them insisting that they will send the exception to frontend to be seen) was actually exceptions being signed and encrypted blob wrapped with a bunch of JS that gathers all the errors (incl. errors that JS on site might've done), adds all the metadata it sees from "browser perspective" and sends it to monitoring endpoint where that is shoved into ES cluster.
Sort of retarded version of distributed tracing that is now in vogue... done somewhere in 2013. But it did catch a bunch of bugs that were "only" showing to users in browser
function LoginButtonClick() {
var selAccount = $("#accounts").val();
LoginCookieSet(selAccount);
if (typeof ga !== 'undefined')
ga('techcu.send', 'event', 'button', 'Click', 'Member Login');
var formAction = "https://online.techcu.com/User/AccessSignin/Password"; // testing url
switch (selAccount) {
case "1":
formAction = "https://online.techcu.com/User/AccessSignin/Password";
if (window.location.host === "dev.techcu.com" || window.location.host === "qa.techcu.com") {
formAction = "https://onlinetest.techcu.com/User/AccessSignin/Password";
}
break;
case "3":
formAction = "https://businessbanking.techcu.com/";
break;
case "2":
formAction = "https://businessbanking.techcu.com/smallbusiness";
break;
default:
formAction = "http://online.techcu.com/User/AccessSignin/Username";
}
if ($('#UsernameField1').val().substr(0, 2) == "**") {
$('#onlineBankingLogin #UsernameField').val($('#UserNameHidden').val());
} else {
$('#onlineBankingLogin #UsernameField').val($('#UsernameField1').val());
}
$('#onlineBankingLogin #PasswordField').val($('#PasswordField1').val());
$('#onlineBankingLogin').attr('action', formAction);
$('#onlineBankingLogin').submit();
}
Pretty sure its caused by "Chrome-only" developers, as going through the hassle of installing a Chrome/webkit based browser gets things working. But really, fuck that. ;)
Ubereats made this change recently. Naturally, there's no way on their website to contact them about it. :/
Demanding the same for less is haggling. As if we can magic time reductions out of our asses without cutting features .
Of course on any project I might be able to deliver the same in less time, but then at the expense of something else. That might be acceptable if the boss thinks he can manage the other clients which work gets delayed.
Their actions are not far from criminal negligence IMHO.
I was a Solbank (one of their brands) customer and can safely say tech isn’t their strong point. Awful UX and a pain to deal with.
It's the core banking system of a big bank. Handling consistent state between the old and new systems while progressively migrating customers would probably have been extremely complex. They also get to have a maintenance window (few people will complain if they get warned their bank and all transactions won't work for 2 hours in the middle of the night on Monday). A "big bang" migration makes more sense, if everything is properly prepared and tested, which it wasn't.
Part of your preparation and testing is the rollback of a partial migration - if you're irreversibly committed to the "big bang" before you know its outcome, then your preparation and testing has failed.
That sounds like ‘A “big bang” migration makes more sense, if it works.’
Your rollback plan should never be an afterthought. Your rollback plan should be designed like you expect to use it. If your rollback plan is “Burn the ships” and fix-forward, you shouldn’t be working at a bank.