Improved fraud prevention with Radar 2.0(stripe.com) |
Improved fraud prevention with Radar 2.0(stripe.com) |
We've been hurt badly as a startup breaking into the US market and getting many of our genuine charges blocked by Radar (and at a "highest risk" level where it is not possible to disable rules).
As a developer, I had the best possible impression of Stripe, as they provide easily the cleanest API and best documentation of any payment provider I've used.
From a business side, it was highly frustrating losing so many of our early US sales. A lot of this was due to US banks blocking the payment, but Stripe did not handle these cases well in our case.
* Error messages presented to the customer are often ambiguous or misleading, and if you're using checkout.js you cannot customize these easily. In some cases, the customer gets a client-side "Card is declined" error, with no communication passed through to the merchant at all, which makes telephone support very difficult.
* Stripe will not provide any kind of telephone or chat support to merchants. By the time a specific blocked payment can be escalated to a "fraud specialist" the customer is usually long gone.
* Stripe will not allow you disable certain Radar rules.
* If a user tries to pay several times (due to their bank blocking the payment as suspicious), and then phones their bank to clear the payment, Stripe will often then block the payment due to "repeated attempted uses of the card". This is highly frustrating as even very determined customers who try several times and then contact their bank to resolve the issue there still get blocked by Stripe, and usually give up at that point.
* We had one payment blocked by Radar due to it being from a "high risk location" (Pakistan if I remember correctly). This represented unacceptable levels of discrimination for us. Machine learning and probability is useful, but ethically it's hugely problematic to deny people service based on their country, race, gender, age, or other attributes that they do not have control over.
My early experiences with PayPal were nothing short of terrible, but BrainTree is looking like a more and more attractive alternative to Stripe, especially with the PayPal integration built in -- if people have issues with their Credit Card they can simply pay with PayPal credit instead.
This, to me, represents the worst that banking fraud protection has to offer. Just yesterday I (from the USA) tried to purchase a software license for a tool I've been using the free version of for a long time. My card was declined, so I used my American Express. About two hours later, I got a call from my bank's fraud department saying they had blocked a transaction to the UK for fraud prevention, and disabled my card to protect me. Apparently the bank (a small US-based bank) block any transaction in the UK as it's a "high risk country"... I'm sorry, but this is the Internet. No one cares where the company is located, and I have no way of knowing beforehand that the payment is processed by Stripe US or Stripe UK. Blocking entire countries for fraud prevention is a really lazy way of doing fraud prevention.
But I've even seen worse at another bank. My area of the US doesn't have Publix grocery stores. Apparently this bank considered shopping at Publix to be unacceptable risk when I was traveling for work, and disabled my debit card because of this. Stopping at Walgreens beforehand and getting dinner the night before wasn't suspicious though.
Bank fraud is a hard problem, and taking lazy solutions doesn't solve that problem. It just hurts customers and hurts businesses.
I had it down to a science, I knew the direct number to their fraud dept and I knew when I should place my call so that I'd usually be connected at just the right time to get the charge authorized with enough time to avoid the website session from timing out, although sometimes I'd have to start over. Eventually, air canada added a timeout popup which helped prevent this.
I tried everything to get BoA to fix this (escalating calls, writing letters, etc). By the end, I gave up and just accepted it when they "put a note" on my account so this "would never happen again". This went on for over 2 years. Thankfully my company switched our cards to another bank and I never had this problem again.
- We give users the ability to disable the default rules and mark transactions as safe if you write in to support@stripe.com (so in the example you gave of a user clearing transactions with his or her bank, if Radar incorrectly blocked a subsequent payment because of high card velocity, you could mark it as safe and subsequent payments would be allowed).
- Radar surfaces the primary reason a transaction is believed to be high-risk, but that is never the only reason (so the primary reason might be that the IP is in country X, but that doesn’t mean there’s a blanket ban on X—just that that reason combined with everything else we saw across thousands of signals resulted in our giving the payment a high score). It didn’t quite make it in under the wire for today’s launch, but we’re working on making the explanations more detailed.
We believe that Radar 2.0 (and in particular Radar for Fraud Teams) should also be helpful here:
- With Radar for Fraud Teams, you can customize the threshold at which Radar blocks charges—so if false positives are very costly for your business (because you have large margins, e.g.), you can tune Radar to reflect that; you can also specify lists of trusted payment attributes to “allow” (if you have known good card numbers, e-mail addresses, IPs, etc.)
- Radar 2.0’s custom machine-learning models (for businesses that have enough data with Stripe) should adapt to the unique circumstances/patterns/trends of your business, and
- Radar 2.0’s ML overall has substantially improved performance, which you should see after you’ve upgraded.
It might not be that one reason, but could it be something like the following two? Because that would be practically equivalent in terms of unacceptability:
1. Customer IP is in a high risk location, and
2a. Vendor rarely does business successfully with people in that location, OR
2b. Vendors around this region don't usually do business successfully with customers in that region.
("Does business successfully" here was meant to both encompass lack of transactions as well as lots of unsuccessful transactions; I'm asking about both possibilities.)
Large percentage of support calls start with "my bank blocked the transaction because it was international / fraud"
Incorporating in the US is not a feasible option. This is where Stripe's "Global" reach and 150 currencies loses a bit of power.
Would be great if there was a way non-US stripe merchants could bill us customers as if they were in the US. Sure it is more of a regulatory/legal issue than a technical one.
Given the kind of company that we are, we lose way more from false-positives than from true-positives. With this system in place, Stripe is punishing legit users and businesses, with no way to change the default behavior.
I enjoy using Stripe as the next guy, but this needs to radically improve. At least provide me as a merchant a button to whitelist / accept a customer and override your rules.
To be fair, Stripe is excellent for developers. For businesses? Not so much. You're either stuck with a monopoly, with shitty documentation and archaic APIs (PayPal) but works well for businesses or you're stuck with a new age (Startup?) that's really great at everything except helping you make money.
Here's me hoping for a better Stripe alternative. There's really a lot of space in this market.
I was lucky that the particular merchant was a friend, so I could personally reach out and see his side of the accounts and submit it as evidence to my bank, but I didn't think for a second that I could trust Stripe to handle that despite their involvement.
Practically though I worked for a company that saw hundreds of purchases from the Vietnamese IP space of which only two were not fraud. For the rest of the world it was in inverse.
It's a moderate pain in cases like Uber, but usually we have very few chargeback issues .
I'm not a big proponent of bitcoin etc, really, there sure are big downsides. But this is the big upside: the credit card system is "good enough" for average Americans and average retail chains, but breaks down all the time with international and/or indie stuff. (See also: random people banned from using Square for life, because the fraud detection algorithm can't handle them.)
I (and the entire Radar team) are on hand to answer any questions you may have!
Can the new system now distinguish between the initial charge to start a subscription and subsequent ones? A recurring cause of those false positives is someone moving house after they've subscribed and not telling us, and thus failing a mandatory address verification when their next charge goes through. The concept of pre-approved and automatically-blocked lists seems like it might help with that problem, but it would be better still if we could just define the rules more flexibly in the first place so the address-related checks are done the first time but if they've passed previously then we're not too concerned if they start failing later.
- Any idea what the total fraud vs genuine transactions ratio is? And how that breaks down across industries? I am assuming that SaaS services don't get as much of this - i mean would people buy bingo cards with a stolen card?
- how does fraud get monetised? Once i have downloaded my millions of credit card numbers from Tor (or stolen my friends mothers wallet) I need to persuade a merchant to deliver me something - but it's always bugged me that they actually have to deliver it. to a physical address. that can presumably be traced. It all seems very low level
(Quick story, years ago, call it the year 2000, was in the UK version of BestBuy and the manager called up a service to verify that this 17 year old kid could have a laptop. The manager asked what's your name ? Ok Your address ? Ok. Date of Birth? March 1954? really? you look a bit young for fifty. It just seems a poor way to commit crime)
The question i am trying to ask is that turning credit card numbers into cash seems like a grind that farmville would be impressed by? is it just lots of low level grunts in shops and online or is there something i am missing?
- what advantages do you get as a payment processor that a merchant does not have? And how is that better / worse than the card provider? I would assume there are people trying the same card in multiple different stores at the same time, so if you spot one attempt you stop them all, whereas individual merchants could not know. But Visa probably spots that i just paid for goods on two continents which you can only spot of both merchants use you? Do you and visa share data or do sequential checks and the like?
- how much do the "obvious" checks help - highly unlikely purchases (3 iphones) timing or physical activity (I probably won't buy books on amazon, clothes on boohoo and petrol in a garage in the same five minutes) versus the more ML / secret squirrel stuff?
cheers
This is a big win for Stripe. Sift is bullying customers into a minimum of 1,000 USD regardless of previous volume or history.
We were in the 10k orders per month around 250 USD with Sift. Now get kicked in the teeth with the minimum 1,000 USD and no real additional value. Not cool.
You know what's cool? Stripe's decision to not charge anyone who was a Stripe Radar user before today. Kudos.
It must be said that Sift does not only apply to Credit Card fraud and has many verticals (content abuse, ecomm, etc), but I think the big stuff is definitely on the chargeback and fraud prevention.
I'd love to move to Stripe Radar, but that doesn't really help with PayPal orders.
Now obviously Signifyd scrub, but we haven't seen a case where we've had clean transactions scrubbed, and when a CB goes through, you're refunded.
Likewise, you no longer have to worry about your CB% going over and you getting blacklisted for life.
Until Stripe steps up and starts providing chargeback protection, their protection is simply lipservice.
- With Radar for Fraud Teams, you can customize the threshold at which Radar blocks charges—so if false positives are very costly for your business (because you have large margins, e.g.), you can tune Radar to reflect that,
- Radar 2.0’s custom machine-learning models (for businesses that have enough data with Stripe) should adapt to the unique circumstances/patterns/trends of your business, and
- Radar 2.0’s ML overall has substantially improved performance, which you should see after you’ve upgraded.
Everything about them is beautiful, simplified and easy - except Radar.
For all the blogs and good intentions of the team, Radar to me means an email 3 days after I've processed and shipped an order telling me it may be fraudulent, followed a month later by a reminder "I'm sorry, you lost your credit dispute for $X" email, followed by a $15 charge by Stripe for no error or bad faith on the part of my company.
When Radar isn't busy sending me reminders that we're having money stolen from us, they are hard at work denying legit charges and sending customers down a Kafka-esque rabbit hole, hell bent on seeing exactly how much friction can be introduced into our website's buying process by a single third party service.
Stripe didn't create the fraud and they are taking on a difficult and emotional part of their business, which is commendable, but it must be said:
1. The false positive rate of Radar is so bad that it renders the product worse than useless - worse because it initially provides a false hope.
2. You can't disable some parts of Radar. Better to throw the whole thing into the sea and use a plugin then be forced into some parts of this.
I know there are good people trying hard to build this product well. Some of the people on it serviced our account in the early days. They had a vision of a better way and they made it real. My hats off to them. Now, people I respect, I have hard words and a hard truth to impart to you:
You are not delivering the value you claim to provide. Do not continue iterating. Discontinue the product. Allow others who focus on this to do it well. You are great at what you do but you are frustratingly, annoyingly, arrogantly bad at this. It pisses us all off to be forced to use it and to be told again and again how great it is going to be or how fixed it is this time. I don't want to engage with Stripe on Radar or "learn more about it", I don't want to be interviewed by you so you can better understand the voice of customer, there is no email or back channel thing you can send to make this better. The beauty of Stripe is that it "just works" but Radar does not just work - and it never will.
For the benefit of other HNers: if you ever have a concern about this sort of issue, we’d love to hear from you (my email is eeke@stripe.com). This is all I do every day; you can never waste my time.
The other point I bring about is rather rhetorical - There are no open standards, model baselines and datasets in the Fraud domain. Compare building a model for fraud detection to building a model for image recognition or object detection There is a standard baseline, standard datasets and your model competes against that baseline. Because of the open nature of image recognition, the models have improved astronomically. I feel that a lack of such openness is fraud is holding back on innovation. I could be wrong in this assessment so please correct me if so.
(And on class imbalance: we spent quite a bit of time experimenting/analyzing how to deal with it—we found that sampling rate has a marginal impact on performance but not a huge one.)
Is there a way to tell when this happens in the Dashboard? There wasn't at the time, but I'm hoping this is maybe now visible somehow? It's obviously helpful to know when trying to help a customer resolve the situation.
This means that if your business model involves "try before you buy" or usage-based billing, you'd better be sure to make an initial charge, otherwise the customer might incur costs before Radar decides to block the charges.
Even if you do require an initial charge, if you allow customers to change their credit card between recurring charges, the new card could be extra risky and "fly under the Radar" until the first charge attempt.
Are there any plans to offer fraud risk and blocking when attaching a card to a customer, or will still be limited to just blocking charges? With Stripe's new emphasis on recurring billing, it seems like this would be important.
We currently see Radar as a liability for us. It might block the occasional fraud and avoid a chargeback, but it also allows customers to incur costs with dodgy cards before we know they're dodgy, and then blocks charges outright before we know.
In software sold on a free-trial model, you assume most trials don’t convert (overwhelmingly due to declining to pay but with a bit of fraud) and then the cost to provision the service (COGS) is, effectively, a marketing expense. COGS in SaaS are typically negligible to low; this is why the industry is OK with providing services on, basically, a digital handshake. If you want to allow users to try out high-COGS services (or highly-abused services) prior to verifying capacity/willingness to pay, you’d need some way to credit score potential customers outside the context of a particular payment.
To date, we’ve generally focused the bulk of our ML efforts on things which apply to the majority of our users, but as we get better at customizing these technologies to specific industries at scale and even on a per-account basis, we could certainly imagine applying them in contexts that are more relevant in your model. I’d love to hear more detail about your use case; feel free to email me (my HN username at stripe.com). If we get closer to shipping something that is probably interesting, we’d be happy to give you a heads up.
- compute a huge number of features, many of which are quite complex (involving data collected from throughout the payment process), in real-time: e.g. how many distinct IP addresses have we seen this card from over its entire history on Stripe, how many distinct cards have we seen from the IP address over its history, and do payments from this card usually come from this IP address?
- train custom models for all Stripe users who have enough data to make this feasible, necessitating the ability to train large numbers of models in parallel,
- provide human-readable explanations as to why we think a payment has the score that it does (which involves building simpler “explanation models”—which are themselves machine learning models—on top of the core fraud models),
- surface model performance and history in the Radar dashboard,
- allow users to customize the risk score thresholds at which we action payments in Radar for Fraud teams,
- and so forth.
We found that getting everything exactly right on the data-ML-product interactions necessitated our building most of the stack ourselves.
That said, we do use a number of open source tools—we use TensorFlow and pytorch for our deep learning work, xgboost for training boosted trees, and Scalding and Hadoop for our core data processing, among others.
It would be pretty useful information to have, since you're collecting it already.
Edit: no polemic intended
The problem is said people can do so much damage to a profit margin as they tend to hit an online store hard and quickly, racking up huge potential reversal costs. For us, being in a specialized digital space makes this especially painful as the digital items can be "used" and no longer resell-able or recoverable, so inventory and real dollar value is lost.
Visa / MC has no real incentive to stop the fraud as the merchant in most cases is liable for the reversal - Visa / MC is just a facilitator that bends towards keeping the buyer happy (same with PayPal). 3DSecure was introduced over a decade ago to alleviate some fraud based on unauthorized \ unknown but uptake has been anemic due to poor buyer experience.
As for converting stolen card to profit, purchasing and delivering high ticket physical items online and reselling is one known method.
I will read the rest of your post after I get some dry pants on.
(Disclaimer: I work for a competitor to Stripe Radar)
This is actually really interesting. An important thing to remember about fraudsters is that they're mostly professionals. Every day they wake up thinking "how do I get around anti-fraud systems." Many are located in jurisdictions that have poor enforcement for cyber crimes, so they're not necessarily worried about official action. However you're right that they do actually need to get the goods shipped without too many questions.
Two common strategies for this:
* You know all those "work from home for $100/hour" ads you see? Some are run by fraudsters who use those people as re-shippers. I.e., the website ships to some guy in the US, and that guy reships the good to the fraudster in Eastern Europe for a cut of the profit. If the fraudsters build up a nationwide network of reshippers, they'd be able to find one who lives close to the billing address of the card they're using.
* There's an even cleverer scam that goes like this: the fraudster creates a merchant account on Ebay or similar. They then select high-value goods available on other websites, say BestBuy, and list them for sale at a substantial discount. Then, when an unwitting customer buys the good from their Ebay store the fraudster places an order from BestBuy using a stolen credit card and has it shipped to the buyer. They get the money from the buyer, the buyer gets the goods, and nobody's the wiser until the chargeback comes in to BestBuy a month later.
In the early days of ecommerce, we jokingly called it 'Toners for Taliban'. They would purchase goods with a stolen card from a company that ships fast. They'd have the item shipped to a rube who answered a "Make money fast! All you need is a computer and a mail box!" advertisement. Then, the rube would resell the item on Ebay and send a cut back. The rube takes the fall, if any.
Detecting this fraud was pretty obvious when you actually had a chance to look. Someone with a billing address in Florida is shipping a video projector to an address in Arizona, from an IP address in Austria? And they always fill out the forms in ALL CAPS? And that same IP has placed orders on a dozen other accounts?
The problem back then was that humans didn't have the time to do that exercise, and ML wasn't up to snuff, yet. Things are obviously better, now. However, at the same time, the fraudsters are probably smarter, too.
Today I learnt ... :-)
ML solutions very often are just learning to codify these 'obvious' scenarios, and as a bonus sometimes less obvious ones.
You could sit in meetings for hours/days listing all the cases the fraud detection should catch, and you'll still end up missing lots. If you're stripe, you have tons of data about fraudulent purchases that you can use to learn and codify these scenarios. Importantly, you can learn which scenarios are most prevalent in your system in particular, fraud at Stripe is very likely to look different than fraud at (say) Wells Fargo.
Easiest way is to ship stuff to some house, then grab the package off the front porch. Best if the house isn't actually inhabited. Yeah, there's some evidence left behind, but it's usually a dead-end.
Once you get your hands on someone else's card info, you bust out by buying as many prepaid Visa cards and store gift cards as their limit allows before the card gets shut down. Then you use them or fence them for cash by selling them on Craigslist.
A little more color: Stripe’s incentives are aligned with those of our users in that we want to let through as many legitimate customers as possible. We keep a very close eye on false positives.
Our machine learning models examine thousands of attributes for each payment and make predictions based on how frequently the observed attributes were associated with fraud in the past. The comparison here is never just on one or two or three attributes (as in your example), and no logic is hard-coded.
https://www.youtube.com/watch?v=4inIBmY8dQI&feature=youtu.be
Regarding Paypal, I believe the fraud that hurts the most is credit card (chargebacks, threshold maximums and card programs!), and Paypal in general is not that bad at that (not as bad as other things).
If Sift sticks with their new pricing, I think it comes down to doing some math.
Our Current Sift Scenario: 3,000 USD a year.
Proposed Sift Scenario: 12,000 USD a year.
Potential Stripe Radar Only Scenario: 0.
Assuming Stripe is as good or possibly better than Sift at catching fraud/automating flows, then we've got 3 to 12k to offset any potential PayPal fraud and still come out on top.
Ok, Not proper math, but worth considering...
Edit: formatting.
Definitely something I'm considering.
But only manually and retrospectively, no? For an online subscription business with extensive automation, if one of us has to go to the dashboard to fix something like this, that sale was probably already lost anyway. More likely, that failed charge already resulted in cancelling the subscription and all that follows, with a relatively low chance the subscriber will start again later.
From various recent HN discussions, I get the feeling that the team at Stripe totally fail to understand how devastating these false positives are to the growth of an online subscription business. For example, I have one business where it's not unusual for the majority of the entire churn in a month to be due to unexplained card payment failures with Stripe. Based on a back-of-envelope calculation, that business would be around 50-100% bigger today if those charges had worked and everything else followed the usual patterns.
It’s possible you’re using our Billing product, which gives subscription logic out of the box. It’s also possible you’re not, and have rolled your own subscriptions.
If you’re using Billing:
We take into account that it’s a repeating charge and special-case that for fraud detection purposes, because it’s highly unlikely that a happy customer decides to become a credit card fraudster N months in. We can also do dunning (automated recovery of subscriptions) via email, so a single charge failure shouldn’t result in you losing the relationship.
If you’re not using Billing: we have a way to mark a charge as safe when you present it via the API, which you could do on e.g. established customers. (On the charge creation request, pass in {"fraud_details" : {"user_report" : "safe"}} or whatever the equivalent is in your language of choice.) You can do this for any charge where your business has prior knowledge regarding the transaction; we'll both a) respect the decision and b) update the fraud model accordingly. Note that your customer's bank is the ultimate authority on whether a charge goes through; they may decide to decline it for a variety of reasons even if you and Stripe believe the charge to likely be good.
You have a business decision to make regarding what you want to happen in the event that all automated systems fail to get a user back to a happily paying state. One option is to automatically cancel the subscription. I recommend you don’t do that in a subscriptions business, unless you have very, very tight margins (e.g. physical product not SaaS). We expose an option in our Billing product to keep the subscription active even if it isn’t being paid; we’d probably recommend you use that option and only cancel subscriptions on an explicit user request. This is a particularly important thing in B2B SaaS, where price points generally give you enough of a budget to get in contact with substantially everyone who churns and where it’s very easy for a single person associated with an account to e.g. ignore a few emails about it accidentally without having the intention to cancel.
Alas, a human also saw the transaction, failed to read the note or look in the history and locked the card up anyway.
We don't see this problem with banks in any other country (not do I ever have the problem in reverse, buying goods and services from foreign websites with my Australian credit card).
For what it is worth, Fastmail has been as good as they can be with this issue, but they're at the mercy of my bank as much as I am.
While we can’t fix the previous experience you had, we’ve rebuilt almost every component of our fraud detection stack over the past year. We’ve added hundreds of new signals to improve accuracy, each payment is now scored using thousands of signals, and we retrain models every day.
We hope these improvements will help. We want our customers to be able to provide you services; that’s what keeps the lights on here. We’d be happy to look into what happened if you have specific websites in mind—feel free to shoot me a note at mlm@stripe.com.
It's a little tough to talk about this in an HN comment but please feel free to shoot me an e-mail (mlm@stripe.com) if you'd like to talk more.
I say this as an American who has never lived outside the US but who deals with international clients regularly: the US seems uniquely inclined (in my experience) to think that everything they need falls within their borders, and everything outside their borders should be treated with suspicion. I've never had a German client want to block all traffic from South Africa. That's just an observation, I make no judgements as to why that is.
I did have an American university for a client who said "we cannot block or otherwise discriminate traffic from anywhere, since we have students or staff in every country outside of North Korea" which is a refreshing outlook IMO.
FWIW, I paid with AMEX (using Stripe not PayPal) and it went through right away.