Show HN: Check if your IP has appeared in a residential proxy network(haveibeenproxied.com) |
Show HN: Check if your IP has appeared in a residential proxy network(haveibeenproxied.com) |
In any case, residential proxies are a godsend. Because most of the everyday services like web shops, govt information portals, even personal blogs sometimes are blocking access usings captchas.
EDIT: they write this on their marketing copy [0]:
> The Spur platform identifies traffic originating from residential proxy networks by analyzing service fingerprints, ASN ownership, and behavioral indicators.
Sounds like guesswork that results in a ton of false positives. And the more innocent IPs are blocked by platforms on the basis of this data, the more the demand increases for residential proxies, from users who need access to essential services. Talk of creating the problem and then selling the "solution".
0: https://synthient.com/context/ip/
1: https://www.wsj.com/tech/kimwolf-hack-residential-proxy-netw... / https://archive.ph/SpKVn
We had a previous discussion about surfacing visitor IP address resproxy status explicitly. Should we have some sort of badge or a more explicit alert to show if a site visitor's IP address is part of a residential proxy network?
Even though it is great for demonstrating the product's value, it is kind of a low-tier value. What can a user actually do when they realize their IP address is part of a residential proxy pool?
The first issue is that residential proxy SDK infiltration is massive. If you start connecting to different IP addresses and constantly check your IP address on our website, you will often see that many of those IP addresses were, at some point, part of a residential proxy pool. We provide frequency information showing how many times an IP address was observed in a residential proxy pool, with a default observation period of 7 days.
Then there is the question of what a user can actually do about it. If it is a controlled IT environment with paranoid IT admins, sure, they can actively monitor traffic and identify why their IPs are showing up in residential proxy pools. They can attempt to do something about it. But it is not easy even then.
Residential proxy SDKs can simply be baked into almost any smartphone or smartphone-derived OS that allows app installation through marketplaces. So, many residential networks are already cooked (because of android TVs). Moderate-scale NAT connections almost always see residential proxy flags, as do public Wi-Fi hotspot IPs, which we also detect.
Identifying the apps that are generating background network traffic is quite hard. You need some level of DNS monitoring or a network sniffer. Alternatively, you need router-level firewall software.
These SDKs are not always sending high-volume, constant traffic that makes them easy to detect. If you see a 100% residential proxy flag for your IP address, then they probably are. But in many cases, the traffic is intermittent and much harder to identify.
Nobody has an answer to what I should do when I see my IP address in a residential proxy pool. It has been accepted in spirit as a "consented malware" for the last few years. It is undetectable and extremely hard to remove because the SDK has been baked into apps themselves.
But it got flagged/downvoted into removal (twice!). Someone here has lots of HN accounts and doesn't tolerate free competition.
Disclaimer: I'm the founder and main researcher of the "flagged" company.
---
On the broader point, we process 3 trillion requests last year, have more than 80 employees, and run a dedicated privacy engineering team led by an ex-cybersecurity company founder. We invest in research on new detection methods and stay closely engaged with the developer community. If there are specific gaps you see in our product, I would be glad to hear them and discuss.
---
Whether any dataset, including residential proxy IP data, is valuable depends heavily on the application. Treating a dataset as invalid because it does not fit one particular model can lead to decisions on shaky ground.
We are regarded as one of the more if not the most accurate IP geolocation providers, and we spend considerable effort on education, solutions architecture, and documentation so customers understand what our data can and cannot support. For example, IP geolocation, even at highest level of accuracy, is not a person identifier. It will never be a 1:1 replacement of GPS geolocation.
Many of the largest companies in AI, anti-bot, fingerprinting, KYC, and CDN spaces use our data. If anti-bot systems and CAPTCHAs were fully reliable on their own, there would be less need for additional signals like residential proxy data. We do not assign a score or label an IP as good or bad. That judgment sits with the customer's own threat or analytics model.
Residential proxy IPs are, by and large, mostly used in web scraping operations of many different forms. If a company sees a moderate to high amount of traffic mimicking human behavior, it can struggle to tell bot traffic apart from real users. Anti-bot mechanisms can help, but they add friction to the user experience, and they are not cheap to run at scale.
Residential proxy detection data is one of the easiest zero-knowledge ways to gather intelligence. There is no need for users to solve a puzzle or for multi-page traversal to collect fingerprint data. All that is needed is the IP address.
Our residential proxy data customers tend to be on the more sophisticated side of cybersecurity. Suggesting that this data is a silver bullet for all their security needs would not reflect well on their expertise or ours. We present the data as is, and from there we work with customers on the right solution for their case.
If there were people under the same CGNAT with someone running one of those exit nodes, then that's not a false positive.
Hidden from the user and provided by the ISP
I've seen this (and verified with others) with other sites like iknowwhatyoudownload.com where they allege your connection downloaded something very illegal (like CSAM) even though you know for certain it never happened and you haven't been hacked.
IIRC, they only show the filename, last-seen timestamp and user agent. I can't verify it though since their website seems to be down?
Being able to check is helpful.
Flagged and then... What exactly?
If people who have a smart TV have their smart TV participate in a residential proxy, then it's billions of IP getting "flagged".
What's the use of flagging those?
When every IP is flagged, none is.
Although I'll grant that it would be interesting to know if your devices are running the proxy, but you'll need an exeuctable tool for that, not a per-ip network tool.
Lack of any IoCs also makes it hard to refute or remedy. Plus I think it paints even Tor relay nodes with the same brush as malicious proxies.
Truly kafkaesque if you start getting restricted and nobody tells you why or even knows what to tell you because the sources have all been mixed and obscured.
More on why here: https://spur.us/blog/i-dont-like-big-gateways-and-i-cannot-l...
Some are quite good at limiting connections, monitoring devices, and contacting students for suspicious traffic.
(cyber consultant, have people who might use this for enrichment in security stacks)
Some info on methodology would be necessary for a paid plan for two purposes, one to audit that the quality of the methodology and the signal is good (it's not hallucinated or checking few sources).
But also to make sure it doesn't clash with other signals if used in conjunction with other sources of tool (if I have another signal, I want to know whether they are redundant or complementary, to avoid interpreting two positives as independently verified.
There's a non-trivial quantity error here that makes the argument of a majority qualitatively incorrect.
It's not billions of IPs that are being used in residential proxies, it's not all smart TVs.
There needs to be a vulnerability and hacked devices OR there needs to be a very low quality and shady vendor that is offering products at too cheap prices and needs to make ends meet in order to compete at that price. Probably chinese.
This would be in the range of 1 to 100 million smart TVs. sorry for the wide range, but definitely not 1Billion or every TV.
So to the extent that the ratio of infected to non infected IPs is low, then providers can block the infected ones to a great effect.
But for now I'm already cut off from half the internet due to endless crimeflare captcha loops... I just don't visit those sites anymore because I literally can't.