They are basically 'won't someone please think of the children?', but with higher stakes.
When it comes to involunary proxies, those are really just one of many possible symptoms stemming from a real problem: Shitty security.
Shitty security is tolerated by our markets, is is protected from fixes due to copyright law, and it is even encouraged by parts of our government that want to exploit the flaws.
It strongly depends on how scraped data is being used.
If your 'cat' hasn't been able to catch their 'mouse' then the cat needs to get smarter, or look for alternative sources of mice, or the cat should be considered 'unviable'.
Have you approached them to get access to their data? Have you explained to them how your service can benefit their business?
(I generally come from a position of suspicion as to why someone wants to scrape data that the owner goes to certain lengths to protect, but then I'm also an 'information wants to be free' kinda person, but the Internet is increasingly an untrustworthy place, so security is overruling narrative).
Let's try this for example. Suppose you want to create a price comparison site.
A lot of the major retailers don't want these, because they want the customer going to their site when they want to buy something, not to the price comparison site that tells the customer which retailer has the best price and then might not always be them.
If the comparison site doesn't have the prices from retailers who sometimes have the best price then it can't serve its function -- customers still have to check sites manually. And comparison sites are pro-consumer whereas blocking them is anti-competitive, so the ones doing the scraping are the good guys.
Do you allow the data you've scraped to be scraped? Do you share it as freely as you desire the 'companies that want to hide it' would? Or do you consider the scraped data is 'hard earned reward for effort' and therefore has value that others should subscribe to your service for?
Specific Domains:
proxyjs.brdtnet.com
proxyjs.luminatinet.com
proxyjs.bright-sdk.com
clientsdk.bright-sdk.com
clientsdk.brdtnet.com
*.brdtnet.com
*.luminatinet.com
*.luminati.io
The overall residential proxy market is too large and often undetected by intelligence tools. BrightData is actually much more compliant and malicious actors wouldn't be allowed access. They have an extensive KYC and use-case vetting process.
Sometimes their system malfunctions (cough cough) and you get charged for those clicks but you fail to report them as fraudulent as you've no proof as the IPs and useragent all appear normal to ad agencies and advertisers.
> Actual data and identity loss of American citizens.
> Malware that can record video and audio from infected devices.
> Infrastructure for foreign covert influence campaigns.
> Botnets used in hacking and DoS attacks
These things have existed for 20+ years. Bad but not exactly a national security threat.
People were dismissive of security far too long. I think it's actually good that AI instills some fear into people causing it to take a lot more seriously than they have previously done so.
More importantly, a residential proxy can be mutual. If a group of people agree to forward traffic to each others’ IP, how is it not their business? It obfuscates their identities, but this is the Internet, not e.g. a government office or test center where you obviously can’t walk in with someone else’s ID.
Banning non-consensual proxies makes sense since those are effectively malware, even though they make anonymity harder, so does banning theft and here you’re stealing someone’s internet. I’m sure we can convince enough laymen to knowingly install proxies by paying them.
Myself being "not a layman" would definitely not allow anonymous usage of Internet that's tied to my home/residence/identity to be used by some internet rando just because they pay me money.
I would support, however, small family/friends groups who know and trust each other well enough to not do anything that'll cause a police raid on each others homes (having endured a police raid and the ensuing 8 months of not being told anything about the progress of whatever they're doing, it's not something I'd wish on a stranger, never mind a friend/family member).
Proxying is least of the worries!
When manufacturers bring out the smart features that require constant data collection, monitoring (obviously in the name of service quality, troubleshooting, firmware / app updates and subscription of services (!!)), it is oblivious to the fact that security and privacy at a personal / state / national level is never considered (for various reasons such as the design, profit interests and overheads / compliance perspectives and what not!) to be part of the ecosystem. Bad actors with sufficient knowledge and tools could exploit and profit from it.
It has become more of a planned obsolescence / profit / greed based industrial / corporate culture in rolling out unwanted stuff / monitoring and controlling as a feature that gets exploited to the core.
Honesntly, I don't have an answer for that, but have some thoughts that I wanted to share.
I do not want a cleaning robot or a water purifier or a printer or a TV or a refrigerator or an oven or a smart light bulb or a smart lock or even my car to have undisclosed, unwarranted connectivity to internet for whatever reasons.
I do not care if it is the manufacture or the service provider or whomsoever it may be!
I want all of the network connectivity options to be explained with all the controlling options and boundaries and data collection that happens on any of the connected medium to be as much transparent with the option of preventing or restricting what one does not want to go out of the device.
Will anyone do that?
It ill never happen! Sadly!
This problem will balloon further without adequate controls and killing the networking at a ground level would only be the only solution!
But, in the case of a connected device having an inbuilt connectivity option (like the m2m based options) in a smart vehicle, even that is not possible!
The nation's security infra is a reflection of the nation's security legislation and regulation.
Aside: TheSInIoT is my DMZ's wifi password.
The interviewee seems pretty inept yet still managed to uncover something really interesting and nefarious.
> At the very least, major American ISPs (Comcast, AT&T) should detect clearly suspicious activity coming from customer IPs and warn them to scan their computers, check their TV apps, and find whatever is turning their internet into a proxy.
What are we saying? Really, what are we saying? We should turn domestic ISPs into domestic surveillance apparatuses to "detect clearly suspicious activity"? What is "clearly suspicious activity"? Section 230 is still law.
Governments could even purchase residential proxies themselves and check whether they're being used for malicious activity.
It isn't that difficult, but I guess governments just aren't that interested.
> It isn't that difficult, but I guess governments just aren't that interested.
I recently switched from my previous ISP because they randomly broke my ability to use my own router, and during the time when I had to default to using their own modem/router hardware before I could get the new ISP to come and set things up, I could barely even get a signal in my office upstairs (which had previously been connected via a mesh endpoint) because their router didn't expose any way for me to split 5 GHz and 2.4 GHz, and the router absolutely refused to let my devices connect via 2.4 GHz despite them having more than 90% packet loss due to the weak 5 GHz signal.
Regardless of how "easy" it is, I don't trust ISPs not to screw it up somehow and probably cause a lot more concrete damage (even if the individual issues they cause are smaller in magnitude) than the theoretical concerns of "national security" that, as far as I can tell from reading this thread, have caused a total of like a few hours of downtime one day in a couple decades.
Continuing to allow CGNAT is what allows the residential proxies to hide because it tumbles the IP identity of bad actors with normal people.
No one is being hurt by someone sharing their internet with me a few times a week.
Just because the idea of net neutrality exists doesn't mean it's true. I'm sure there's a specific context for it, and 'security' is not that context. It was about data/packet prioritisation wasn't it? Unrelated to security.
Yes, yes, we should fight deceptive software that doesn't disclose it has a proxy function. Fine. But there's no way to square 1) fighting all residential proxies, including ones run with informed consent and 2) preserving the public's access to general-purpose compute at home.
I'd rather live with the proxies than for someone to tell me that if I run squid, I'm damaging national security.
The I guess the site is either useless, or can be used in combination with customer's own research that includes sites not included in the comparison. The site is still useful, it's just not exhaustive. There's nothing much in the world that's exhaustive, it's all a range of percentages.
Also, maybe the source that isn't scrape-able becomes less popular as a result of not being included in the price comparison list. If they're always more expensive, then neither you nor they have any advantage in listing on your site. If they're always cheaper, then your site may have no purpose to serve.
Everyone seems to want _everything_ despite _everything_ not being necessary to provide a service. The lack of a certain set of data may itself be a data point, and should be used as marketing material for or against the company resisting the scraping.
Just because residential proxies are somewhat of an 'easy answer' doesn't mean they're right. Use more creativity and imagination! Or follow a new idea. It's not like price comparison sites are a revolutionary idea.
(I'm a consumer who happily does significant research before a buying decision, going to various sites and checking prices, warranties, model numbers, reviews, availability, delivery times, and all the shit. And prefer doing the research myself than trusting a price comparison site, so I'm not the target market, thus my bias in my commentary and opinion on that topic).
That's cutting the tether from a _lot_ (there's no way to overstate this) of useful information, but that's potentially one of the great things about the open AI/LLM models, is that all that info is baked in there.
Disclaimer: As far as I understand it. Please educate me if I'm way off the mark.
Also, how much of the content of reddit is in Common Crawl? (same disclaimer applies to this comment)
I also understand 'the archival mindset', I hoard a bunch of data. But I also understand the logarithmic graph of futility.
If it has to be scraped then there may be other problems (which include lack of resources to make the data API accessible).