I have started a discussion at https://groups.google.com/forum/#!topic/streembit-dev/Xl2DpG... to start working on the problem and propose a solution for the issue.
Disclaimer/disclosure: I wrote the Streembit application which is mentioned in this thread, so I am heavily biased towards decentralized, peer-to-peer solutions I try to explain in this blog what we try to achieve with Streembit at http://streembit.github.io/2016-02-23-What-is-Streembit/.
We use open standards such as JOSE, JWS, JWT, JWE and the audio & video is WebRTC that is open source as well. All crypto are FIPS compliant and the ECC and ECDH uses recommended curves. The IoT is directly mirrors our work in W3C WoT. I am working on W3C WoT security and move the code from there to Streembit. The objective is with Streembit to stay based on open standards.
It's a relatively easy problem to solve, but I'm not sure that hardcoding IP addresses into programs is the best solution for users who do not compile from source.
Out of curiosity I once reviewed the source code for a certain "onion router" program and found a handful of hardcoded IP addresses. Servers at MIT if I recall correctly. If these servers are inaccessible, what is the effect?
Perhaps "bootstrapping" is simply a matter of voluntarily downloading a trusted file, containing some reachable IP addresses.
Can millions of people all download the same file? If no, then how does Firefox, Adobe Flash or Chrome get installed on so many computers?
I consider the web (cf. internet) as more or less "bootstrapped". This is because of the preference for domain names versus IP addresses. It necessitates lookups.
Bootstrapping all starts with one file: root.zone.
Since 1993, this file has not changed very often. The IP address for the server where one can download it does not change very often either.
Once I have that file, whether the root servers are working is irrelevant. Don't need them. Only need TLD nameservers, .com, etc. to be working. And of course one can download a local copy of the .com zone, etc., as a backup.
But assuming those root servers are working I do not even need this file. Long ago I memorized the IP addresses for one root server and one com nameserver.
As long as I have a means to send DNS packets[1] I can look up the IP address for any domain name and I can "surf the web". Or at least get myself out of a jam without the use of recursive DNS servers.
1. Even netcat will work, as explained in its tutorials. I do not use "dig" or BIND libraries. Nor do I ever need to set the recursive bit. I can do the resolution using non-recursive requests, using some helper scripts/programs I wrote. Interestingly, using this method beats the speed of a cold cache in nearly all cases.
For the newly decentralized web, we will need supernodes that do not pass traffic but only provide port information to allow users to connect with each other through NAT.
The IP addresses of these supernodes will be the "bootstrap". There could be millions of supernodes.
Today, the "web" is addresses of servers controlled by companies pandering to advertisers.
The "decentralized web" is addresses of users that can connect with each other to form communities. Those communities can be connected with each other, and so on.
The "content" being served to users by today's web is largely "user-generated". But overrun with advertising and "analytics". The content is served indiscriminantly to the open internet. And even if there are access controls, because the data is centralized with a handful of companies it's an easy target for third parties who want access to it.
Tomorrow's decentralized web allows user to selectively exchange their content with each other, without the companies and advertisers in the middle. Third parties wanting access may have to be more clever and more selective.
<<< Servers at MIT if I recall correctly. If these servers are inaccessible, what is the effect? >>>
Precisely, that is the problem.
<<< Perhaps "bootstrapping" is simply a matter of voluntarily downloading a trusted file, containing some reachable IP addresses. >>>
Could be a solution. The issue is again, from where we download it? If we download it from a centralized source then the decentralization does not exists. There are suggestions to solve this with Bitmessage or Telehash, but those are both having this very issue with bootstrapping. Using Bitmessage or Telehash for bootstrapping an another network is just kicking the can down the road. I understand you didn't suggest this :-) I am just saying.
The problem with DNS is that an authority can ban a domain name and the idea of permissionless systems is that there is no central authority should control the access of users.
mDNS and UDP multicast work fine on local networks and we are working to solve this on global networks as well. IPv6 anycast looks promising but I haven't got yet the prototype.
The point is that it only occurs when we have a specific need and a niche vertical to initially leverage it in. In one form it's bitcoin for payments. In another form its IoT and essentially sensor networks. You can imagine autonomous vehicles being a huge p2p network for speed as opposed to feeding all the data back to the cloud.
At this point the net may well be seeing its third such cycle.
Streembit is currently the only such project actively participating to W3 Web of Things standardization group and already incorporates all currently evolving standards put forward by the W3 Web of Things (WoT). We're working on a proposal to standardize "IoT over P2P" that we can put forward to IETF.
[0] https://streembit.github.io
[2] http://blog.valbonne-consulting.com/2016/06/09/streembit-hel...
[3] https://en.wikipedia.org/wiki/Kademlia
[4] despite popular believe P2P networks suffer from centralization issues in P2P bootstrapping. Also Ethereum is not truly decentralized as the DAO attack shows
Disclaimer: I do not financially benefit from evangelising Streembit though I'm a voluntary contributor to the project so my position is certainly one that I would want to see them succeeed.
This seems to me the core problem and contradiction of the current initiative: As I understand it, the whole promise of decentralized services is that:
1) Because most of the logic is run by peers, the "upkeep" for the designers of the system is very low, if existent at all. Ideally, the system is completely self-sustaining (as long as there are peers) and the designers don't have to do anything to keep it running. (In the face of bugs and hackers this is probably utopic as you'd need at least some way to do security patches. It should however be significantly less than to run/rent servers)
2) Because most of the logic is run by peers, its the peers who decide how the system evolves, not the original designers. One consequence of this (and the main reason why users would want a decentralized system) is that the designers can't change the system for political/strategic/profit reasons.
If VCs are now pushing startups in building decentralized products with the traditional expectations of growth and/or profit, I don't see how we can get a product that satisfies point (2).
My fear is that anything that comes out of such a partnership would be "flawed by design" in the same way that DRM and many IoT products are: Such a product would look on the surface just like a "real" decentralized system, except with the one thing removed that makes decentralized system more useful than centralized ones: The lack of control points. Because you'd need such a control point to actually steer the system in a direction that satisfies investors.
Let's hope people continue to try to create services that a big percentage of people use and then resist to sell to the big data companies.
They seem to understand where they are too - it goes beyond just buying out competitors. Look at Facebook's Internet Lite plan for India.
This too, shall pass.
Which was a spectacular failure...
Side note: I found it amusing how the article pretended that decentralized downlownding of copyrighted media stopped after Napster, and that:
> these technologies ended up being limited to a few services, such as Skype.
I'm having trouble interpreting the section with this quote as anything other than the author being hugely misleading. According to Wikipedia, BitTorrent alone has at least 150 million users.
We have to look at why these centralized services have succeeded: for instance the network effects, the economies of scale, their business models are attractive to ambitious VC investments, they have better UX because they have more resources, and so on. Then you can either try to build something which matches these advantages, or you can try to do an end-run where you offer the market something more important.
Privacy and security are things the market is starting to value but I don't think these alone can woo everyone away from the network effects, better UX etc. of the centralized giants.
Anything which brings people out of an ad-heavy experience has a lot of appeal right now. I think we'll see more freemium and paid services in the future which is good because the economies of scale in the ad business are brutal to small businesses.
Anything we do to create more robust p2p and client-side encryption platforms is important. When building an app today it's just much easier to run it on a server and keep all the data on a server. p2p storage, identity and so on needs to become just as easy.
And finally we have to ask how this re-decentralized stuff is going to attract investment and build big businesses, because if the big money is always on the other side it's going to be very hard to win.
To be honest I think the technical challenges are fascinating but it really needs to start out with the business model and investment challenges, if you can figure out how a lot of money can be made off of a decentralized search engine or social network, now you have some real allies and resources and marketing behind you. This requires us to think about why open source business models are typically not as lucrative as proprietary walled gardens, and why they have a hard time getting traction in b2c.
Some years ago in the university I wrote a paper about that in a seminar to summarize the possible counter meassures, but from what I remember, there were no really practical solutions. I think the most promising one is the use of a centralized certificate authority, which reintroduces some of the problems p2p wanted to solve. Does anyone know if new ideas have come up in the last years?
I believe their system will work. Or require only minor tweaks.
Combine that with using the global overlay only for bootstrapping of common-interest sub-networks and you'll limit the incentives that an attacker will have to attack the global overlay while also reducing the effectiveness because a single non-fake contact will be sufficient to join the subnetwork.
It's not an absolute defense (sybil-resistance without central authority is hard) but in practice it won't be worth it for attackers.
Attackers will probably have a lot of computational resources (think of AWS, botnets, GPU computations), while typical users don't (mobile phones).
I haven't had time yet to look into the IPFS details yet, but thanks for the reference.
I hope that in the next century, we'll chuckle with nervous laughter at how absurd the loaded assumptions in quotes like this were, and at how reasonable it seemed at the time.
Unless we're supposed to parse "slew of startups" as singular...
Yes, that's how that works, grammatically. The noun in that sentence is the slew, which, in this instance, happens to be of startups.
That's a fairly modest problem. You need a bunch of nodes with above-average capacity to handle the bootstrap traffic and a way to occasionally update lists of such nodes. But a bunch of 10$ per quarter virtual boxes at a hoster of your choice and a bunch of domains with DNS SRV records can do that job.
In the absence of any/multicast-for-everyone someone will have to pay a little money or (ab)use a handful of non-decentralized services for the bootstrapping but that's more of a philosophical problem than a practical one.
There certainly is no single point of failure, you can have plenty of redundancy in the bootstrap process.
The closest and best solution for this problem was the early days idea of Bitcoin to use IRC to publish the listening seed information, but IRC still not decentralized, so IMHO decentralized bootstrapping is a real problem which we need to solve.
I kicked out a discussion for this at https://groups.google.com/forum/#!topic/streembit-dev/Xl2DpG...
I think it is the misunderstanding of how decentralized applications work. The seeds decide about nothing during runtime. In an ideal scenario the seeds execute the application without being able to make any decisions terms of the business logic. That's how the trust between seeds can be established by executing the common functions i.e. Bitcoin mining or performing the Ethereum smart contracts by executing the application.
The issue is, we do not want the peers decide anything or be able to modify anything term of business logic. The objective is that all seeds execute "the" honest, published version of the software. Because such cannot be guaranteed, fundamental security issues exist in decentralized computing, some node can act dishonestly, and therefore vulnerability to 51% attacks and Byzantine Generals Problem exist.
To take the WWW as an example, the W3C once tried to unilaterally update the system with XHTML2 and utterly failed, because none of the peers were adopting the change. But even the browser vendors don't have (or didn't have at least) the full control: they were bound by backwards compatibility of existing sites and by the threat of new browsers or forks emerging. Finally, the peers can change what they are executing by using extensions or forked browsers.
Decentralization can be a achieved to a great degree by simply taking federation to its extreme.
IPv6 anycast looks quite promising[0][1][2] to help solve the problem with fully decentralizing bootstrapping of P2P nodes. This is a central issue currently affecting all p2p like networks including Ethereum, BitTorrent, Gnutella, Bitcoin, ...
main issue is probably lack of IPv6 anycast support (??) due to misinformation, broken middle boxes, and and lack of understanding in how to properly configure your network correctly for IPv6 (there are so many features making it also more secure, yet the impression I get is that people don't want to learn about it and in practice just aim for IPv4 maximum compatibility ... ).
[0] http://ryandoyle.net/assets/papers/Distributed_Bootstrapping...
[1] http://doc.tm.uka.de/2009/ipv6-contest-p2p-bootstrap.pdf
[2] https://www.ipv6council.de/fileadmin/summit09/presentations/...
It requires root or user-namespace shenanigans to assign the v6 address needed by the p2p client.
And access to BGP tables.
And they don't estimate how long such a bootstrap process would take, since for small networks it would potentially have to scan millions of candidate addresses.
The need to bootstrap is ubiquitous. Even a user's internet connection is "bootstrapped". She has to know at least an RFC1918 address to get started.
Disseminating a list of addresses of working supernodes so users can form networks and connect to each other should not be an insurmountable problem.
The list does not have to disseminated via the network. Remember the days of POP and dial-in numbers? If the user has no internet connection then how did she get the dial-in numbers?
This is not a difficult problem.
re: ability to "ban a domanin name"
When authorities "ban a domain name" via DNS, they only ban lookups using certain DNS servers. The server at the IP address associated with the domain name could still be accessible.
The reason banning DNS lookups in order to take sites "offline" is so effective is because usually these sites are doing something shady and need to keep changing IP addresses frequently. No one knows what IP address they will be using in the future. They are very reliant on DNS.
Otherwise, if we are dealing with a legitimate site that changes its IP only infrequently, it would be futile to try to "ban" it via DNS.
It would be like expecting every nerd worldwide to forget that ftp.internic.net is associated with 192.0.32.9 or that example.com is associated with 93.184.216.34.
Some will have saved this information. There are publicly available archives of historical DNS data.
EDIT: I should mention I'm on satellite internet (there's quite literally no options for broadband where I live. I will, however, use my phone line for SSH via dial-up occasionally), so it's probably not very practical to run SMTP. For that, I have a Fastmail account.
The key point is that any node in a p2p network can be used to join the network, so any number of independent entities can publish lists of ways to join the network.
Certainly, i would also love some multicast/anycast support on the IP level, but we don't have that, so making sure that you have many independent entry points into the network is the next best thing.
Maybe you're thinking distributed, not decentralized? https://i.imgur.com/YsiiKeq.png
A bigger problem is how software is developed. The development itself is centralized, but even worse, traditional software development approaches cannot predict all future problems, but rely on fixing them to keep the software working, which means it must be delivered to users in a centralized way too.
http://ryandoyle.net/assets/papers/Distributed_Bootstrapping...
http://grothoff.org/christian/dasp2p.pdf
http://www.net.in.tum.de/fileadmin/TUM/NET/NET-2014-08-1/NET...
You say: "i.e. changing hardcoded bootstrapping data every N minutes, so new users could bootstrap from different nodes"
The fundamental issue is, changing the data where? The node who wants to connect to the peer network, obviously cannot obtain the information from the peer network itself (as node is not connected), so obtain the information from where? Currently, all decentralized applications, including my system Streembit use the techniques of obtaining the information from a centralized source - which is the oxymoron of decentralization. If a web services or other centralized applications provide the new node with the list of existing listening/connected nodes then the solution is surely not decentralized. Government agencies or cyber-criminals only need to attack the hard coded, listening seed nodes and then the network is done and never can be back again until a new list of hardcoded seed nodes is published via the application source code or via other channels.
As I said above, Satoshi's original idea to obtain the seed info from IRC was the closest to decentralization, but since IRC is centralized itself I am sure you can see how far that is from the a decentralized bootstrapping.
We have the solutions on local decentralized networks such as mDNS and UDP multicasting which uses protocol level solutions, and we are investigating to solve the problem at Streembit with IPv6 anycasting.
<<< A bigger problem is how software is developed. The development itself is centralized, but even worse >>>
I disagree. Open source software can be forked and then you can adopt as much democratic development methods and governance as you want.
Ok, you are assuming that the node already has the binary somehow. But that's not the case. In the real world we have to ship binaries to users. And this is where you can put your different bootstrapping data for different users.
You can go a lot farther: let users generate a binary distribution of the software to share with each other and hardcode bootstrapping data there obtained from a running network by that user.
> Open source software can be forked
The majority of users are not going to do that, they just don't have the skills. At best there will be a few popular distributions of the same "decentralized" software, with majority of installations controlled by a few entities. At worst - just one centralized entity that controls every installation.
Other techniques and several research papers suggest that scanning of a wide range of IP by guessing what could be a connected node is an option as well. In theory it certainly works, if you scan the whole internet then soon or later you will find a connected node and in theory it does solves the problem of centralized bootstrapping, but all these papers also agree that such scanning could be terribly inefficient.
What I mentioned, mDNS and UDP multicasting work fine on local networks, and we are experimenting with IPv6 anycast on global network to solve this problem.
Well, I am not assuming anything. I am talking about a fundamental problem of decentralized networks: the current bootstrapping of networks is centralized. Please refer to the quoted papers and there are many other research papers as well which describe this existing problem.
<<< But that's not the case. In the real world we have to ship binaries to users. And this is where you can put your different bootstrapping data for different users. >>>
You are misunderstanding the problem. The point is
a) we should never ever hard code the seed information into the application and consequently ship it with the binary. If we do rely on the current approach then you always connect to the seeds of ETH foundation, Bitcoin foundation and to my company's seeds in the case of Streembit which is the oxymoron of decentralization.
b) we should have a protocol level solution for entity discovery instead of application level solution such embedding the seed info in the source and then compile it into the app. When I say protocol level solution I refer to mDNS and UDP multicasting which works on local networks just fine and I am proposing IPv6 anycast for entity discovery on global networks.
The simple truth is that Bitcoin, Ethereum and all cryptocurrencies conveniently ignore this issue. The companies, lead developers, foundations or whoever run the show maintain the seed nodes, but such solution is surely not a decentralized solution.
No! This is a problem that solves itself once you solve a binary distribution problem. But none of the papers you refer to address the problem of the centralized binary distribution and jump right to the protocol level for some reason. It doesn't work like that.
What difference it makes if you disseminate a fundamental design problem (i.e. the bootstrapping of the network is centralized) with a different type binary distribution? Whichever method you use for binary distribution, the distribution will deliver the very same existing problem. Again, please refer to the quoted papers to understand why the centralized network bootstrapping is an issue.
On the note of binary distribution, yes that is an issue as well and it would be nice to have a decentralized binary distribution, but again, that is an entirely different problem. BTW, I think our application Streembit can be used for decentralized binary distribution as well.
No. The problem of centralized bootstrapping that is an existing and real issue of decentralized networks (see the quoted papers and my above explanation) does not depend on the problem of "binary distribution". In fact it has nothing to do with "binary distribution". If a user builds the software from source - so the user avoids any "binary distribution" - then the user still have the problem of centralized network bootstrapping.
It seems you don't understand that the problem of centralized bootstrapping is a generic information technology problem of all users, regardless what was the method of "binary distribution" if any. That's fine, it was a good discussion, but I exit from this debate with you which is becoming meaningless now :-) Thanks for sharing your view" :-)
Again, sources do not emerge out of thin air and have to be shipped to users somehow. No matter what you do, user must get the software first to bootstrap. And since bootstrapping always requires information from the sources it cannot be considered a separate problem until you solve the problem of shipping that information to users in a decentralized way.
All of the so called "decentralized" software is fundamentally broken at the moment.
I am perfectly comfortable with disseminating the source via the centralized Github, though it would be nice to have a decentralized source control system. Users trust me and get the source from my repository. Users who don't trust me can fork the software and modify or distribute for themselves from the repo of a trusted person. The source distribution is centralized, but so be it. Once the users have the source/binary via Github the application addresses several use cases in a decentralized manner. I don't see how the centralized source repository invalidates the runtime soundness and robustness of a decentralized payment or communication P2P app. At the sametime, the runtime soundness and robustness of decentralized payment or communication P2P applications seriously affected by the centralized bootstrapping.
<<< All of the so called "decentralized" software is fundamentally broken at the moment. >>>
Quite true, but it is broken, because of the centralized bootstrapping, and not because of the source code is managed by a centralized repository application. I can drive with my car to your place and give the source code to you in person, which will be the ultimate decentralized source code distribution, but the issue I was talking about (centralized bootstrapping) will still exists once you run the software. I am talking about design and runtime problems of decentralized networks, and you have been talking about something different. Anyhow, thanks for the chat and all the best! :-)
