Kudos to everyone. It's been a long road, and I'm glad that the codebase that initially started as a text editor finally found a new home.
When you're a small startup and you're just starting up, you can create a single MongoDB instance (ignore everything about you've heard about Web Scale) and stuff data into it as needed, without thinking much about the structure. You can add in contracts on your database functions, which slowly specify the contract, as you learn more about what your project is really about. To get a sense of that style of development, please see what I wrote in "How ignorant am I, and how do I formally specify that in my code?"
http://www.smashcompany.com/technology/how-ignorant-am-i-and...
MongoDB is great for ETL. You can pull JSON from 3rd party APIs and store it in its original form, then later transform it into the different forms you need.
In large Enterprises, you will inevitably be trying to get multiple services and databases to work together. The old style for dealing with this was the ESB (Enterprise Service Bus) or SOA (Service Oriented Architecture) but in recent years most of the big companies I've worked with have moved toward something like a unified log, as Jay Kreps wrote about in "The Log: What every software engineer should know about real-time data's unifying abstraction". If you haven't read that yet, go read it now:
https://engineering.linkedin.com/distributed-systems/log-wha...
In this context, MongoDB can offer a flexible cache for the most recent snapshot your service has built, based off of what it read from Kafka.
Some people are sabotaged by MongoDB, and they start treating canonical data as a cache. Obviously that leads to disaster. I believe this is what happened to Sarah Mei. Her experiences caused her to write "Why You Should Never Use MongoDB"
http://www.sarahmei.com/blog/2013/11/11/why-you-should-never...
The one rule I would suggest is that you always need to be clear, in your own head, which collections are canonical and which are cache. When I talk to teams who are new to this, I tell them to use a naming convention, such as adding a "c_" to the start of every collection that is canonical. All other collections can be assumed to be caches. And the great thing is, it is very cheap to create caches. You can have 20 caches for the same data, in slightly different formats. You can have one cache where the JSON is optimized to what the Web front-end needs, and another cache where the JSON is optimized for the mobile app, and another cache where the JSON is optimized for an API for external partners. Just don't fall into the trap that Sarah Mei mentions, where you treat everything as a cache. You need to be clear in your head which data is canonical. If you are using Kafka the way Jay Kreps mentions, then the data in Kafka is canonical and everything in MongoDB is a cache. But at smaller operations, I've used MongoDB to hold both the canonical data and the caches, in different collections.
This strategy seems like it forgoes what I consider an important step in any project, which is, thinking critically about your data model and getting that right before you start building code on top of that structural foundation.
I could see doing what you're describing to build a prototype, which I would then extrapolate my learnings from, and subsequently toss out, but this seems like a dangerous way to get started with something that will end up in production (and potentially maintained for years to come), as it glosses over the importance of coming up with a really coherent data model, and let's face it, data is the heart and soul of most projects.
Am I wrong?
It's very much for prototypes, and especially greenfield projects. If I was, instead, doing something like building a new service, inside an Enterprise that was already using something like the unified log architecture that Jay Kreps has described, then I would certainly think hard about what the schema would be for the particular service I was building -- after all, in such situations you're never going to pull all of the data out of Kafka, so you automatically have to figure out what part of the data you want. LinkedIn currently stores 900 terabytes of data in its Kafka instance, and I'm unlikely to write a new service that actually needs all of the 900 terabytes of data. So merely by thinking about the question "What of this data do I need?" I'm already implicitly thinking about a schema.
Having said all of that, how often have you written a service where you got the schema 100% correct on your first try, and no further changes to the schema were needed. Possibly you are smarter than I am, but I personally have never done that. All of my first attempts need later adjustment.
Document-stores have their uses but they are not the best for everything. Thinking about structure isn't hard and all modern RDBMS have JSON fields now if you need that flexibility, while still giving you ACID, transactions and the power of SQL.
SOA has nothing to do with ESB/distributed logs/event sourcing, and none of that has to do with MongoDB or document-stores. Event-sourcing is unnecessary for most, MongoDB is not a good event-sourcing system, and the point of materialized views on a stream is that they can be modeled in whatever database works best, not to just accept what the stream gives you.
The last part about treating a database as a cache is also strange. Use a cache if you need one, but that's a much more complex topic then just having a few collections that are caches. And again, it has nothing to do with MongoDB or document-stores being the correct architecture for everything.
"In large Enterprises, you will inevitably be trying to get multiple services and databases to work together. The old style for dealing with this was the ESB (Enterprise Service Bus) or SOA (Service Oriented Architecture)"
You write "SOA has nothing to do with ESB" yet both are attempts at "you will inevitably be trying to get multiple services and databases to work together" which was literally the sentence before the one you are reacting to.
As to this:
"SOA has nothing to do with ESB/distributed logs/event sourcing, and none of that has to do with MongoDB or document-stores"
The point of my comment is that MongoDB is so flexible it can replace other approaches to the problem of "get multiple services and databases to work together" including ESB and SOA.
At this point, I can not think of any reason to ever use an SQL database. Either your canonical data will be in Kafka, or it can go in MongoDB. There is no need for SQL databases, ever.
About this:
"MongoDB is not a good event-sourcing system"
Obviously, which is why I included a link to the Jay Kreps essay. Kafka is better for an event-sourcing system. I'm not sure how you misread that part.
About this:
"The last part about treating a database as a cache has nothing to do with databases"
It feels like you are almost deliberately trying to misread what I wrote. My whole point was that MongoDB is so flexible, it can work as a cache, and also as a store for canonical data (in those circumstances when you are not storing your canonical data in something like Kafka).
Is that more clear?
> There is no need for SQL databases, ever.
What? This is nonsensical and ideological.
> MongoDB is so flexible, it can work as a cache
That's meaningless because any database can be used as a cache, but caching is not as simple as storing some records in a collection.
You seem to be caught up in some hype around Kafka, event-sourcing and MongoDB when this is a rather poor architecture choice for most applications. If you're going to say that relational databases are useless, you need some solid arguments instead of mixing up a bunch of buzzwords.
"You seem to be caught up in some hype around Kafka"
I hope you realize that when I mention Kafka I am merely offering it as an example, and the reason why I mention it is because it is the technology that Jay Kreps talks about in the essay that I linked to.
It is absolutely true that are many types of unified logs, and some successful companies have been built around different kinds of logs. For instance, at https://www.parse.ly/ (an analytics firm) their canonical source of truth is their Nginx logs. They have kept every Nginx log since the company was founded in 2009. Parsely also uses Kafka and Cassandra, but if someone accidentally deleted all of their Kakfa data, and all its backups, and all of their Cassandra data, all of its backups, they could regenerate all of their customers data simply by re-analyzing the Nginx logs, going back to 2009.
So there are different approaches to the notion of a unified log. And I mentioned Kafka only as an example. I apologize if that was unclear.
"A database is not comparable to SOA or ESB."
It feels like you are deliberately misreading what I wrote. SOA and ESB and the unified log architecture that Jay Kreps wrote about are 3 different approaches to deal with the problem of "In large Enterprises, you will inevitably be trying to get multiple services and databases to work together."
That may not be the only reason for SOA and ESB and unified logs, but multiple services with multiple databases is clearly a very large reason for the rise of these 3 different architectures. It is trivial to find articles in which "multiple services and databases" are mentioned as the impetus giving rise to each of these 3 architectures.
Is that more clear?
None of these things replace anything else, they are all separate layers of solving a problem. They are not "3 different architectures" and can all be used at the same time even. Talking about a unified log and SOA to say that MongoDB is the best database for everything only shows a poor understanding of the underlying technical concepts.
About this:
"and can all be used at the same time even"
Obviously. I've worked at many companies that use multiple architectures. Nothing I said implied these were mutually exclusive. But you must be aware that a system needs to know what its canonical data is, or it will get into trouble? That is the caveat. I've seen multiple architectures in use at large companies, but this is typically seen as a red flag, and indeed, I'm often hired to help solve exactly that problem.
About this:
"SOA is an architectural style where the entire business domain functions as independent components talking to each other. ESB is one type of communication between those components. A unified log is a way to persist those communications. A database is a data persistence system."
Finally, we agree about something. But you are simply describing what things are. Did you have some point you were trying to make?
About this:
"None of these things replace anything else, they are all separate layers of solving a problem."
At large companies I'd expect to see different architectures presiding over different parts of a system, but if I arrive at the company and I'm told it is using SOA and ESB and a unified log and I ask the CTO "Which of these systems is responsible for maintaining the canonical version of your data?" and the CTO responds "All of them" then I know I'm dealing with a company that is in deep trouble.
None of these topics (canonical data, SOA, ESB, unified log, event sourcing, kafka, caching) have anything to do with MongoDB and why it's the best database for every situation. You have not made any arguments for this. You're just talking about other things and then following up with patronizing comments about me being confused or disagreeing just to disagree.
Unfortunately at this point I don't think you have any arguments forthcoming. Let's end it here.
EDIT: Replying to your email here:
"I am assume you are reasonably intelligent, so I'm not clear why you wrote this.... Obviously, my point was that MongoDB was flexible enough that it worked well for different architectures. Since that seemed to confuse you, I repeated this point, over and over again, in different ways, hoping you would understand me. I am puzzled why you are having so much trouble understanding what should be a fairly straightforward idea? "
Again, not confused, please stop assuming that especially if you're going to have such an outrageous assertion to begin with. Other databases like Postgres are also flexible enough to work for different architectures. You should actually describe different architectures (not random concepts) and WHY you think MongoDB is better than everything else in that scenario. That would be an actual argument.