Revisiting the principles of data-oriented programming(blog.klipse.tech) |
Revisiting the principles of data-oriented programming(blog.klipse.tech) |
Once you have a relational schema that the business can look at and understand, then you go implement it with whatever tools and techniques you see fit.
This is what “data-oriented” programming means to me. It’s not some elegant code abstraction. It’s mostly just a process involving people and business.
Even for non serious business, these techniques can wrangle complexity that would otherwise be insurmountable.
I still think the central piece of magic is embracing a relational model. This allows for things like circular dependencies to be modeled exactly as they are in reality.
You might be surprised to learn that modeling your problem in terms of normalized relational tables ultimately achieves similar objectives. The more normalized, the more packed you will find their in-memory representations.
1: https://blog.klipse.tech/visualization/2021/02/16/data-relat...
Also when it comes to data validation, OO performs all of this validation too and in a much more compact and code-oriented, extensible way. Why would I write a separate schema when the object itself knows what it will accept, what is optional, and what range the values should be in? I'd imagine the schema and code could become incoherent.
As for why you'd want to do this, well, one reason is that it makes it easier to bounce data between different services. You don't need to perform any sort of conversion if you're operating directly on the data you're receiving and sending.
The second argument for this style is perhaps more ideological. In the Clojure community in particular, complexity is seen as arising from coupled components. The more things you can decouple, the less complex your codebase. The less complex your codebase, the more reliable and extensible it is.
Edit: another potential advantage is that its easier to use generic functions to interrogate and manipulate data that isn't encapsulated in specific types or objects.
That's not really true in classical OO or DOP. There is always code that depends on specifics of some data. In classical OO it's extremely common to de-marshal data straight off the wire and into a class (AKA hydration). From then on the thing that interacts with the data structure directly is the object instance.
For example, I have a Wizard object. My wizard has a wand, we store the wand object on our Wizard object. Simple. But then my wizard casts a spell, and does damage to a goblin. Do we put the cast method on the wand, the wizard? There is no real reason to pick one over the other (this problem comes up a lot with game development, which is why this pattern is more common there...Spring is another example, aspect-oriented programming/dependency injection works from similar principles). It is far easier to separate that out totally and have a pure, reusable cast function that takes the wizard, goblin, and weapon.
Another aspect of this problem (which Rust, as an example, makes clear) is that you introduce runtime bugs or hurt performance when you start carrying around a lot of references everywhere. Once you start to think about what actually needs a reference to another object (in Rust, this is limited by the borrow checker) then you realise why OOP doesn't work in some cases.
OO doesn't perform validation, your code performs validation on the data. You can write a separate schema, you can write one schema, but the problem is that OOP tries to fit a round peg in a square hole with some applications.
Very generally, it is harder to make mistakes if you use something like data-oriented. If you have a lot of code with calculations or interactions, it is very pure, easy to test, and fits well with how people think about those elements (one area I have found is financial applications, I actually worked this out and then found out data-oriented program existed when building financial-related stuff). In these cases, introducing OOP means state changing in unpredictable ways (and then someone comes into the project, doesn't understand the abstraction, calls a method that is named erroneously and it all goes wrong).
You did pick: "My wizard casts a spell". The cast method goes on the wizard.
Because data is just data and the meaning to it is given at the time of application. If you want to couple validation to the data itself - how do you decide which N of the meanings to validate against?
If you want to process the data you must use some language to access parts of the data. Data must have a symbolic representation, not juts be 1s and zeros. Or it can be a bit-stream, but even then you need a language that knows the different between 1 and 0.
person.name
extracts the field 'name' from the data. To manipulate the data, the program must know that there is such a field as 'name' it can ask for.
I mean, it's the whole point. You want to have something that will give you compile errors whenever you change anything to make sure that you go all over the cases where the change has an impact
As for why see for example Command Query Separation (the data oriented way) vs Tell Don't Ask (the encapsulate everything way).
Nothing forbids a function that applies validation to inputs before returning a data object? Extensibility can be done through functional means(e.g. higher order functions, function composition, lens) or oop(strategy pattern and equivalents, code object composition and inheritance,...). Not sure what you mean by more compact and code-oriented?
Code is always coupled to an interface, implicitly or explicitly. In the case of oop, code is coupled to the class, which can represent something specific with very concrete semantics (e.g. employee, author) or something generic that is meant to be subclassed(e.g. person).
Data is going to have an implicit schema regardless because that is just how data works. And once there is a schema, it may as well be expressed explicitly independently of the code because then you get the whole basket of standard schema operations for free (validate the data against a schema, provide a schema to an external consumer when moving data around, talking/operating more generally on schema to manipulate data, generating glue code or APIs programmatically).
Your description sounds like you are using your objects as schema references which is fine, but if there is a 1:1 correspondence with schema then you are already doing data oriented programming, and if there isn't then you can't have 3rd party libraries that support schema-based operations. And losing those schema-based operations hasn't gained anything because the data still has a schema, it just isn't well organised.
TLDR; Data oriented programming isn't essential. But if you plan on passing data around between systems schema should be mandatory, and if you pass data around within a system schema are recommended.
> Why would I write a separate schema when the object itself knows what it will accept, what is optional, and what range the values should be in?
In practice, I have seen a fair number of complex objects where that information is obscure. If there isn't an explicit schema there is a chance of bugs where the object doesn't understand the data it is ingesting and that it won't share its knowledge that there is a problem until it fails in some obscure way in runtime. It wastes a lot of time fixing those bugs because the easiest way to clean that up is to tease out an explicit schema & start thoroughly validating inputs.
Nah, code exists in a different universe then "data" and it's decoupled by default. Runtime data is completely unaware of "code." that is unless you do something called "reflection" which is sort of a rarely used feature.
For example C, is not OOP. Linus Torvalds hates OOP, so linux is in written in C. Go was created by Robert Pike who's also subtly against it, and it shows in the language. Additionally Rust pretty much gets rid of objects as well. React is also moving away from class based representation of components.
These are just modern languages that are moving away from OOP. In addition to this... behind the modern languages there's a whole universe and history of other styles of programming.
Not against OOP... But I'm saying it's not a good sign if OOP is the only perspective you're capable of seeing.
Wait, what? Just add another field/property to the existing class. It's a silly example anyway as normally you would just add a function to concatenate the two strings.
The stated advantage is to be able to add the new property "on the fly". I suppose this means without changing the code. It does beg the question "what can existing code possibly do with this" (other than display it in a generic way or count the number of fields)? Furthermore, adding something new is rarely much of a problem as it is a non-breaking change. A more difficult example would be removing the "firstName" field. Assessing the impact of such a change in a large code base would be extremely difficult. Get good a grep and hope that the test suite is comprehensive.
Unfortunately, not all that glitters is gold. It feels extremely beginner oriented, only touched basic concepts taught at uni level and it shows a huge disconnect between the theory and the real world work of a developer leveraging data in any way, shape or form.
Don't buy the book, it's so not worth it.
https://blog.klipse.tech/databook/2022/06/22/generic-data-st...
The examples also contradict his other principles, i.e. immutability.
//Anonymous type with integer property
var foo = new {Number = 1};
//Compile time error if you try to assign a the integer property to a string
string s = foo.Number;
I guess you mean dependent types[1], but if you don't, I'd appreciate an elaboration. If you do mean DTs, how might it look for a hetero collection?
[1] If anybody has any good intros to dependent typing in C#, that'd be much appreciated. A web search throws up some pretty intimidating stuff.
Even if you don't use that, you could certainly orient your data as "structs of arrays instead" of "arrays of structs" (so to speak). It's fairly common in games.
Could anything be more confusing with a large code base? Also, lots of nice key not found and invalid cast exception errors to debug with this approach. Sometimes boxing makes a material difference to performance as well.
If your primary data structure is
Map<Integer, List<String>>
On the other hand, if your primary data structure is
Map<CustomerId, List<Purchase>>
He also seems to be unaware that you can have generic code with specific types.
An example of what I mean is Spring. Obviously, from what I recall, that goes to other extreme with lots of XML configuration. But there is no need for vague types that can cause all sorts of mischief at runtime. The key idea is splitting code from data, not necessarily the representation of the data (although that can come into it if you have performance-sensitive apps).
This post (and the book it points to) is perhaps teaching a new generation what has been known for a long time: the "body" of your business is the data, not the code. E.g. if you have limited space on a thumbdrive and can only keep one thing in a datacenter fire, your database or your codebase, you keep the database.
[1] https://youtu.be/LKtk3HCgTa8 [2] https://youtu.be/2V1FtfBDsLU
This is not OOP, this is a way to do functional programming in a class-based language that lacks top-level function declarations / modules.
While this might seem a nit pick it makes me sceptical about the rest of the content.
Static functions introduce friction against making impure functions, but not an overwhelming amount of it. If it's all you have, then there are worse safety blankets, but it's not going to solve your lack of buy-in problem. If everyone is on board then you don't need static functions. If they aren't, static functions aren't going to save you, they're just going to extend the amount of time you suffer before you wise up and get a new job somewhere else.
One code example is worth 1000 images, and 1 image is worth 1000 words.
Always use code block to illustrate your point, as it help reader understand better your point.
Writing a book is more about getting reader into your thought rather than make them think.
> Principle #2: Representing data with generic data structures.
OK probably this is not what I expected.
If that's happening a lot that's not really FP anymore, is it?
But when you use a framework that enforces OOP, it's quickly difficult to use DOP.
It is why C++ continues to grow. Some complain about that, but every single feature got there over fierce opposition by making some common programming problem more tractable.
Orientation should have been 'a preference for' not 'a dogmatic adherence to'. A hot-dog based diet still contains bread, ketchup, mustard and pickles, possibly some sort of cheese. A hot dog diet is just hot dogs, which is much, much less interesting, and there is no question that it is unhealthy, whereas the former might have some plausible deniability (especially if you add beans).
Preference is silly. The problem dictates its solution. The good programmer listens to the problem.
I am writing business line applications - I don't have much need for "generic" functions like outlined in the article. My framework/language provides for example generic .Sum() I could use if I implement specific interface.
But usually I have to make specific sum and put it in database or in the interface.
Like I need to sum age or sum prices or sum amount of items in inventory - and I have to show these in the interface. I think it is quite BS to say there can be "generic" data structure and "generic" functions in context of business line application.
Other stuff I was doing was warehouse automation system and if I had X,Y,Z coordinates I had these in generic data structure named Coordinates - but any function that was going to do anything with coordinates had to be implemented in the context of machine. For example lift should never operate on X cooridinate I could calculate distances - but then there was never use case to calculate distance between machines because these had static access points and one would calculate distances to these access points only.
In OOP data and method are unionized into primitives that form the basic building blocks of your program.
This is unique to OOP. It must be the definition. Defining it in terms of message passing, encapsulation and inheritance are not as good because these concepts are used in other paradigms as well.
Isomorphisms exist everywhere. You could say nothing is OOP because it compiles into assembly anyway where pretty much all those concepts don't even exist. You can even say all of assembly can decompile into OOP so everything is OOP.
Think about it. If linus torvolds hates OOP. What is it that he hates? You can say, Linus is an idiot and everything is OOP so he's wrong.
Or you can understand what is it about OOP that he hates and understand the delta between my examples and what is traditionally thought of as OOP.
None* of my examples contain a class. That should be a hint, as that syntax is pretty much used in most OOP languages.
*React still contains class based components except the creators now recommend moving away from class based syntax to functional components.
but you can strictly enforce it in pretty much every relevant OOP language - Java, C#, C++, C, Rust, D, ... by defining a class / struct / record with these two members, which is the only thing that matters. No one programs in abstract design principles, only in actual programming languages.
struct Name {
std::string firstName
std::string lastName
}
struct AnotherName {
std::string lastName
std::string firstName
}
struct YetAnotherName {
std::string middleName
std::string lastName
std::string firstName
}
This isn't theoretical. I often don't want and often cannot use inheritance-based polymorphism. If I'm using a language where that is the only option, I'm struck writing tons of redundant, error prone, pointless, and brittle glue code. The amount of glue explodes combinatorially. That glue code can contain errors that the type checker won't find.
The inverse of this problem is also interesting. Someone wrote a function to concatenate the strings in Name. I can't put AnotherName into it unless the original author had the forethought to make their function templated. I guess the future of C++ is that all code ever lives in headers.
It's really not as cut and dry as you said.
In this case it just mean you have to match your game semantics to the decision of where to put the ability. That doesn't mean it would be a good idea to perform fighting by simple method calls on the objects though. It is very likely one would want to extract the abilities granted by skills and object to a separate entity that change seldom enough.
That doesn't mean I see what problems a schema solves here. That mostly seems like unecessary indirection..
Also note I don't try to refute the problem in general, just this specific example. Maths have lots of examples that doesn't have the semantic baggage that makes the question solvable.
You can also have the spell on the wand and require n mana to be put into it for activation (argument), now warrior type goblins won't be able to use it either.
There is no "correct way" with OOP: There are as many ways as you've got the energy to think about and all of them are leaky somewhere. You can still write maintainable code with it, but making it sound clear and obvious where what should go is just wrong. It heavily depends on a multitude of factors and can get nauseatingly complex at times
This is a good tutorial on this topic - https://ericlippert.com/2015/04/27/wizards-and-warriors-part...
const p = Point(3, 5)
p('getX') // 3
p('up', 11)('toString') // "Point(3, 16)"
const Point = (x, y) => (method, ...args) => {
switch (method) {
case 'getX': return x;
case 'getY': return y;
case 'toString': return `Point(${x}, ${y})`
case 'up': return Point(x, y+args[0]);
// ...
}
}
At best what I've done here is eliminate some of the vagueness surrounding the definition, which is really all that's needed in most cases.
While none of the OOP solutions are outright wrong, neither one are "obviously right".
With full dependent types one can write generic types like std::array and use them with runtime parameters. In dependently-typed languages there are two main types: Sigma (dependent pair) and Pi (dependent function). Example of Sigma type in pseudo C#:
(uint n, Array<int, n> a); // array of any size
Array<T, (n + 1)> push(Type T, uint n, Array<T, n> a, T x) { ... }
Array<int, 3> x = push(int, 2, [1, 2], 3); // [1, 2, 3]
A bit larger example:
void Console.WriteLine(string format, params object[] args);
WriteLine("{%int} {%bool}", ?); // ?'s type is an array with an int value and a boolean value.
I know it's possible to have DT in C#, I think your literal of '3' in the above example is constructed using the successor function (well, its moral equivalent) at compile time, I just can't find the article I read and there's very little out there for DT in C# at all. And it's over my head until I sit down with a good example and work it through.
I could use some simple DT for list lengths in my project.
(defalias NamedMap
(HMap :mandatory {:first-name Str, :last-name Str}))
(ann full-name [NamedMap -> Str])
(defn full-name [{:keys [first-name last-name]}]
(str first-name " " last-name))
const post = {
id: 123,
content: "...",
published: true,
}
// TS infers the type of `post` to be an unnamed "map-ish" type:
// { id: number, content: string, published: boolean }
now, dependent types allow you to express pretty much arbitrary shapes of data, so you can do heterogenous collections as well. i haven't read about it enough to do a map, but a list of tuples (equivalent if you squint) is "easy" enough:
[
("id", 123),
("content", "..."),
("published", True),
]
-- a function describing what type the values of each key are
postKeyValue : String -> Type
postKeyValue k =
case k of
"id" -> Int
"content" -> String
"published" -> Bool
_ -> Void -- i.e. "no other keys allowed"
-- now we're gonna use postKeyValue *at the type level*.
type Post = List (k : String ** postKeyValue k)
-- "a Post is a list of pairs `(key, val)` where the type of each `val` is given by applying `postKeyValue` to `key`.
-- (read `**` like a weird comma, indicating that this is a "dependent pair")
in general if you want to learn more about DT's, i'd probably recommend looking at a language like Idris with "native support" for them. backporting DT's onto an existing typesystem usually makes them much harder to read/understand that they actually are (and don't get me wrong, they're mindbending enough on their own).
if you don't want to bother with that, i'd look at Typescript - it's combination of "literal types", "type guards" and "function overloads" can get you some of the power of DT's. see this article for some examples: https://www.javiercasas.com/articles/typescript-dependent-ty...
I'm actually talking about C# because I'm working in it and I'd like to make some compile-time guarantees if possible. Or at least know how to assure a method that they are getting a list with at least 2 values in it, for example. It may not be worth the effort but it would be nice to know how.
I've got books on DT, idris, and another DT lang, trouble is there's no call for any of this stuff in industry so they get repeatedly pushed to the bottom of the priority stack. Sickening, innit.
there's things you can do to track list length at the type level, but it usually involves putting your data in a special-purpose linked-list thingy: https://docs.idris-lang.org/en/latest/tutorial/typesfuns.htm...
(the `S` and `Z` refer to peano-style natural numbers)
although if you go that way, you can actually get a lot of that without dependent types! here's an example i found of someone doing a similar construction in C#: http://www.javawenti.com/?post=631496
last but not least, in TS you can use the builtin support for arrays as tuples and just do:
type AtLeastTwo<T> = [_1: T, _2: T, ...rest: T[]]
enforcing invariants means reducing the number of types that satisfy the invariant. I wouldn't call doing what you ask for "enforcing" any invariant.
> Bonus: write it for any struct containing firstName and lastName.
well,
auto concatenate(auto t) { return t.firstName + " " + t.lastName; }
> I guess the future of C++ is that all code ever lives in headers.
or in modules, which makes it very similar than other languages with generics instantiation
template<typename T>
concept WesternishName = requires (T t) {
{ t.firstName } -> convertible_to<string>;
{ t.lastName } -> convertible_to<string>;
};
auto concatenate(WesternishName auto t) { return t.firstName + " " + t.lastName; }I need to sit and read and butt heads with these if I can find the time.
It's so odd there are so few examples of C# DT. The one good one I found a year ago seems to have disappeared. Maybe a conspiracy?
Looking forward to Idrisizing or Agdicating some time!
another term that might be useful is "GADT" - it's kinda like a weaker form of DT, more easily expressible in, uh, normal languages. the C# one i linked is really more like a GADT, bc it all stays at the type level w/o bringing "normal" values into it. another way to do it would be a sealed class with a private constructor + static methods:
class Foo<T> {
private Foo(...)
static Foo<int> Bar() { ... }
static Foo<string> Zap() { ... }
}
so that way you can have the type param and use it to track something, but limit what can be put there - in this case, only int or string. type-safe syntax trees are a common case for sth like this (though in that case you'd probably go for an abstract base + subclasses, like in the link).
Sure, the code kinda sucks either way, but the data oriented approach works exponentially better as the object interactions become more complicated. A “cast” function called as part of the event loop can look up all the game state in the state DB as it needs to. wand.cast(…) is a lot more brittle, ESPECIALLY once one wants to start reusing some of the code in sword.swing(), etc.
That's not how it works. Multiple dispatch looks like a normal procedure call, it would look like
cast(item, user)
cast(item, user, worldstate)Or, as I sometimes think of it, it can be an OOP design if multiple dispatch is actually a set of messages supported by an implicit, singleton, and usually hidden, “multiple dispatch“ object that handles the dispatch logic. You don’t have to write “dispatcher.cast(a, b, c)” but that is because the language provides the syntactic sugar (and often, an efficient implementation).
>the data oriented approach works exponentially better as the object interactions become more complicated
Maybe it's just me, but I still don't see the difference. To use your example, you'd have a function like:
cast(caster, targets, weather, time, location)
or you can create an object and call it something like Spell and do something like:
class Spell
def initialize(caster, targets, weather, time, location)
...
end
def valid_target?
...
end
def valid_caster?
...
end Perhaps it's just my mental conception of things. My mental model of a class is a data structure plus a bunch of functions that implicitly take the data structure as a parameter. I realize that there can be more to it than that but it works for the purposes of this discussion.
Ultimately it's a code organization question either way. The original question is what class does it go in? Changing it to data structure + functions just changes the question to what module/file does the cast function go in? Maybe that's an easier question to answer but I guess I just don't see it.
That's not OO. A class without instantiation is just namespacing.
What exactly does this change? In a well-written program you should be able to write code that adapts to context so that you don't have to pass absolutely everything.
> Sure, the code kinda sucks either way, but the data oriented approach works exponentially better as the object interactions become more complicated
Maybe what you want is actually https://mitpress.mit.edu/books/software-design-flexibility.
I dunno. From https://en.wikipedia.org/wiki/Multiple_dispatch
"Multiple dispatch or multimethods is a feature of some programming languages in which a function or method can be dynamically dispatched based on the run-time (dynamic) type or, in the more general case, some other attribute of more than one of its arguments.[1] This is a generalization of single-dispatch polymorphism [...]"
It's just a nicer OOP to me. *shrug* But thanks.