Native Reflection in Rust(jack.wrenn.fyi) |
Native Reflection in Rust(jack.wrenn.fyi) |
Basically, the other system is a dynamic library which sends and receives C structures from my application. Those structures are then mapped into a buffer that is supposed to have the same size and there are pointers with metadata pointing into the buffer that are supposed to be exactly like the struct elements. Those structures can have arbitrary complexity, and are passed around through type erasure (essentially char*).
I wrote a "reflection" code for the other team, which runs when they register the struct instance to be sent, checks if there's a matching PDB [0] around, reads it, and outputs a json including the metadata needed, which can then be used to define the structures' metadata on our side correctly.
This is all in C/C++ since in some contexts we have soft real-time requirements, else I would have used any of the many RPC frameworks available.
This has been working for several years now.
This is not a generic solution but it's good enough for in-house communication between 2 systems that are maintained by different parts of the organization, where the API between them, that like I said is based on passing around char* buffers, has been more or less set in stone a long time ago. Conway's law [1] and all that. Sigh.
[0] We are a Windows shop although the same thing should work with DWARF info, same as the OP library works. In fact he says "It may never work on Windows, which does not use DWARF to encode debug info" but I can say that the same approach does work on Windows, for C++ at least. The PDB format might be a tad undocumented, but its documentation has been improved in the last decade or so since I started working on my library. Writing some small test programs is enough to understand how to access it, if all you need is meta info on C-style structures. Other stuff is more... challenging. But it wasn't necessary for my use-case.
Yeah, I know, it's our problem, not theirs. It's something I cannot fix on my own without a huge effort. I've tried pushing for it for more than a decade, and at some point my wish was sort of abducted by my boss' boss as an excuse to create a DSL [0]. This did solve some huge problems but also created many others. It didn't solve that char* / h file problem since it doesn't really have an FFI.
[0] Domain specific language, custom-made for our own internal users. I've come to hate DSLs since I have to support that one, which never wanted.
Regardless, quite an ingenious use of standard ELF features, I didn't think this would be possible in Rust without adding some kind of VM around reflection code.
* invokes local_type_id to get the memory address of your value’s static implementation of local_type_id
* maps that memory address to an offset in your application’s binary
* searches your application’s debug info for the entry describing the function at that offset
* parses that debugging information entry (DIE) to determine the type of local_type_id’s &self parameter.
This is a rather strange thing to bolt onto a language. I could see this as an external tool. The use case seems to be programs which used "async" so much they can't figure out the resulting state machine. External debug tools to view and examine the async state machine might be helpful.
My experience with Rust has been that debugging of safe code is just not a problem. Print statements and logging are enough.
Very impressive work!
Reflection seems more helpful when the programming language is little unsounded.
My crate is suitable for cases where you cannot know (or control) the set of types you might need to reflect on in advance. It's primary use-cases are related to debugging.
Compile time reflection AFAIK is available in D and Zig, and is planned for C++.
Ruts does not support inheritance either. But I have never missed either feature in a Rust program.
Thx OP for providing an example.
I don't think you understand how people in dynamic languages use any types at all.
Even if not, this is still very useful for debugging
[0]: https://doc.rust-lang.org/cargo/reference/profiles.html#rele...
It is an external tool. This is a crate, not a part of the compiler.
This seems like a valuable library. It’s impressive that it can be so powerful in a compiled language. C and C++ are much older but don’t have anything quite like this.
It can just be an extremely fun and cute demo, without practical application.
I don't have the context to judge the probability of that in this specific case (lots of technical nitty-gritty comes in to this sort of thing), but I've certainly seen similar things happen in other communities.
You can already script gdb to provide rich views of any data structure.
Rust has no run-time type information except limited downcasts via `dyn Any` or explicitly derived traits on per-type basis, and these features compile to type-specific monomorphic code rather than calling some run-time reflection.
I would however welcome static reflection with open arms. In Rust in particular, I’d prefer it if derive was implemented using static reflection, rather than proc macros.
In C++ with SFINAE you can effectively do CTTI-style programming in C++. C++ has long had runtime type reflection as well (RTTI), though it needs to be compiled in. Looks like there's a boost library for CTTI.
struct Foo<T>(T);
struct Fooi32(i32);
struct Foof64(f64);
Rust is not the only language that does this, to be clear.
If you're interested in a quick intro on the compiler side of this, you can read https://rustc-dev-guide.rust-lang.org/backend/monomorph.html
package main
import (
"fmt"
)
type wrapper[T any] struct {
Value T
}
func (w wrapper[T]) String() string {
return fmt.Sprintf("{%v}", w.Value)
}
func stringWrapped[T any](n int, v T) string {
if n == 0 {
return fmt.Sprintf("%v", v)
}
return stringWrapped(n-1, wrapper[T]{Value: v})
}
func main() {
n := 0
fmt.Scanf("%d", &n)
result := stringWrapped(n, "test")
fmt.Println(result)
}
[1]: https://play.rust-lang.org/?version=stable&mode=debug&editio...
[2]: https://play.rust-lang.org/?version=stable&mode=debug&editio...
^ This blows the stack because it keeps calling itself with no break condition, but shows how the type system accepted the code.
In C++ you can monomorphize as long as you can somehow prove the recursion terminates at compile time (for example by threading a static recursion counter).
And now the thing is: with transparent signature ascriptions, functors are monomorphised in SML, instead of everything being hidden behind signatures (as is in the case of Rust with traits when you use dyn), which has semantic consequences. E.g. a struct returned by a functor may contain a type. You can't perform proper type-checking without monomorphising, because you don't know what the exact type is. E.g. in the following program, the final line couldn't be type-checked without monomorphisation:
signature ITERABLE = sig
type ElemT
type SrcT
val new_iter: SrcT -> unit -> ElemT option
end
signature LIST_ELEM_TYPE = sig
type T
end
functor ListIterFun (ListElemType: LIST_ELEM_TYPE): ITERABLE = struct
type ElemT = ListElemType.T
type SrcT = ElemT list
fun new_iter l = let val lr = ref l
in
fn () => case !lr of
nil => NONE
| (x::xs) => (lr := xs; SOME x)
end
end
structure IntElemType: LIST_ELEM_TYPE = struct
type T = int
end
structure IntListIter = ListIterFun(IntElemType)
val next = IntListIter.new_iter [1, 2, 3, 4, 5]
There was a lot of discussion of this issue when Go was settling on a design for its generics, under the phrase "reified generics".
It's probably exactly how templates work, except the details are invisible to users.
https://hacks.mozilla.org/2020/11/warp-improved-js-performan...
Given
fn foo<T>(a: T, b: T) -> T { a + b }
error[E0369]: cannot add `T` to `T`
--> src/lib.rs:1:32
|
1 | fn foo<T>(a: T, b: T) -> T { a + b }
| - ^ - T
| |
| T
|
help: consider restricting type parameter `T`
|
1 | fn foo<T: Add<Output = T>>(a: T, b: T) -> T { a + b }
| +++++++++++++++++
[1]: https://play.rust-lang.org/?version=nightly&mode=debug&editi...
pub fn foo<T: AsRef<X>>(x: T) {
inner_foo(x.as_ref());
}
fn inner_foo(_: &X) { todo!() }
[1]: https://rustc-dev-guide.rust-lang.org/backend/monomorph.html...