C++ Reflection for Fragment Engine

Reflection systems – allowing a program to inspect it’s own code at runtime – is extremely handy for game engines. For example, rather than hard coding a list of objects that can be spawned by the console or be selectable from the editor, the engine can just look for all classes that either subclass a particular type or have a particular flag. Another example would be for networking – and is something that I’ll be writing for some upcoming coursework – whereby instead of writing bespoke methods per class for networking, a programmer can just add a ‘replicated’ flag to a particular field and a generic method handles updating automatically. The same would work for Remote Procedure Calls (RPC) – just mark a method as an RPC method and the reflection engine (And some form of preprocessor) can handle it as appropriate. C# has a built in reflection system, as well as a very handy attribute system (You tag types, methods and fields as you define them in a pretty intuitive way). Sadly, Fragment Engine is built in C++ (Which may be ported to D, providing slightly more features here, but it’s C++ for now).

So, C++ is kind of barebones and doesn’t provide much fanciness with regards to introspection and reflection (Bundling type information into an exe would both bloat it massively and just annihilate security techniques). C++ has the typeid operator, which is handy and even works at runtime, but its main purpose is to test if two types are the same or not based on whatever internal name the compile gives types. We can’t find out anything about the types themselves from this, and as such, we have to make it ourselves. This system is partially inspired by the system in Unreal Engine 4, albeit with the aim to avoid the creation of a separate preprocessing tool for as long as possible.

Basic Reflection Engine Requirements

The engine I will be creating must do the following:

• Gathering of all type information for use at runtime – A future goal would to also support this through libraries (Mods and plugins, etc).
• Allow the spawning of any type stored within the engine’s database from a string identifier.
• Support getting and setting values from fields within supported types using a string identifier for that field (and potentially string version of the value, that must then be converted to the correct type).
• Support [recursive] serialization and deserialization of supported types to any format (JSON, XML, Raw binary).
• Allow fields to be tagged and have metadata applied to them. ie: Replicated, valid ranges, prevent serialization etc.
• All this must be done via preprocessor macros and C++ – no preprocessing tool or code generator until I really have to (I can probably use templating much more successfully for this, but for now it’s preprocessor only).

So, lets begin!

Handling Types

The first thing I need the reflection engine to do would be to gather and store a collection of types. While I would like to do something simple and restricted to the header alone, this didn’t turn out to be practical when I started (Although a recent juggling of the code might make this possible again soon). In any case, I add reflectable types via the following:

//Example.h
class Example : public FEClass
{
    FE_DECLARE_CLASS(Example, FEClass)
};
//Example.cpp
FE_DEFINE_CLASS(Entity)

Here, the class must inherit from FEClass or a class that eventually inherits from FEClass. FEClass acts as a base for all reflectable classes such that there is a common base type to spawn from, as well as (potentially) providing core methods for reflection purposes – A ToString() method comes to mind. FE_DECLARE_CLASS sets up a number of static methods and fields so that a programmer can get type information of a class by simply calling Example::GetReflectionInfo().

FE_DEFINE_CLASS is where the true magic (and horror) happens. This macro creates a new class that inherits from FEClassReflectionInfo – holding all type specific reflection info, as well as those of its fields – and sets an instance of this class to the static private variable created by FE_DECLARE_CLASS. This uses the fact that static constructors run before main() to ensure that all type information exists as the engine starts – the constructor runs a method defined in a protected portion of the base class to add itself to the main engine class. To add to the horror factor, I previously borrowed JavaScript’s concept of self-executing functions to run this code before main. Thank god that’s gone, eh?

Handling Fields

In the end, fields turned out to be easier than anticipated. As I wanted to hide all the reflection info construction stuff from the programmer (So they can’t add their own arbitrary fields to the type info instance and ruin everything, etc), I needed a way to call code that’s only visible in one place. This place turned out to be the constructor of the reflection info. Great, but how do I put code there that details field types? There’s no way to inject code, and you can’t add code elsewhere that adds field info to the type info – that means exposing an ‘Add’ method and trusting the programmer not to call it!

Actually, there is a sort of way to inject code. Preprocessor macros support variadic arguments (Basically, you can supply as many arguments as you want). Normal programming languages then bundle all these arguments into an array that the function can then iterate through. The C++ preprocessor…isn’t as powerful. There are some fancy tricks to loop through these arguments, but we don’t need them here. In fact, putting __VAR_ARGS__ somewhere in your macro’s definition just dumps out all the supplied arguments as is. Luckily, we only need to write code, so all we do is supply code!

FE_DEFINE_CLASS creates the type info class for a specific type. Therefore, instead of creating the macro as FE_DEFINE_CLASS(CLASS_NAME), we put FE_DEFINE_CLASS(CLASS_NAME, …), with the ellipses indicating that it’s varadic. Then we just dump VAR_ARGS at the bottom of the constructor, and we can supply code as parameters that we want to inject into this constructor. It feels hacky as hell, but it works, and ends up quite elegant. Now, I know by ‘code’, you may think I mean doing something like:

FE_DEFINE_CLASS(Example, AddField("fieldOne", "int");, AddField("fieldTwo", "float"))

And while that could work, it could get verbose. Instead, we introduce another macro! (Hooray?). This macro is called _FE_PROPERTY, with the underscore indicating that you shouldn’t call that macro directly. Instead, you should use either FE_PUBLIC_PROPERTY, FE_PRIVATE_PROPERTY and FE_READONLY_PROPERTY. These are all very similar, but set a few core flags. In Fragment Engine, I define a ‘public’ property as one that’s modifyable in an editor, as well as one that’s saved to and read from disk. A ‘private’ property is the same, but should be readonly or hidden in the editor, while a ‘readonly’ property is one that’s visible at runtime to debugging tools, but is not serialized at all in any form. This way of flagging properties will probably change soon, but it’s how it’s implemented now. These macros expand into a bunch of code that defines stuff like the field’s type, its offset and size in bytes and so forth. The actual _FE_PROPERTY macro looks like this:

#define _FE_PROPERTY(CLASS, NAME, PRIVATE, READONLY)    \\
AddProperty(L#NAME, MakeShared<FEReflectableProperty>(  \\
    L#CLASS,                                            \\
    L#NAME,                                             \\
    typeid(CLASS::NAME).name(),                         \\
    offsetof_s(CLASS, NAME),                            \\
    sizeof(CLASS::NAME),                                \\
    PRIVATE,                                            \\
    READONLY                                            \\
));                                                     \\

And is called like this:

FE_DEFINE_CLASS(Example,
    FE_PUBLIC_PROPERTY(Example, fieldOne)
    FE_PRIVATE_PROPERTY(Example, fieldTwo)
)

Note how the programmer doesn’t need to know or care about the type of the field, or any of the other deep down stuff like byte offsets and whatnot. This is stuff they would have to write if we didn’t have access to the magic wizardry of the preprocessor! With this, the code that FE_XYZ_PROPERTY expands into is dumped at the end of the constructor for the type info, and it just ‘works’. We have the byte offsets and field sizes, meaning that we can now iterate over these fields (in their raw form) and with some super dodgy casting, can get and set arbitrary data at runtime! Sounds like a security nightmare, and it probably is! Even so, if we write a deserializer, this can read in data from a file, spawn classes and populate their fields entirely automatically. If the deserializer handles textual data, it can just nab the type name (as a string) of a field from the type info, and pass the string to a converter to get the actual data out of it.

What’s Next?

Well, you’ve might have guessed that I’m not a fan of the FE_PUBLIC/PRIVATE/READONLY_PROPERTY system, and you’re right. I aim to change it to support any number of supported flags and settings, as well as potentially arbitrary ones defined by the programmer. This would be something in the form of:

FE_PROPERTY(Example, field1, PUBLIC, READONLY, PROP_META("description", "An example field"), REPLICATED)
FE_PROPERTY(Example, field2, PRIVATE, PROP_RANGE(5, 15))

Everything after the field name would then be a set of defines containing more code. This would then likely add and change properties on the result of AddProperty (once I make it return the new FEReflectableProperty that’s passed into the second parameter).

In addition, there’s no specific per-instance reflection (as in, the network system can’t find out what values have been changed on a particular instance yet, unless it were to store a copy of that entity’s state at the previous time of sending data). This may either be implemented as another secondary structure, or a copy of the type info might assigned to each new instance of a reflectable class (They must be created via the Reflection Engine rather than through ‘new’ at the moment, so assignment can either be handled there, or the base constructor of FEClass).

Summary

Overall, this reflection system at the moment is very simple but very powerful, and it’s feature set will only improve in the future. I’ve only provided a very simple summary of what I’ve done at the moment, but as this system matures, expect follow up posts providing more in depth details on the intricacies. Even so, I feel that I’ve provided a basic explanation that you may use to create your own reflection engine if you so desire.