CppSharp / Continue Mono/.NET bindings for QtSharp (Dimitar Dobrev)

Summary

This summer was not as successful for QtSharp itself as it was for its underlying generator of CppSharp. There are two main reasons for this.

One is that the very first task for QtSharp, type maps, required support for templates. They proved extremely difficult both to plan and to implement. This requirement was further complicated by the fact that support for templates had to be generalised so that it can enter CppSharp rather than just QtSharp. This way everyone can use them and even if I had hacked it just in QtSharp, sooner or later somebody would have had to double the effort by porting it away.

The second reason is that as I am also the maintainter of CppSharp, I had to spare some of my available time fixing users’ issues. CppSharp had reached production status even before the summer had started and many people relied on it for their bindings. It was impossible to constantly ignore them because that would have endangered the trust in the project and eroded its user base and therefore the project itself. And without it, QtSharp would have stopped functioning too.

As a result, of the 5 sub-tasks the first one is half-complete while the rest have yet to be started.

Work

As explained above, templates are what took most of my time and effort during the summer. I am going to explain the problem they present, the approach I came up with to solve it and the progress of its implementation so far.

Problem

Templates serve the same purpose as generic classes in .NET but they are very different in implementation. A simple template class looks like this:

template <typename T>
class DLL_API DependentValueFields : public Base<T>
{
public:
    class Nested
    {
    };
    DependentValueFields();
    ~DependentValueFields();
    DependentValueFields& returnInjectedClass();
    DependentValueFields returnValue();
    DependentValueFields operator+(const DependentValueFields& other);
    T getDependentValue();
    void setDependentValue(const T& value);
private:
    T field{};
    union {
        int unionField;
    };
};

C++ is a compiled language so it lacks a virtual machine to produce the necessary binary code for specialisations of templates. This means that any specialisations must be compiled in advance since an external C++ library cannot “know” what specialisations it’s going to need. Therefore when working in C++, specialisations are compiled to where they are used i.e. the programmer’s own code. This is obviously impossible in the case of CppSharp because there’s no C++ of the user’s.

Approach

Consequently a new approach is required. There were two major routes to take:

Manual - consists of implementing each template in the target language. This would work with any type arguments but it is a large amount of work, per template and per binding. In addition, interaction with the bound C++ is impossible because in order to pass such an object to native code, it still has to be properly marshalled and this is simply impossible with random type arguments. For these reasons, I have completely ruled this option out even though it would have worked for just QtSharp rather than CppSharp in general;
Automatic - consists of wrapping a subset of the specialisations, most often the ones already used in the wrapped library. This only provides partial support but it’s adequate for interacting with the C++ original. Because the place of the support of templates is in CppSharp rather than QtSharp as explained at the beginning, and this approach is universal as CppSharp is, this has been my choice. It has, however, the additional requirement to generate and compile C++ for all used specialisations. The reason is that C++ compilers only emit object code within the external library when specialisations are used for fields as an optimisation technique. In any other cases, such as parameters, specialisations are compiled to the user’s C++ which we are here forced to simulate ourselves.

Automatic, however, is too broad a term. There are many ways to automate binding of templates, some of them listed below:

Generate a separate class for each specialisation - this involves the syntax for regular rather than generic classes. It is the approach used by SWIG and other generators. I have ruled it out mostly because the resulting API looks and feels different than the original C++ and also forces on users knowledge about templates and their representation which they don’t necessarily need or care about. There’s one more reason not to use this approach and it is that it lacks polymorphism unless a fake base class is introduced for all specialisations, and even that would only work in some cases. This approach can only be applied to the marshalling structures and I have done so:

 [StructLayout(LayoutKind.Explicit, Size = 8)]
 public unsafe partial struct __Internalc__S_DependentValueFields__b
 {
     [FieldOffset(0)]
     internal byte field;

     [FieldOffset(4)]
     internal global::CSharp.DependentValueFields._.__Internal _;
 }

 [StructLayout(LayoutKind.Explicit, Size = 12)]
 public unsafe partial struct __Internalc__S_DependentValueFields____S_IndependentFields__I
 {
     [FieldOffset(0)]
     internal global::CSharp.IndependentFields.__Internal field;

     [FieldOffset(8)]
     internal global::CSharp.DependentValueFields._.__Internal _;
 }

 [StructLayout(LayoutKind.Explicit, Size = 8)]
 public unsafe partial struct __Internalc__S_DependentValueFields____S_T1
 {
     [FieldOffset(0)]
     internal global::CSharp.T1.__Internal field;

     [FieldOffset(4)]
     internal global::CSharp.DependentValueFields._.__Internal _;
 }

Generate one generic class with no methods in it and place all specialised methods in extensions - this is similar to the above but it has the advantage of providing a more natural syntax. On the other hand, since only the supported specialisations have extensions, the auto-completion would simply not show any members for unsupported ones. Virtual methods can be simulated too by calling regular protected internal methods in the class. That would be a step away from the real API but it’s acceptable because they are at least not public. Unfortunately, the approach also has disadvantages, minor (no properties as extensions may only be methods), major (lack of polymorphism just as in the the first approach) and decisive (impossible to implement interface which is crucial in many cases, such as IList for std::vector or QList). The last one caused me to rule this approach out too;
Generate one generic class with each method containing a switch by type arguments - in each switch case the respective native code for the matched specialisation is called. If the user has provided unsupported type arguments, he gets an exception at run-time which explains in detail what happened. It is quite inconvenient to receive no warning about such an issue at compilation time but such a disadvantage is not as big as the ones the other two approaches have so this is my final choice. It would have worked if “where” constraints for generics could specify multiple conditions chained with a logical “or” but this is unfortunately not supported by C# and .NET:

public IndependentFields()
    : this((void*) null)
{
    var __T = typeof(T);
    if (__T.IsAssignableFrom(typeof(int)))
    {
        ...
        global::CSharp.IndependentFields.__Internal.ctorc__S_IndependentFields__I((__Instance + __PointerAdjustment));
        return;
    }
    if (__T.IsAssignableFrom(typeof(bool)))
    {
        ...
        global::CSharp.IndependentFields.__Internal.ctorc__S_IndependentFields__b((__Instance + __PointerAdjustment));
        return;
    }
    if (__T.IsAssignableFrom(typeof(global::CSharp.T1)))
    {
        ...
        global::CSharp.IndependentFields.__Internal.ctorc__S_IndependentFields____S_T1((__Instance + __PointerAdjustment));
        return;
    }
    if (__T.IsAssignableFrom(typeof(string)))
    {
        ...
        global::CSharp.IndependentFields.__Internal.ctorc__S_IndependentFields____N_std_S_basic_string__C___N_std_S_char_traits__C___N_std_S_allocator__C((__Instance + __PointerAdjustment));
        return;
    }
    throw new ArgumentOutOfRangeException("T", string.Join(", ", new[] { typeof(T).FullName }), "global::CSharp.IndependentFields<T> maps a C++ template class and therefore it only supports a limited set of types and their subclasses: <int>, <bool>, <global::CSharp.T1>, <string>.");
}

Implementation

Once I had settled on the way to port them, I started work. For now templates are only supported in the C# generator. There were two major efforts, to make the generator capable of producing the generic syntax and to fix all cases of the various types of members, such as constructors, methods, properties and indexers. There was one problem which unfortunately forced me to partially involve the second automatic approach. I need to explain it with some C++ code:

template <typename T>
class HasMethodReturningDependentPointer
{
public:
    const T* returnDependentPointer();
};

Now imagine we specialise this using our generated C#:

HasMethodReturningDependentPointer<char>

In this case returnDependentPointer would return const char* in C++. However, const char* is represented as a string in C#. Since C# does not support pointers to type arguments, the returned const T* in C++ would be represented as just T in C#. So the C# wrapper of returnDependentPointer would return just char in the case of this specialisation which cannot be cast to string which is what const char* is represented as. The same goes in the other direction, parameters with a const T* type. This forces any methods taking or returning a pointer to a type argument to be moved to extension methods, one per specialisation. It might require revision in the future because of interfaces i.e. the very reason I ruled the second automatic approach out:

public unsafe static partial class BasicStringExtensions
{
    [StructLayout(LayoutKind.Explicit, Size = 0)]
    public partial struct __Internal
    {
        ...

        [SuppressUnmanagedCodeSecurity]
        [DllImport("Std-symbols", CallingConvention = global::System.Runtime.InteropServices.CallingConvention.Cdecl,
            EntryPoint="?c_str@?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@QEBAPEBDXZ")]
        internal static extern global::System.IntPtr CStr(global::System.IntPtr instance);
    }

    ...

    public static string CStr(this global::Std.BasicString<char, global::Std.CharTraits<char>, global::Std.Allocator<char>> @this)
    {
        var __arg0 = ReferenceEquals(@this, null) ? global::System.IntPtr.Zero : @this.__Instance;
        var __ret = __Internal.CStr(__arg0);
        return Marshal.PtrToStringAnsi(__ret);
    }
}

Remaining work

Once I overcame the major problems, work was a matter of running QtSharp and then fixing any invalid generated code as needed. Fortunately, each separate case could be fully tested so there are going to be no regressions. There are two major problems remaining:

Removal of invalid specialised methods from the generated C++ - bodies of templated functions may include method calls, most often operators, involving type arguments. This cannot always work because a type argument might lack the required members. A simple example is a template list class with an method which finds the index of an object by using comparison. This problem can be solved by using the clang::Sema class and its InstantiateFunctionDefinition. I would like to insert my special thanks to my mentor Joao Matos who had the brilliant idea of me compiling the offending code with my locally compiled Clang copy in order to see the right function in the stack trace. This function reports errors when the passed declaration cannot be compiled. So I can detect such functions early on and ignore then before they reach code generation. It has, however, yet to be done;
Support for type arguments external to the library the template itself is in - I can use Qt as an example. QList is in QtCore, QWidget is in QtWidgets which depends on QtCore. So QList<QWidget> has to be supported but it cannot directly. If it’s simply added to the switch case, the generated code will not compile because of missing internal representations. Therefore I am going to create dictionaries taking a key pointing to the specialisation (such as its Clang USR) and a lambda to call each function, with the functions located in the external library. This is also not completed yet.

Once the issues above are resolved, CppSharp is going to enjoy fully functional support for templates, which means QtSharp and any other binding would get them for free.