Python type aliases don't use forward references

[giarc3]

In generated Python, our type aliases are sorted alphabetically and placed towards the end of the file. If there are nested aliases, this fails to compile when they're in the wrong order ("Foo" is not defined).

# Type alias
Bar = Foo
...
# Type alias
Foo = str

For function type hints, we use quotation marks to allow forward references (described here). I tried this for the type aliases, but got lots of warnings Variable not allowed in type expression and unknown types when they're used in functions.

I also tried changing the type aliases to

# Type alias
Bar: typing.TypeAlias = "Foo"
...
# Type alias
Foo: typing.TypeAlias = "str"

this worked well, but was only added in Python version 3.10 (is the minimum supported version for uniffi documented somewhere?).

My temporary solution is just to avoid nested aliases, but I have some complex types that would benefit from another solution to this problem.

[mhammond]

Type aliases are complicated in Python and we use strings exactly to avoid the problem with forward references. I don't understand what change you are proposing should be made to uniffi (ie, this seems a problem with Python in general and nothing specific to uniffi?)

[giarc3]

Sorry for not being clear. You said "we use strings exactly to avoid the problem with forward references" but we're getting compilation errors because we're not doing anything like that for aliases today, only for function types. But when I went down the path of dealing with the problem the same way, I ran into the "Variable not allowed in type expression" error.

So I can think of two possible ways to deal with this problem, but was hoping others could come up with something better:

1. Implement sorting of type aliases. Ideally it would be able to automatically sort based on references to other types, but an easier solution could be a way to manually indicate ordering.
2. Switch to typing.TypeAlias. This wouldn't work for pre-3.10 Python, which is why I was asking what the minimum supported version is today.

[mhammond]

We support all currently supported Python versions, which IIRC, is currently 3.8+ (even though CI is currently using 3.11)

I'm afraid I still don't have a good understanding of the problem you are hitting though - would it be possible to demonstrate it by making the smallest possible patch to one of our tests? Eg, test_simple_fns has one test, test_futures has another, but similar tests in any other fixture would be great too. We'd need such a test for any potential fix anyway, so this would help get the ball rolling.

[giarc3]

Sure, I opened a PR here #2068 to demonstrate the error in the proc-macro fixture, but I can move it to a different test if you'd prefer

[mhammond]

That's perfect, thanks!

[mhammond]

In that PR, you are trying to define a "custom type" in terms of another custom type. The custom type docs indicate that they are used to wrap a "builtin" - there's no intention they are able to wrap non-builtin types. In other words, the bug in that PR seems to be that we don't fail with a clear error about the wrapped type.

I'd like to better understand your use-case - in that example, why wouldn't your API just use CustomNewtype directly? ie, what value does ASecondCustomNewtype offer?

[giarc3]

I'd like to better understand your use-case

Sure! The example in the test was a minimal reproducible example; I definitely would just use CustomNewtype in that case.

My use case is mainly around composition of types in dicts. I have several functions with similar-looking signatures that are actually distinct and like to use newtypes to help distinguish this.

For example, I'd like to do

pub struct FieldId(pub String);
pub struct EncryptedField(pub Vec<u8>);
pub struct EncryptedFields(pub HashMap<FieldId, EncryptedField>);

When I use custom_newtype! on these 3 structs, it will generate aliases in alphabetical order and EncryptedFields won't yet know what a FieldId is.

[mhammond]

In other words, the bug in that PR seems to be that we don't fail with a clear error about the wrapped type.

Welp, I guess I jumped the gun there - we do support any type, not just "builtin" types. In the linked PR I updated the docs and added tests for some of these cases.

[giarc3]

That's fantastic, thanks so much for doing that @mhammond