How to know if a CommandParam is a pointer?

[MarijnS95]

Currently migrating some more of ash to vk-parse to use a new property. During implementation it seems there's no easy way to know if a CommandParam is a pointer type, because vk.xml has the * outside its <type> element. It's sometimes - but not always - possible to assume pointers when len.is_some().

Consider the following XML:

<command queues="graphics,compute" renderpass="both" cmdbufferlevel="primary,secondary">
    <proto><type>void</type> <name>vkCmdPushDescriptorSetWithTemplateKHR</name></proto>
    <param externsync="true"><type>VkCommandBuffer</type> <name>commandBuffer</name></param>
    <param><type>VkDescriptorUpdateTemplate</type> <name>descriptorUpdateTemplate</name></param>
    <param><type>VkPipelineLayout</type> <name>layout</name></param>
    <param><type>uint32_t</type> <name>set</name></param>
    <param noautovalidity="true">const <type>void</type>* <name>pData</name></param>
</command>

This results in:

CommandDefinition {
    queues: Some(
        "graphics,compute",
    ),
    successcodes: None,
    errorcodes: None,
    renderpass: Some(
        "both",
    ),
    cmdbufferlevel: Some(
        "primary,secondary",
    ),
    pipeline: None,
    comment: None,
    proto: NameWithType {
        type_name: Some(
            "void",
        ),
        name: "vkCmdPushDescriptorSetWithTemplateKHR",
    },
    params: [
        ...
        CommandParam {
            len: None,
            altlen: None,
            externsync: None,
            optional: None,
            noautovalidity: Some(
                "true",
            ),
            objecttype: None,
            definition: NameWithType {
                type_name: Some(
                    "void",
                ),
                name: "pData",
            },
            validstructs: [],
        },
    ],
    alias: None,
    description: None,
    implicitexternsyncparams: [],
    code: " void  vkCmdPushDescriptorSetWithTemplateKHR ( VkCommandBuffer  commandBuffer ,  VkDescriptorUpdateTemplate  descriptorUpdateTemplate ,  VkPipelineLayout  layout ,  uint32_t  set , const  void*  pData );",
}

Notice how there's nothing on pData telling that it is a pointer, not even a member containing the raw text inside <param>. There's a code member on the entire command - containing the entire command as a C function declaration, which would be very tedious to parse by hand.

[krolli]

Ah, my "favorite" part of vk.xml format. Part that I put off because even if I solved it, everyone was using the vkxml crate interface that I couldn't change and expose the data.

There's a code member on the entire command - containing the entire command as a C function declaration, which would be very tedious to parse by hand.

That code member is actually part of my planned solution, which was to parse the whole declaration and provide that data in more structured format (which I have not settled upon yet). I think it seemed easier to me at the time to just put whole function declaration into single string, instead of making separate string with each parameter. Even now, this approach seems better to me, as it would then be able to handle structs and function pointers more easily as well.

Started work on the parser is in c.rs where you can see simple lexer and tokenizer. What is still needed:

• coming up with representation for results of parsing that would be reasonably easy to work with
• turning token stream into this representation
• exposing this parsed data
• adding parsing logic to places where XML is mingled with C code (eg. function pointers, structs, commands), probably on as-needed basis.

Unfortunately, I'm a bit busy with other stuff these days and it's hard to say when I will be able to finish this part. But in my head, this is one of the major missing pieces on the road to 1.0.

[MarijnS95]

I guess most are trying to move off of vkxml as it doesn't get any updates anymore (even if we were to open a PR) and I don't want to maintain a codepath that takes in both a vkxml and vk_parse type, which is why I'm running into this while trying to use the newly added validstructs attribute.

To me it'd be great if we could at least have a code member for every param instead of having to manually pull these out of code for the entire command, so that the user can decide what they need to parse (after all, the text contained in <param> is already exactly what one should need) but in the end that's rather trivial to parse compared to understanding and representing the types as you rightfully state.

I'm also at a massive lack for time now and pretty much always, so will have to decide how to proceed here with validstructs support without blocking massively on this or having to refactor a lot when it comes online.

[krolli]

My main worry is that tags in xml might not enclose all of the declaration correctly. As I remember, vk.xml processing was just taking those parts and removing xml tags while leaving text in between them as contents of vulkan.h, so there is nothing ensuring that tags are used consistently and correctly (correctly for our purposes, that is). I'm also unsure about some more complicated declarations, such as member of a struct being function pointer. Those are quite rare and maybe won't be in issue in practice, though.

I think you're right and we can try adding code field for every param or member in addition to code for entire declaration. If you then use those to generate code, any wrong uses should (hopefully) trigger some kind of error. At the time of writing the parser, I didn't have a good way to validate these things, but that is different now.

Hopefully I manage to find time to do it in upcoming days.

[MarijnS95]

My main worry is that tags in xml might not enclose all of the declaration correctly. As I remember, vk.xml processing was just taking those parts and removing xml tags while leaving text in between them as contents of vulkan.h, so there is nothing ensuring that tags are used consistently and correctly (correctly for our purposes, that is).

If it isn't, that may be considered a spec bug? It seems there's only one way to find out: support this and run it through some (the same?) code that (currently) parses the types and names?

I'm also unsure about some more complicated declarations, such as member of a struct being function pointer. Those are quite rare and maybe won't be in issue in practice, though.

Also only one way to find out :)

any wrong uses should (hopefully) trigger some kind of error. At the time of writing the parser, I didn't have a good way to validate these things, but that is different now

That's how we (Ash) have been going at this so far. Some generator/compilation failures were our own, others were invalid spec. I managed to integrate Ash code generation and compile-testing into Khronos' Vulkan CI (but infallible) if only to give an indication of the current state of the spec.

[krolli]

Commit 615ffb6 adds code to command params. Looking at the xml structure, I honestly can't remember why I chose to have just code on whole command, as there doesn't seem to be anything outside proto and param elements. Maybe there was something in the past, but I'm not going to scrape through over two hundred versions to check.
Let me know how it works.

[MarijnS95]

Thanks. I don't have any code ready to parse code currently, but was working on a proof-of-concept to use vk-parse in part of ash to access new properties. Will see if I can add anything simple or (likely) if that's going to require major code changes.

[MarijnS95]

@krolli thanks for the code field! I've put the work in to create a simple ad-hoc parser for just these command parameters and it's working out flawlessly: ash-rs/ash#719 🥳

[krolli]

Glad to hear that. 👍