Test framework research effort

[MarcusFelling]

What type of testing should we focus on?

Types of tests:

1. Static analysis - test code without deploying it

   • compiler
   • linter: will be covered by Issue [Story] Provide a linter for Bicep #428
   • dry run: can be accomplished via what-if

2. Unit tests - test a single unit of code in isolation

   • test single module in isolation from Azure

     Note: what value is this when infra code is all about talking to Azure?

3. Integration tests - deploy real infrastructure and run validations

   • test single, deployed, module
   • examples: check outputs against expected values. Verify HTTP request returns 200.

4. End to end tests - test entire infra works together

   • test all modules deployed together

     Note: Can be slow and flaky, would need to be incremental somehow (dedicated environment). Can get expensive to deploy all resources.

[stan-sz]

Does this item cover testing the templates + parameters itself? E.g. scenario: having a template that uses some string-manipulation functions, I'd like to assert that for a given set of string inputs it handles it in an expected way. Think of it as unit tests for templates.

[bashscr]

I would like to see something similar to this as well. I just recently started using the ARM template test toolkit, so it would be immensely helpful to have a similar tool for Bicep.

[alex-frankel]

I just recently started using the ARM template test toolkit, so it would be immensely helpful to have a similar tool for Bicep.

@bashscr - just as FYI, this is going to be covered by #428 which will incorporate the TTK ruleset. This issue goes beyond the scope of what TTK covers.

[jesseloudon]

Having previously used the Molecule-Azure driver & test framework to develop new Ansible roles on Azure via IaC I’d ❤️ to see a Bicep equivalent to ‘molecule test’ e.g.

‘az bicep test’

https://ansible.readthedocs.io/projects/molecule/getting-started/#run-a-full-test-sequence

[cloudfright]

It would be great to be able to run a security analysis against the templates e.g. looking for Azure policy violations, embedded credentials etc. Right now, this can be done in the release pipeline, but it would be great to be able to catch these issues on the developer's machine before a commit was even made.

[slavizh]

It would be great to be able to deploy any resource available in Azure in some kind of pseudo environment. Once deployed you will be able to browse the resources and their configuration but on the backend they will not exist. For example if you have deployed SQL database you will not be able to connect to the SQL database as it is not there but you can see it as resource in that pseudo environment. Of course this kind of pseudo environment will have some limitations like for example you will be able to deploy custom script vm extension but the actual custom script will not be running as there will not be actual VM. The point of this pseudo environment would be to have over 90% success if you deploy the same code in actual Azure environment. Will also allow you to save cost and even time as there are some resources that are very expensive to deploy and keep running and others take quite some time to deploy.

Additionally what-if should become 100% accurate.

[SimonWahlin]

My biggest pain with ARM right now is probably the accuracy of what-if. Having a 100% accurate what-if would be golden!

I also agree with @slavizh here, another major pain point is the time it takes to test-deploy new templates and validate what happens on re-deploying the same template. For some scenarios, my workflow today can be:

1. Write a template
2. Deploy said template
3. Wait for 40 minutes for deployment to finish
4. Use the export API to capture the state of the resources deployed
5. Do what-if deployment
6. Deploy same template again
7. Export state again
8. Compare first state with second state and the difference between them with the what-if output

At this rate it can take half a day for an iteration. Would be super useful if I could deploy my own ARM mock API resource (this could involve a cost) to which I could deploy everything. Have my deployment being processed by actual resource providers and then be able to perform a get to see what was deployed (although nothing is actually deployed).

This would allow me to quickly deploy templates and verify the result without incurring a lot of cost, which would be amazing!

[sam-cogan]

I'll echo some of the other statements, that we need to make the use of What-If both accurate, but also easy to use as part of a testing framework. The more tests that can run using this, rather than having to actually create the actual infrastructure, the faster tests can run and get feedback.

Obviously some tests will have to create infrastructure, but the less that need to the better and faster tests will be.

[Apiformes]

Test the deployment locally based on the yml pipeline file. Sometimes the yml file is wrong. Some times what-if is successful only the real deployment fails. And would be great if there is a way to get feedback if manual resources or changes to existing resources are done so they can be added to the code.

[StefanIvemo]

What I'm struggling with the most in my day-to-day work is the inaccuracy of What-If. I run What-If deployments before each deployment and also in pipelines triggered by PRs as a first check of the committed code. It generates way to much noise today.

What @slavizh is suggesting above is very interesting, being able to mock-deploy resources would be super useful. The time it takes to deploy some service (and remove them) can be very long, and if we can speed this up without actually deploying real resources it would really help. Take a service like Virtual WAN and all it's dependent services, it takes a long time to deploy and is very important for my entire Azure platforms functionality. I need to be sure that the templates I deploy with changes don't mess things up. If I could deploy them into a pseudo environment and verify that things actually work it would be great.

[maikvandergaag]

I think, 1 and 3 would be great to have. Some kind of security validation should also be great against the policies and security center. Besides that but that has already been mentioned several times the What-if should be 100% accurate.

[MaximRouiller]

Just want to make to ensure one thing. The What-If should be used for actual validation against existing resources in a resource group or resource unicity on Azure (webapps, domains, etc.)

Actual issues like name validation (storage, web app names, etc.) should actually be done client-side as much as possible.

While the What-If preflight check will validate everything, the experience is subpar for users.

[alex-frankel]

Actual issues like name validation (storage, web app names, etc.) should actually be done client-side as much as possible.

This is one of the things in swagger we are not taking advantage of yet, but need to be. Tracked in #1679

[BernieWhite]

@MarcusFelling I think something that provides extensibility for the community to expand on would be a winner. Something that can be injected in the build process.

Just dropped Bicep support for PSRule for Azure (200+ WAF aligned rules) over here: https://azure.github.io/PSRule.Rules.Azure/using-bicep/

Currently my implementation wraps bicep build (a NuGet that implements core build/ emitter would be nice).

Ideally, I'd like to be able to get the emitted template but with a source map so that any line references are relevant. Currently I can only return line references to a compiled template, not bicep file.

[anthony-c-martin]

with a source map so that any line references are relevant

There's some on-going work for #855 at the moment - stay tuned!

I think something that provides extensibility for the community to expand on would be a winner. Something that can be injected in the build process.

Would you mind expanding a bit on what you'd like this extensibility to look like? Are you referring to the ability to implement custom linter analyzers, or something else?

[BernieWhite]

@anthony-c-martin Yes a custom analyzer. Love what the team has already done with the linting in the VSCode extension. My main goal here is to have a way to provide feedback to a user that they are configuring a Azure resource in a insecure or problematic way using static code analysis. Ideally providing feedback as close to the code as possible.

An API or custom visitor to get a resource graph and linked child/ sub-resources out would be great.

[codepic]

Since AzSK is in Sunset in favor of AzTS which is more of an Continuous Assurance tool, it leaves the need for Security Intellisense, Security Verification Tests and CI/CD validation before anything ends up in any subscriptions.

I've been involved in DevOps-projects in global organizations and without security intellisense, security verification testing and CI/CD integration with organization policies, it becomes very expensive to maintain good security. A typical workflow would be reality:

1. Developer pushes changes into development environment
2. At some point, the code ends up in QA
3. Security department has a daily scan of this environment which spits out a report of stuff that needs fixing
4. Developers would go and fix what they can
5. But they will not know if they actually fixed the issue or introduced new ones until the next daily scan
6. A meeting would ensue in which people go through a security report in Excel (duh!) during which time, no meaningful work can be done

To get away from this vicious cycle, I've been introducing AzSK with Organization policies so that developers would know before any deployment that there's an issue that needs fixing thus giving them a chance to avoid aforementioned meetings.

An optimal scenario would be:

1. The ability to log into the Azure Tenant in Visual Studio Code
2. Selecting a target subscription/environment/resourcegroup as the security baseline (AzTS policies perhaps?)
3. VSCode intellisense showing warnings based on findings as you type
4. An Azure Devops extension to do Security Verification Testing during CI/CD
5. Ability to override the Error/Warning/Information level of findings in the pipeline (greenfield development may be doing stuff that security has not yet been able to catch up on
6. Azure DevOps dashboard integration to provide transparency on findings, what need fixing pre-release, what needs to be know etc.

[christiaan-janssen]

Is there any update? We are looking at using Bicep for a large company, but really would like a test framework.

[alex-frankel]

We are just now getting started with some experiments on what this could look like. cc @polatengin who is helping to lead that effort and can share more details (if we have them yet).

[cloudfright]

@polatengin do you have anything you're able to share please? This is a hot topic for Bicep adoption! 🙏

[ghost]

Hi MarcusFelling, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! 😄 🦾

[polatengin]

Quick update, during late 2022 and early 2023 we updated benchpress project with;

• many Resource Types (full list can be found on https://github.com/Azure/benchpress/tree/main/Modules/BenchPress.Azure/Public)
• added authentication mechanism part of the framework, so, helper functions do the authentication first
• added canonical demo applications
• added documentation to all helper functions
• fine-tuned the production pipeline
• improved the developer experience
• made many small improvements

[ghost]

Hi MarcusFelling, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! 😄 🦾