Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beta support for Swift Testing, and other improvements. #867

Merged
merged 50 commits into from
Jul 8, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
2d45183
wip
mbrandonw Jul 3, 2024
3cda2a7
wip
mbrandonw Jul 3, 2024
e71cde1
wip
mbrandonw Jul 3, 2024
822361a
wip
mbrandonw Jul 3, 2024
4201a54
wip
mbrandonw Jul 3, 2024
2408cc7
tests
mbrandonw Jul 3, 2024
8f8da79
wip
mbrandonw Jul 3, 2024
3e37ab0
migration guide
mbrandonw Jul 4, 2024
1efdcec
wip
mbrandonw Jul 4, 2024
b479fa4
wip
mbrandonw Jul 4, 2024
7760adf
wip
stephencelis Jul 4, 2024
7ad8430
Update Sources/SnapshotTesting/AssertSnapshot.swift
stephencelis Jul 4, 2024
cf5666e
wip
mbrandonw Jul 4, 2024
9a6f879
formatting
mbrandonw Jul 4, 2024
104d43f
wip
stephencelis Jul 4, 2024
d69e681
format
stephencelis Jul 4, 2024
9125d57
more
stephencelis Jul 4, 2024
0b9826b
wip
stephencelis Jul 4, 2024
ca6f4dc
fix
stephencelis Jul 4, 2024
d8fecd2
Merge remote-tracking branch 'origin/main' into swift-testing
stephencelis Jul 4, 2024
d84b345
Merge remote-tracking branch 'origin/main' into swift-testing
stephencelis Jul 4, 2024
ac36312
Make record mode opaque.
mbrandonw Jul 4, 2024
246c2e1
more docs
mbrandonw Jul 4, 2024
9d2c475
wip
mbrandonw Jul 4, 2024
9810df4
wip
mbrandonw Jul 4, 2024
0c6b449
Added new 'failed' record strategy, and wrote some tests.
mbrandonw Jul 4, 2024
a30e1ad
remove test artificats
mbrandonw Jul 4, 2024
54945d9
wip
mbrandonw Jul 4, 2024
f18aa61
more docs
mbrandonw Jul 4, 2024
f78bb29
fix linux tests
mbrandonw Jul 4, 2024
4463f37
more test fixes
mbrandonw Jul 4, 2024
6b5147e
test clean up
mbrandonw Jul 4, 2024
a3dd71d
debugging
mbrandonw Jul 4, 2024
6031605
debug
mbrandonw Jul 4, 2024
7617c60
wip
mbrandonw Jul 4, 2024
b87309a
fix
mbrandonw Jul 4, 2024
2de533c
fix tests
mbrandonw Jul 4, 2024
ef4f24e
wip
stephencelis Jul 4, 2024
aaf6ce4
wip
stephencelis Jul 4, 2024
f98a7e7
wip
stephencelis Jul 4, 2024
30c3469
make snapshot configuration optional
mbrandonw Jul 8, 2024
50e71d5
make snapshot configuration optional
mbrandonw Jul 8, 2024
42feef6
clean up
mbrandonw Jul 8, 2024
bd68b8f
fix
mbrandonw Jul 8, 2024
3eba8e2
indent
mbrandonw Jul 8, 2024
9a5361e
typo
mbrandonw Jul 8, 2024
c4842c3
more clean up
mbrandonw Jul 8, 2024
2dc8dab
record before difftool
mbrandonw Jul 8, 2024
ea63fe0
more tests
mbrandonw Jul 8, 2024
f4227b3
clean up test code
mbrandonw Jul 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
migration guide
  • Loading branch information
mbrandonw committed Jul 4, 2024
commit 3e37ab028ccb7c6fad089b09b85af50c8e77e00a
62 changes: 32 additions & 30 deletions Sources/InlineSnapshotTesting/AssertInlineSnapshot.swift
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,10 @@ import Foundation
line: UInt = #line,
column: UInt = #column
) {
let record = (isRecording == true ? .all : isRecording == false ? .missing : nil)
?? SnapshotTestingConfiguration.current?.record
?? _record
let record =
(isRecording == true ? .all : isRecording == false ? .missing : nil)
?? SnapshotTestingConfiguration.current?.record
?? _record
withSnapshotTesting(record: record) {
let _: Void = installTestObserver
do {
Expand All @@ -62,15 +63,15 @@ import Foundation
break
case .timedOut:
recordIssue(
"""
Exceeded timeout of \(timeout) seconds waiting for snapshot.

This can happen when an asynchronously loaded value (like a network response) has not \
loaded. If a timeout is unavoidable, consider setting the "timeout" parameter of
"assertInlineSnapshot" to a higher value.
""",
file: file,
line: line
"""
Exceeded timeout of \(timeout) seconds waiting for snapshot.

This can happen when an asynchronously loaded value (like a network response) has not \
loaded. If a timeout is unavoidable, consider setting the "timeout" parameter of
"assertInlineSnapshot" to a higher value.
""",
file: file,
line: line
)
return
case .incorrectOrder, .interrupted, .invertedFulfillment:
Expand Down Expand Up @@ -101,33 +102,34 @@ import Foundation

var failure: String
if syntaxDescriptor.trailingClosureLabel
== InlineSnapshotSyntaxDescriptor.defaultTrailingClosureLabel
== InlineSnapshotSyntaxDescriptor.defaultTrailingClosureLabel
{
failure = "Automatically recorded a new snapshot."
} else {
failure = """
Automatically recorded a new snapshot for "\(syntaxDescriptor.trailingClosureLabel)".
"""
Automatically recorded a new snapshot for "\(syntaxDescriptor.trailingClosureLabel)".
"""
}
if let difference = snapshotting.diffing.diff(expected ?? "", actual ?? "")?.0 {
failure += " Difference: …\n\n\(difference.indenting(by: 2))"
}
recordIssue(
"""
\(failure)
"""
\(failure)

Re-run "\(function)" to assert against the newly-recorded snapshot.
""",
file: file,
line: line
Re-run "\(function)" to assert against the newly-recorded snapshot.
""",
file: file,
line: line
)
return
}

guard let expected
else {
recordIssue("""
TODO: No expected value to assert against.
recordIssue(
"""
No expected value to assert against.
""",
file: file,
line: line
Expand All @@ -140,14 +142,14 @@ import Foundation

let message = message()
syntaxDescriptor.fail(
"""
\(message.isEmpty ? "Snapshot did not match. Difference: …" : message)
"""
\(message.isEmpty ? "Snapshot did not match. Difference: …" : message)

\(difference.indenting(by: 2))
""",
file: file,
line: line,
column: column
\(difference.indenting(by: 2))
""",
file: file,
line: line,
column: column
)
} catch {
recordIssue("Threw error: \(error)", file: file, line: line)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Migrating to 1.17

Learn how to use the new `withSnapshotTesting` tool for customizing how snapshots are generated and
diffs displayed.

## Overview

This library is under constant development, and we are always looking for ways to simplify the
library, and make it more powerful. This version of the library has deprecated some APIs,
introduced a new APIs, and includes beta support for Swift's new native testing library.

## Customizing snapshots

Currently there are two global variables in the library for customizing snapshot testing:

* ``isRecording`` determines whether new snapshots are generated and saved to disk when the test
runs.
* ``diffTool`` determines the command line tool that is used to inspect the diff of two files on
disk.

These customization options have a few downsides currently.

* First, because they are globals they can easily bleed over from test to test in unexpected ways.
And further, Swift's new testing library runs parallel tests in the same process, which is in stark
contrast to XCTest, which runs parallel tests in separate processes. This means there are even more
chances for these globals to bleed from one test to another.
* And second, these options aren't as granular as some of our users wanted. When ``isRecording``
is true snapshots are generated and written to disk, and when it is false snapshots are not
generated, _unless_ a file is not present on disk. The a snapshot _is_ generated. Some of our users
wanted an option between these two extremes, where snapshots would not be generated if the file
does not exist on disk.

And the ``diffTool`` variable allows one to specify a command line tool to use for visualizing
diffs of files, but only works when the command line tool accepts a very narrow set of arguments,
e.g. `ksdiff /path/to/file1.png /path/to/file2.png`.

We have greatly improved upon all of these problems by introducing the new
``withSnapshotTesting(diffTool:record:operation:)-7xmhk`` tool for customizing snapshots. It
allows you to customize how the `assertSnapshot` tool behaves for a well-defined scope.

Rather than overriding ``isRecording`` or ``diffTool`` directly in your tests, you can wrap your
test in `withSnapshotTesting`:

```swift
withSnapshotTesting(diffTool: .ksdiff, record: .all) {
// Assertions in here
}
```

If you want to override the options for an entire test class, you can override the `invokeTest`
method of `XCTestCase`:

```swift
class FeatureTests: XCTestCase {
override func invokeTest() {
withSnapshotTesting(diffTool: .ksdiff, record: .all) {
super.invokeTest()
}
}
}
```

And if you want to override these settings for _all_ tests, then you can implement a base
`XCTestCase` subclass and have your tests inherit from it.

Further, the `diffTool` and `record` arguments have extra customization capabilites:

* `diffTool` is now a [function](<doc:SnapshotTestingConfiguration/DiffTool-swift.struct>)
`(String, String) -> String` that is handed the current snapshot file and the failed snapshot file.
It can return the command that one can run to display a diff:

```swift
extension SnapshotTestingConfiguration.DiffTool {
static let compare = Self {
"compare \"\($0.path)\" \"\($1.path)\" png: | open -f -a Preview.app"
}
}
```

* `record` is now an [enum](<doc:SnapshotTestingConfiguration/Record-swift.enum>) with 3
choices: `all`, `missing`, `none`. When set to `all`, snapshots will be generated and saved to
disk. When set to `missing` only the snapshots that are missing from the disk will be generated
and saved. And when set to `none` snapshots will never be generated, even if they are missing.
This option is appropriate when running tests on CI so that re-tries of tests do not
surprisingly pass after snapshots are unexpectedly generated.

## Beta support for Swift's native testing library

This release of the library provides beta support for Swift's native testing library. Prior to this
release, using `assertSnapshot` in a `@Test` would result in a passing test no matter what. That
is because under the hood `assertSnapshot` uses `XCTFail` to trigger test failures, but that
does not cause test failures when using Swift Testing.

In version 1.17 the `assertSnapshot` helper will now intelligently figure out if tests are running
in an XCTest context or a Swift Testing context, and will determine if it should invoke `XCTFail`
or `Issue.record` to trigger a test failure.

For the most part you can write tests for Swift Testing exactly as you would for XCTest. However,
there is one major difference. Swift Testing does not (yet) have a substitute for `invokeTest`,
which we used alongside `withSnapshotTesting` to customize snapshotting for a full test class.

There is an experimental version of this tool in Swift Testing, called `CustomExecutionTrait`, and
this library provides such a trait called ``Testing/Trait/snapshots(diffTool:record:)``. It allows
you to customize snapshots for a `@Test` or `@Suite`, but to get access to it you must perform an
`@_spi(Experimental)` import of snapshot testing:

```swift
@_spi(Experimental) import SnapshotTesting

@Suite(
.snapshots(
diffTool: .ksdiff,
record: .all
)
)
struct FeatureTests {
}
```

That will override the `diffTool` and `record` options for the entire `FeatureTests` suite.
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Migration guides

Learn how to upgrade your application to the newest version of this library.

## Overview

This library is under constant development, and we are always looking for ways to simplify the
library, and make it more powerful. As such, we often need to deprecate certain APIs in favor of
newer ones. We recommend people update their code as quickly as possible to the newest APIs, and
these guides contain tips to do so.

> Important: Before following any particular migration guide be sure you have followed all the
> preceding migration guides.

### Guides

- <doc:MigratingTo1.17>
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Powerfully flexible snapshot testing.
### Essentials

- ``assertSnapshot(of:as:named:record:timeout:file:testName:line:)``
- <doc:MigrationGuides>

### Strategies

Expand Down
2 changes: 1 addition & 1 deletion Sources/SnapshotTesting/SnapshotTestingConfiguration.swift
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
/// For example, to specify to put an entire test class in record mode you do the following:
///
/// ```swift
/// class StringifyTests: XCTestCase {
/// class FeatureTests: XCTestCase {
/// override func invokeTest() {
/// withSnapshotTesting(record: .all) {
/// super.invokeTest()
Expand Down
Loading