Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add custom filter functions initial support #173

Merged
merged 1 commit into from
Jan 8, 2021

Conversation

quasilyte
Copy link
Owner

Custom filters are simple Go functions defined in ruleguard rule files
that can be used as Where clause predicates.

They're compiled to a bytecode that is then interpreted when
the matched rule needs to apply its filters. The performance is
good enough for now (faster than yaegi) and can be further
improved in the future.

There are various limitations in what can be used in custom
filters and what's not. It's not well-documented right now,
but the compile errors try to be helpful.
What can be compiled will be executed like a normal Go would.

One use case for this new feature can be found in #129

Here is the solution to #129 which is not possible:

	func implementsStringer(ctx *dsl.VarFilterContext) bool {
		stringer := ctx.GetInterface(`fmt.Stringer`)
		return types.Implements(ctx.Type, stringer) ||
			types.Implements(types.NewPointer(ctx.Type), stringer)
	}

	func stringerLiteral(m dsl.Matcher) {
		m.Match(`$x{$*_}`).
			Where(m["x"].Filter(implementsStringer)).
			Report(`$x implements fmt.Stringer`)
	}

Custom filters are more flexible than predefined filters, but they
generally require more coding. As a rule of thumb: they should be
used only when there is no matching builtin filter.

Note: right now many simple operations, like integer multiplication,
are not implemented. They can be added trivially, but I wanted to
present a working concept with a minimal amount of code for this first PR.
Upcoming changes that extend the supported features set are going
to be easier to review.

It's also possible to allow calling byte-compiled functions from other byte-compiled functions.
But it's not there yet (I also need examples where it can be applied to get a better
understanding of the subject).

@quasilyte quasilyte force-pushed the quasilyte/experimental_quasigo_ext branch from d1d55f7 to c81f921 Compare January 6, 2021 22:49
Copy link
Collaborator

@cristaloleg cristaloleg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

return func(params *filterParams) matchFilterResult {
// TODO(quasilyte): what if bytecode function panics due to the programming error?
// We should probably catch the panic here, print trace and return "false"
// from the filter (or even propagate that panic to let it crash).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other words: simple defer-recover routine?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. But I'm not sure yet what should that recovered panic do.
Silently ignoring them is a bad thing.

I think the panic trace won't show which byte-compiled function failed. It could be a good idea to annotate the panic with this information and re-panic with extra info.

`float32`: types.Typ[types.Float32],
`float64`: types.Typ[types.Float64],
`complex64`: types.Typ[types.Complex64],
`complex128`: types.Typ[types.Complex128],
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need all this types? or we can limit to int64 and/or float64 ?

Copy link
Owner Author

@quasilyte quasilyte Jan 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These types are needed so you can write:

ctx.GetType(`float32`)

   instead of

types.Typ[types.Float32]

In other words, it's needed for the filter implementation on the user side.

It's not 100% necessary as we could just export the same types.Typ map and all type kind constants, but we'll still need GetType for more complex cases:

// may dynamically load "fmt" package, finds `Stringer` type in it, caches the result
ctx.GetType(`fmt.Stringer`)

It also works for 3rd party packages with FQN:

// may dynamically load "github.com/foo/bar" package, finds `Baz` type in it, caches the result
ctx.GetType(`github.com/foo/bar.Baz`)

@quasilyte quasilyte force-pushed the quasilyte/experimental_quasigo_ext branch 5 times, most recently from a276b3f to beb322b Compare January 8, 2021 02:04
that can be used as Where clause predicates.

They're compiled to a bytecode that is then interpreted when
the matched rule needs to apply its filters. The performance is
good enough for now (faster than `yaegi`) and can be further
improved in the future.

There are various limitations in what can be used in custom
filters and what's not. It's not well-documented right now,
but the compile errors try to be helpful.
What can be compiled will be executed like a normal Go would.

One use case for this new feature can be found in #129

Here is the solution to #129 which is not possible:

```go
	func implementsStringer(ctx *dsl.VarFilterContext) bool {
		stringer := ctx.GetInterface(`fmt.Stringer`)
		return types.Implements(ctx.Type, stringer) ||
			types.Implements(types.NewPointer(ctx.Type), stringer)
	}

	func stringerLiteral(m dsl.Matcher) {
		m.Match(`$x{$*_}`).
			Where(m["x"].Filter(implementsStringer)).
			Report(`$x implements fmt.Stringer`)
	}
```

Custom filters are more flexible than predefined filters, but they
generally require more coding. As a rule of thumb: they should be
used only when there is no matching builtin filter.

Note: right now many simple operations, like integer multiplication,
are not implemented. They can be added trivially, but I wanted to
present a working concept with a minimal amount of code for this first PR.
Upcoming changes that extend the supported features set are going
to be easier to review.

It's also possible to allow calling byte-compiled functions from other byte-compiled functions.
But it's not there yet (I also need examples where it can be applied to get a better
understanding of the subject).

Also fixes #167 and #170
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants