Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add data sources to create and manage ingest processors #67

Merged
merged 47 commits into from
Feb 15, 2022

Conversation

olksdr
Copy link
Contributor

@olksdr olksdr commented Feb 14, 2022

This PR adds data sources to manage ingest processors for ingest pipelines

related #57
closes #56

@olksdr olksdr added enhancement New feature or request Elasticsearch Elasticsearch related APIs labels Feb 14, 2022
@olksdr olksdr self-assigned this Feb 14, 2022
@olksdr olksdr requested a review from Crazybus February 14, 2022 08:01
Crazybus
Crazybus previously approved these changes Feb 15, 2022
Copy link
Contributor

@Crazybus Crazybus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!


By default, the processor uses the GeoLite2 City, GeoLite2 Country, and GeoLite2 ASN GeoIP2 databases from MaxMind, shared under the CC BY-SA 4.0 license. Elasticsearch automatically downloads updates for these databases from the Elastic GeoIP endpoint: https://geoip.elastic.co/v1/database. To get download statistics for these updates, use the GeoIP stats API.

If your cluster can’t connect to the Elastic GeoIP endpoint or you want to manage your own updates, see Manage your own GeoIP2 database updates.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if "see Manage your own GeoIP2 database updates" should be a direct link to https://www.elastic.co/guide/en/elasticsearch/reference/current/geoip-processor.html#manage-geoip-database-updates. Just to be clear that it isn't something that is inside of this documentation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, even though we have the link to the doc, but I think it's a good idea to make a direct link as well here. Will change now


Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.

This processor comes packaged with many [reusable patterns](https://github.com/elastic/elasticsearch/blob/8.0/libs/grok/src/main/resources/patterns).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have an easy way to update all of the version references (in this case 8.0) so that these links don't go stale?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, but we can point to master and later to main when it get's renamed?

Copy link
Contributor

@Crazybus Crazybus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@olksdr olksdr merged commit cb4ac95 into elastic:main Feb 15, 2022
@olksdr olksdr deleted the feat/data-processors-1 branch February 15, 2022 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Elasticsearch Elasticsearch related APIs enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Support for ingest pipelines
2 participants