-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Maps] [File upload] Geojson upload #36410
[Maps] [File upload] Geojson upload #36410
Conversation
# Conflicts: # x-pack/plugins/maps/public/shared/layers/sources/source.js
…er add/next button
9ffccc0
to
0526dc4
Compare
Scratch that. There's a fix in place with this PR. Colors are now filtered so that transient layer colors are not factored in to the next default color option. |
💔 Build Failed |
💔 Build Failed |
retest |
💔 Build Failed |
retest |
💔 Build Failed |
Need to remove Uploaded file
|
…ad-integration # Conflicts: # yarn.lock
0526dc4
to
aaa5953
Compare
💔 Build Failed |
__kbn__feature_id__
from data. Its getting inserted into records
#38614
* Add file upload x-pack plugin * Clean up * Remove unneeded cluster config * Remove unneeded test * First pass basic telemetry (not connected). * Basic telemetry connected * Review feedback * Revise telemetry to use savedObjectRepository. Capture metrics on app and file types * Lots of cleanup, consolidation of logic * Clean up, reorg * Update telem tests and telem functions * Add back import data model * Clean up and update telemetry tests * Fix telemetry test issues and update corresponding code * Up chunk limit to 30 MB * Add file upload telemetry to saved objects management builder * Missing space * Add descriptive comments to dynamic keys in telemetry fields * [Maps] [File upload] Geojson upload (#36410) * Client side basics * File added and default named correctly * Connect transient layer removal to file import component * Simplify transient layer removal * Move file import ui over to new file upload plugin and make more generic * Add post-processing option to file upload. Make component json-specific * Add flag for source indexing * Revise import layer workflow to be separate from add layer workflow * Differentiate between normal sources and import sources. Add back layer add/next button * Update indexing boolean in component after file upload & parse * First pass rough indexing from maps working on specific use cases * Update parsing logic to handle geojson formats * Index pattern added following index creation * Pass onsuccess callback to file_upload to add layer to maps app * Handle mulltipolygon type in es geo utils * Add functionality to make es layer permanent and switch to edit panel * Add to index if exists or create new * Make plugin API more intuitive. Set up to handle remove action * Pass transient removal call through to file import * Clean up layer viewing logic for temp and perm layers * Remove change source from import screen * Add option to provide mappings array and pass geo_point and geo_shape to array * Add support for multiple mappings select and index naming * Match style of import file button to add layer for now * Remove duplicate case handling resulting from merge * Move geo processing logic over to file upload plugin for reuse * Remove old geo_shape formatting from geo_point code * Set default index data type. Remove unneeded stringify request logic * Check for custom processor object which contains function * Move file picker to separate component * Some cleaning. Add geojson clean & validate code * Catch file parsing errors and notify user * Disable index type if valid file not referenced * Set error messaging on invalid index name used * Add index pattern checking logic and error handling * Dynamically populate geo index options * Set index data type earlier in the workflow. Don't duplicate requests * Pass back index ready status from plugin and connect to layer next button * Increase max bytes to ~50MB * Don't parse files over max size & warn user. Also, remove toasts and warn similarly to other components * Uploaded file default label: 'fileToImport' -> 'Import File' * Expand out feature properties for mapping * Pass through telemetry data to plugin back-end * Clean up indexing flow. Separate creation of index patterns. Add new index pattern create callbacks * Pass back info for indexing failures but don't connect to UI yet * Fix telemetry test issues and update corresponding code # Conflicts: # x-pack/plugins/file_upload/server/telemetry/telemetry.test.ts * Add file upload telemetry to saved objects management builder * Missing space * Add descriptive comments to dynamic keys in telemetry fields * Divide up the import layer add workflow on the maps side and tweak to avoid layer color change * Fix bug affecting file preview replacement not updating coordinate index type * Remove index pattern. Organize effects. General clean up * preIndexTransform -> transformDetails * Update proptypes * Cleaning, organizing * Add index name guidelines. Show conditionally * Add file size, type guidance. Filter on file size, not chunk size. Small tweaks to file/index tips * Zoom to layer extent on preview of imported data * Revert "Revise import layer workflow to be separate from add layer workflow" This reverts commit 3b35f53. * Handle import file determination to accomodate add layer flow. Update card wording * Decompose addLayer component into smaller pieces. Add placeholder import progress component * Dynamically change footer button text for context * Move import card to top of sources * Get basic progress tracking in place * Allow second layer (indexed layer) to get removed. Reconnect file remove. Small UI tweaks * Add link to management for further index mods * Fix i18n failures * Add file parsing progress indicator & text * Reset importView on source reset * Add dynamic chunking to handle fluctuating data sizes common to geo features * Don't duplicate index request if request in flight * Modify json upload and import to use JS classes & react class component state * Transition remaining file_upload components over to class/component structure with state * Move functions into index_settings class * Review feedback * Add fetch with timeout * Split out import editor into separate component and clean up logic * Clean up add layer panel/button title logic * More cleanup * Tweaks to success and error handling flow * Handle success/error handling on add/view indexed data * Jump to indexing complete on error. Handle nothing returned to client app * Update name/location of source select css file * Update import source card border color and icon * Suggest name for index based upon file name * Add validation to auto-suggested index name/pattern * Use constants for geo_point and geo_shape in mappings * Update geojson upload card/description * Catch-all review feedback * Fix internationalization syntax errors * Review feedback * Get index names and patterns only when needed * Make addLayer async again to fix zoom to extent issue, waits on just the async syncDataForLayer function (which is async) * Remove panel description as derived state and shift to function * Remove geojson fit to extent for now * Remove unused class selector * Remove unneeded i18 wrapper for what's already passed in as an i18 element * Revise import state to be handle via redux. Some json upload args changes * Review feedback. Some cleanup and bug fixes * Roll back store actions changes and layer_addpanel changes related to color change * Follow scss file naming conventions for source_select * Review feedback * Restore clobbered layer_control view to master state. Add source_select css * Update import to use plugin local indexPatternService, not maps' * Review feedback, mostly i18n. Also add index to scss path * i18n translation updates * Assign error message to values rather than error object * Update getMapColors to filter out transient layer * Wrap Feature as FeatureCollection in Maps * Add jest tests for geo processing functions. Add fixes for single feature handling * i18n * Review feedback. Test cleanup/fixes * Update layer add panel footer logic to still show when source not selected * Fix issue of not recognizing MultiPoint type. Remove throw logic for now * Update telemetry with newly required placeholder function * Prevent external modification of nested geojson objects * i18n translation updates * Revert "Fix issue of not recognizing MultiPoint type. Remove throw logic for now" This reverts commit d692f91. * Revert "Prevent external modification of nested geojson objects" This reverts commit 0ea9fd3. * yarn.lock update * [File upload] Remove dynamic fields from mappings, code and telemetry test (#38902) * Remove dynamic fields from mappings, code and telemetry test * Add file-upload-telemetry to spaces and es archiver test mappings * Don't create telemetry saved object if none exists, create on first update instead * Back out es archiver mappings update
* Add file upload x-pack plugin * Clean up * Remove unneeded cluster config * Remove unneeded test * First pass basic telemetry (not connected). * Basic telemetry connected * Review feedback * Revise telemetry to use savedObjectRepository. Capture metrics on app and file types * Lots of cleanup, consolidation of logic * Clean up, reorg * Update telem tests and telem functions * Add back import data model * Clean up and update telemetry tests * Fix telemetry test issues and update corresponding code * Up chunk limit to 30 MB * Add file upload telemetry to saved objects management builder * Missing space * Add descriptive comments to dynamic keys in telemetry fields * [Maps] [File upload] Geojson upload (elastic#36410) * Client side basics * File added and default named correctly * Connect transient layer removal to file import component * Simplify transient layer removal * Move file import ui over to new file upload plugin and make more generic * Add post-processing option to file upload. Make component json-specific * Add flag for source indexing * Revise import layer workflow to be separate from add layer workflow * Differentiate between normal sources and import sources. Add back layer add/next button * Update indexing boolean in component after file upload & parse * First pass rough indexing from maps working on specific use cases * Update parsing logic to handle geojson formats * Index pattern added following index creation * Pass onsuccess callback to file_upload to add layer to maps app * Handle mulltipolygon type in es geo utils * Add functionality to make es layer permanent and switch to edit panel * Add to index if exists or create new * Make plugin API more intuitive. Set up to handle remove action * Pass transient removal call through to file import * Clean up layer viewing logic for temp and perm layers * Remove change source from import screen * Add option to provide mappings array and pass geo_point and geo_shape to array * Add support for multiple mappings select and index naming * Match style of import file button to add layer for now * Remove duplicate case handling resulting from merge * Move geo processing logic over to file upload plugin for reuse * Remove old geo_shape formatting from geo_point code * Set default index data type. Remove unneeded stringify request logic * Check for custom processor object which contains function * Move file picker to separate component * Some cleaning. Add geojson clean & validate code * Catch file parsing errors and notify user * Disable index type if valid file not referenced * Set error messaging on invalid index name used * Add index pattern checking logic and error handling * Dynamically populate geo index options * Set index data type earlier in the workflow. Don't duplicate requests * Pass back index ready status from plugin and connect to layer next button * Increase max bytes to ~50MB * Don't parse files over max size & warn user. Also, remove toasts and warn similarly to other components * Uploaded file default label: 'fileToImport' -> 'Import File' * Expand out feature properties for mapping * Pass through telemetry data to plugin back-end * Clean up indexing flow. Separate creation of index patterns. Add new index pattern create callbacks * Pass back info for indexing failures but don't connect to UI yet * Fix telemetry test issues and update corresponding code # Conflicts: # x-pack/plugins/file_upload/server/telemetry/telemetry.test.ts * Add file upload telemetry to saved objects management builder * Missing space * Add descriptive comments to dynamic keys in telemetry fields * Divide up the import layer add workflow on the maps side and tweak to avoid layer color change * Fix bug affecting file preview replacement not updating coordinate index type * Remove index pattern. Organize effects. General clean up * preIndexTransform -> transformDetails * Update proptypes * Cleaning, organizing * Add index name guidelines. Show conditionally * Add file size, type guidance. Filter on file size, not chunk size. Small tweaks to file/index tips * Zoom to layer extent on preview of imported data * Revert "Revise import layer workflow to be separate from add layer workflow" This reverts commit 3b35f53. * Handle import file determination to accomodate add layer flow. Update card wording * Decompose addLayer component into smaller pieces. Add placeholder import progress component * Dynamically change footer button text for context * Move import card to top of sources * Get basic progress tracking in place * Allow second layer (indexed layer) to get removed. Reconnect file remove. Small UI tweaks * Add link to management for further index mods * Fix i18n failures * Add file parsing progress indicator & text * Reset importView on source reset * Add dynamic chunking to handle fluctuating data sizes common to geo features * Don't duplicate index request if request in flight * Modify json upload and import to use JS classes & react class component state * Transition remaining file_upload components over to class/component structure with state * Move functions into index_settings class * Review feedback * Add fetch with timeout * Split out import editor into separate component and clean up logic * Clean up add layer panel/button title logic * More cleanup * Tweaks to success and error handling flow * Handle success/error handling on add/view indexed data * Jump to indexing complete on error. Handle nothing returned to client app * Update name/location of source select css file * Update import source card border color and icon * Suggest name for index based upon file name * Add validation to auto-suggested index name/pattern * Use constants for geo_point and geo_shape in mappings * Update geojson upload card/description * Catch-all review feedback * Fix internationalization syntax errors * Review feedback * Get index names and patterns only when needed * Make addLayer async again to fix zoom to extent issue, waits on just the async syncDataForLayer function (which is async) * Remove panel description as derived state and shift to function * Remove geojson fit to extent for now * Remove unused class selector * Remove unneeded i18 wrapper for what's already passed in as an i18 element * Revise import state to be handle via redux. Some json upload args changes * Review feedback. Some cleanup and bug fixes * Roll back store actions changes and layer_addpanel changes related to color change * Follow scss file naming conventions for source_select * Review feedback * Restore clobbered layer_control view to master state. Add source_select css * Update import to use plugin local indexPatternService, not maps' * Review feedback, mostly i18n. Also add index to scss path * i18n translation updates * Assign error message to values rather than error object * Update getMapColors to filter out transient layer * Wrap Feature as FeatureCollection in Maps * Add jest tests for geo processing functions. Add fixes for single feature handling * i18n * Review feedback. Test cleanup/fixes * Update layer add panel footer logic to still show when source not selected * Fix issue of not recognizing MultiPoint type. Remove throw logic for now * Update telemetry with newly required placeholder function * Prevent external modification of nested geojson objects * i18n translation updates * Revert "Fix issue of not recognizing MultiPoint type. Remove throw logic for now" This reverts commit d692f91. * Revert "Prevent external modification of nested geojson objects" This reverts commit 0ea9fd3. * yarn.lock update * [File upload] Remove dynamic fields from mappings, code and telemetry test (elastic#38902) * Remove dynamic fields from mappings, code and telemetry test * Add file-upload-telemetry to spaces and es archiver test mappings * Don't create telemetry saved object if none exists, create on first update instead * Back out es archiver mappings update
* [File upload] New plugin: file upload (#36404) * Add file upload x-pack plugin * Clean up * Remove unneeded cluster config * Remove unneeded test * First pass basic telemetry (not connected). * Basic telemetry connected * Review feedback * Revise telemetry to use savedObjectRepository. Capture metrics on app and file types * Lots of cleanup, consolidation of logic * Clean up, reorg * Update telem tests and telem functions * Add back import data model * Clean up and update telemetry tests * Fix telemetry test issues and update corresponding code * Up chunk limit to 30 MB * Add file upload telemetry to saved objects management builder * Missing space * Add descriptive comments to dynamic keys in telemetry fields * [Maps] [File upload] Geojson upload (#36410) * Client side basics * File added and default named correctly * Connect transient layer removal to file import component * Simplify transient layer removal * Move file import ui over to new file upload plugin and make more generic * Add post-processing option to file upload. Make component json-specific * Add flag for source indexing * Revise import layer workflow to be separate from add layer workflow * Differentiate between normal sources and import sources. Add back layer add/next button * Update indexing boolean in component after file upload & parse * First pass rough indexing from maps working on specific use cases * Update parsing logic to handle geojson formats * Index pattern added following index creation * Pass onsuccess callback to file_upload to add layer to maps app * Handle mulltipolygon type in es geo utils * Add functionality to make es layer permanent and switch to edit panel * Add to index if exists or create new * Make plugin API more intuitive. Set up to handle remove action * Pass transient removal call through to file import * Clean up layer viewing logic for temp and perm layers * Remove change source from import screen * Add option to provide mappings array and pass geo_point and geo_shape to array * Add support for multiple mappings select and index naming * Match style of import file button to add layer for now * Remove duplicate case handling resulting from merge * Move geo processing logic over to file upload plugin for reuse * Remove old geo_shape formatting from geo_point code * Set default index data type. Remove unneeded stringify request logic * Check for custom processor object which contains function * Move file picker to separate component * Some cleaning. Add geojson clean & validate code * Catch file parsing errors and notify user * Disable index type if valid file not referenced * Set error messaging on invalid index name used * Add index pattern checking logic and error handling * Dynamically populate geo index options * Set index data type earlier in the workflow. Don't duplicate requests * Pass back index ready status from plugin and connect to layer next button * Increase max bytes to ~50MB * Don't parse files over max size & warn user. Also, remove toasts and warn similarly to other components * Uploaded file default label: 'fileToImport' -> 'Import File' * Expand out feature properties for mapping * Pass through telemetry data to plugin back-end * Clean up indexing flow. Separate creation of index patterns. Add new index pattern create callbacks * Pass back info for indexing failures but don't connect to UI yet * Fix telemetry test issues and update corresponding code # Conflicts: # x-pack/plugins/file_upload/server/telemetry/telemetry.test.ts * Add file upload telemetry to saved objects management builder * Missing space * Add descriptive comments to dynamic keys in telemetry fields * Divide up the import layer add workflow on the maps side and tweak to avoid layer color change * Fix bug affecting file preview replacement not updating coordinate index type * Remove index pattern. Organize effects. General clean up * preIndexTransform -> transformDetails * Update proptypes * Cleaning, organizing * Add index name guidelines. Show conditionally * Add file size, type guidance. Filter on file size, not chunk size. Small tweaks to file/index tips * Zoom to layer extent on preview of imported data * Revert "Revise import layer workflow to be separate from add layer workflow" This reverts commit 3b35f53. * Handle import file determination to accomodate add layer flow. Update card wording * Decompose addLayer component into smaller pieces. Add placeholder import progress component * Dynamically change footer button text for context * Move import card to top of sources * Get basic progress tracking in place * Allow second layer (indexed layer) to get removed. Reconnect file remove. Small UI tweaks * Add link to management for further index mods * Fix i18n failures * Add file parsing progress indicator & text * Reset importView on source reset * Add dynamic chunking to handle fluctuating data sizes common to geo features * Don't duplicate index request if request in flight * Modify json upload and import to use JS classes & react class component state * Transition remaining file_upload components over to class/component structure with state * Move functions into index_settings class * Review feedback * Add fetch with timeout * Split out import editor into separate component and clean up logic * Clean up add layer panel/button title logic * More cleanup * Tweaks to success and error handling flow * Handle success/error handling on add/view indexed data * Jump to indexing complete on error. Handle nothing returned to client app * Update name/location of source select css file * Update import source card border color and icon * Suggest name for index based upon file name * Add validation to auto-suggested index name/pattern * Use constants for geo_point and geo_shape in mappings * Update geojson upload card/description * Catch-all review feedback * Fix internationalization syntax errors * Review feedback * Get index names and patterns only when needed * Make addLayer async again to fix zoom to extent issue, waits on just the async syncDataForLayer function (which is async) * Remove panel description as derived state and shift to function * Remove geojson fit to extent for now * Remove unused class selector * Remove unneeded i18 wrapper for what's already passed in as an i18 element * Revise import state to be handle via redux. Some json upload args changes * Review feedback. Some cleanup and bug fixes * Roll back store actions changes and layer_addpanel changes related to color change * Follow scss file naming conventions for source_select * Review feedback * Restore clobbered layer_control view to master state. Add source_select css * Update import to use plugin local indexPatternService, not maps' * Review feedback, mostly i18n. Also add index to scss path * i18n translation updates * Assign error message to values rather than error object * Update getMapColors to filter out transient layer * Wrap Feature as FeatureCollection in Maps * Add jest tests for geo processing functions. Add fixes for single feature handling * i18n * Review feedback. Test cleanup/fixes * Update layer add panel footer logic to still show when source not selected * Fix issue of not recognizing MultiPoint type. Remove throw logic for now * Update telemetry with newly required placeholder function * Prevent external modification of nested geojson objects * i18n translation updates * Revert "Fix issue of not recognizing MultiPoint type. Remove throw logic for now" This reverts commit d692f91. * Revert "Prevent external modification of nested geojson objects" This reverts commit 0ea9fd3. * yarn.lock update * [File upload] Remove dynamic fields from mappings, code and telemetry test (#38902) * Remove dynamic fields from mappings, code and telemetry test * Add file-upload-telemetry to spaces and es archiver test mappings * Don't create telemetry saved object if none exists, create on first update instead * Back out es archiver mappings update * Update zh-CN translations
Replaces #34748
Adds UI for geojson upload on top of the file upload plugin. Specifically, provides the following capabilities: