Skip to content

Commit

Permalink
Feat/migrate elasticsearch src connector (Unstructured-IO#3174)
Browse files Browse the repository at this point in the history 
### Description
Migrate elasticsearch connector with support for what used to be batch
ingest docs but not it support for the download step to generate
additional file data.

---------

Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: rbiseck3 <rbiseck3@users.noreply.github.com>
  • Loading branch information
@rbiseck3 @ryannikolaidis
3 people authored Jun 13, 2024
1 parent ad69bdc commit f7b0a37
Show file tree
Hide file tree
Showing 28 changed files with 871 additions and 462 deletions.
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## 0.14.6-dev4
## 0.14.6-dev5

### Enhancements

Expand Down
130 changes: 65 additions & 65 deletions ...tput/elasticsearch/movies/0-57554198.json → ...tput/elasticsearch/movies-0-57554198.json
View file
Original file line number Diff line number Diff line change
@@ -1,107 +1,107 @@
[
{
"element_id": "0deeb41dfdab49b5df593a4ba334e9f5",
"type": "Title",
"element_id": "9cd9874c944c6a15749fe5767312a79a",
"text": "American",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "0",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "American",
"type": "Title"
"index_name": "movies",
"document_id": "0"
}
}
}
},
{
"element_id": "67937dd038457ef0f8870cb8f9d48f4e",
"type": "Title",
"element_id": "8f5aaeeb3adb7c714883dc505bb3e093",
"text": "Cecil Hepworth",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "0",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "Cecil Hepworth",
"type": "Title"
"index_name": "movies",
"document_id": "0"
}
}
}
},
{
"element_id": "1c55f514a4021d3678eecf237d6e741b",
"type": "NarrativeText",
"element_id": "12ccda2495f7762d017239f2350d19f0",
"text": "Alice follows a large white rabbit down a \"Rabbit-hole\". She finds a tiny door. When she finds a bottle labeled \"Drink me\", she does, and shrinks, but not enough to pass through the door. She then eats something labeled \"Eat me\" and grows larger. She finds a fan when enables her to shrink enough to get into the \"Garden\" and try to get a \"Dog\" to play with her. She enters the \"White Rabbit's tiny House,\" but suddenly resumes her normal size. In order to get out, she has to use the \"magic fan.\"",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "0",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "Alice follows a large white rabbit down a \"Rabbit-hole\". She finds a tiny door. When she finds a bottle labeled \"Drink me\", she does, and shrinks, but not enough to pass through the door. She then eats something labeled \"Eat me\" and grows larger. She finds a fan when enables her to shrink enough to get into the \"Garden\" and try to get a \"Dog\" to play with her. She enters the \"White Rabbit's tiny House,\" but suddenly resumes her normal size. In order to get out, she has to use the \"magic fan.\"",
"type": "NarrativeText"
"index_name": "movies",
"document_id": "0"
}
}
}
},
{
"element_id": "d4573238a68b34aca089a14b8c9dce33",
"type": "NarrativeText",
"element_id": "41fd36ef0e905da7713e8ddd106d0ce3",
"text": "She enters a kitchen, in which there is a cook and a woman holding a baby. She persuades the woman to give her the child and takes the infant outside after the cook starts throwing things around. The baby then turns into a pig and squirms out of her grip. \"The Duchess's Cheshire Cat\" appears and disappears a couple of times to Alice and directs her to the Mad Hatter's \"Mad Tea-Party.\" After a while, she leaves.",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "0",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "She enters a kitchen, in which there is a cook and a woman holding a baby. She persuades the woman to give her the child and takes the infant outside after the cook starts throwing things around. The baby then turns into a pig and squirms out of her grip. \"The Duchess's Cheshire Cat\" appears and disappears a couple of times to Alice and directs her to the Mad Hatter's \"Mad Tea-Party.\" After a while, she leaves.",
"type": "NarrativeText"
"index_name": "movies",
"document_id": "0"
}
}
}
},
{
"element_id": "f0521d4ea49ac1a578292a56ab061ff2",
"type": "NarrativeText",
"element_id": "799236a3e532626889744fdaa3a7c1b6",
"text": "The Queen invites Alice to join the \"ROYAL PROCESSION\": a parade of marching playing cards and others headed by the White Rabbit. When Alice \"unintentionally offends the Queen\", the latter summons the \"Executioner\". Alice \"boxes the ears\", then flees when all the playing cards come for her. Then she wakes up and realizes it was all a dream.",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "0",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "The Queen invites Alice to join the \"ROYAL PROCESSION\": a parade of marching playing cards and others headed by the White Rabbit. When Alice \"unintentionally offends the Queen\", the latter summons the \"Executioner\". Alice \"boxes the ears\", then flees when all the playing cards come for her. Then she wakes up and realizes it was all a dream.",
"type": "NarrativeText"
"index_name": "movies",
"document_id": "0"
}
}
}
}
]
78 changes: 39 additions & 39 deletions ...tput/elasticsearch/movies/1-57554198.json → ...tput/elasticsearch/movies-1-57554198.json
View file
Original file line number Diff line number Diff line change
@@ -1,65 +1,65 @@
[
{
"element_id": "7202f8ae8a26285a8a5eb189e776a211",
"type": "Title",
"element_id": "e9e2949adb0a1004997619eb751aaa52",
"text": "American",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "1",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "American",
"type": "Title"
"index_name": "movies",
"document_id": "1"
}
}
}
},
{
"element_id": "17827353dda8ff8a73b1163034a43134",
"type": "Title",
"element_id": "05a3d52e5ecf195049b8612341f75d61",
"text": "Wallace McCutcheon and Ediwin S. Porter",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "1",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "Wallace McCutcheon and Ediwin S. Porter",
"type": "Title"
"index_name": "movies",
"document_id": "1"
}
}
}
},
{
"element_id": "eca6c6dee49ceabe9001dca09d88370b",
"type": "NarrativeText",
"element_id": "181a299a05a094830d30f240349061d5",
"text": "Boone's daughter befriends an Indian maiden as Boone and his companion start out on a hunting expedition. While he is away, Boone's cabin is attacked by the Indians, who set it on fire and abduct Boone's daughter. Boone returns, swears vengeance, then heads out on the trail to the Indian camp. His daughter escapes but is chased. The Indians encounter Boone, which sets off a huge fight on the edge of a cliff. A burning arrow gets shot into the Indian camp. Boone gets tied to the stake and tortured. The burning arrow sets the Indian camp on fire, causing panic. Boone is rescued by his horse, and Boone has a knife fight in which he kills the Indian chief. [2]",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "1",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "Boone's daughter befriends an Indian maiden as Boone and his companion start out on a hunting expedition. While he is away, Boone's cabin is attacked by the Indians, who set it on fire and abduct Boone's daughter. Boone returns, swears vengeance, then heads out on the trail to the Indian camp. His daughter escapes but is chased. The Indians encounter Boone, which sets off a huge fight on the edge of a cliff. A burning arrow gets shot into the Indian camp. Boone gets tied to the stake and tortured. The burning arrow sets the Indian camp on fire, causing panic. Boone is rescued by his horse, and Boone has a knife fight in which he kills the Indian chief. [2]",
"type": "NarrativeText"
"index_name": "movies",
"document_id": "1"
}
}
}
}
]
78 changes: 39 additions & 39 deletions ...tput/elasticsearch/movies/2-57554198.json → ...tput/elasticsearch/movies-2-57554198.json
View file
Original file line number Diff line number Diff line change
@@ -1,65 +1,65 @@
[
{
"element_id": "0f4e168c7c67f7a998388d2a33dceb6e",
"type": "Title",
"element_id": "304a2118c16f40aaa72398eb7e4fe5b0",
"text": "American",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "2",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "American",
"type": "Title"
"index_name": "movies",
"document_id": "2"
}
}
}
},
{
"element_id": "6c8cc23e3f49b1324f9a6af33fc3ea5f",
"type": "Title",
"element_id": "5582d78d3be8d660dc39830d4dc9256f",
"text": "Unknown",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "2",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "Unknown",
"type": "Title"
"index_name": "movies",
"document_id": "2"
}
}
}
},
{
"element_id": "471f498be18c1e0c67baa1faf5f480e5",
"type": "NarrativeText",
"element_id": "5ee24dbcfbf63d973b671a46a2fd2d2e",
"text": "Before heading out to a baseball game at a nearby ballpark, sports fan Mr. Brown drinks several highball cocktails. He arrives at the ballpark to watch the game, but has become so inebriated that the game appears to him in reverse, with the players running the bases backwards and the baseball flying back into the pitcher's hand. After the game is over, Mr. Brown is escorted home by one of his friends. When they arrive at Brown's house, they encounter his wife who becomes furious with the friend and proceeds to physically assault him, believing he is responsible for her husband's severe intoxication. [1]",
"metadata": {
"languages": [
"eng"
],
"filetype": "text/plain",
"data_source": {
"version": "1",
"record_locator": {
"document_id": "2",
"hosts": [
"http://localhost:9200"
],
"index_name": "movies"
},
"version": 1
},
"filetype": "text/plain",
"languages": [
"eng"
]
},
"text": "Before heading out to a baseball game at a nearby ballpark, sports fan Mr. Brown drinks several highball cocktails. He arrives at the ballpark to watch the game, but has become so inebriated that the game appears to him in reverse, with the players running the bases backwards and the baseball flying back into the pitcher's hand. After the game is over, Mr. Brown is escorted home by one of his friends. When they arrive at Brown's house, they encounter his wife who becomes furious with the friend and proceeds to physically assault him, believing he is responsible for her husband's severe intoxication. [1]",
"type": "NarrativeText"
"index_name": "movies",
"document_id": "2"
}
}
}
}
]
Loading

0 comments on commit f7b0a37

Please sign in to comment.