12 Releases published by 1 person

• v0.12.2-beta.1

  published Jun 10, 2024

• v0.12.2-beta.2

  published Jun 13, 2024

• v0.12.2

  published Jun 13, 2024

• v0.12.3-beta.1

  published Jun 15, 2024

• v0.12.3

  published Jun 19, 2024

• v0.13.0-beta.1

  published Jun 24, 2024

• v0.13.0-beta.2

  published Jun 24, 2024

• v0.13.0

  published Jun 24, 2024

• v0.13.1-beta.1

  published Jun 25, 2024

• v0.14.0-beta.1

  published Jul 1, 2024

• v0.14.0-beta.2

  published Jul 4, 2024

• v0.14.0

  published Jul 6, 2024

71 Pull requests merged by 15 people

• feat: add scalar bitmap index to lance

  #2560 merged Jul 8, 2024

• feat: use hyperloglog for cardinality estimation for dictionary encoding

  #2555 merged Jul 8, 2024

• feat(python): expose fast search flag in python

  #2573 merged Jul 7, 2024

• feat: add inverted index

  #2526 merged Jul 7, 2024

• chore: skip tensorflow tests

  #2570 merged Jul 6, 2024

• fix: propagate storage_options to create_index for reading precomputed partition file

  #2568 merged Jul 5, 2024

• fix: failed to return index stats if the first partition is empty

  #2565 merged Jul 5, 2024

• fix: avoid multiple threads loading same index partition

  #2559 merged Jul 4, 2024

• fix: propagate shuffle_partition_concurrency from Python SDK

  #2564 merged Jul 4, 2024

• feat: allow users to create a file writer without a schema

  #2558 merged Jul 4, 2024

• ci: cargo check on various platforms

  #2545 merged Jul 4, 2024

• feat: reserve cores for IO

  #2496 merged Jul 3, 2024

• feat: support u8 for existing distance types

  #2501 merged Jul 3, 2024

• feat: create standalone IVF training API in python

  #2553 merged Jul 3, 2024

• feat: provide a fast ANN search plan

  #2554 merged Jul 2, 2024

• feat: add dictionary encoding to lance

  #2409 merged Jul 2, 2024

• refactor: remove generic PQ impl and trait

  #2529 merged Jul 2, 2024

• ci: prep for java release

  #2382 merged Jul 2, 2024

• fix: propagate storage_options to LanceFragment.create and LanceDataset.commit

  #2547 merged Jul 1, 2024

• chore: disable pprof on non-Linux targets

  #2523 merged Jun 29, 2024

• fix: build HNSW with wrong distance type

  #2533 merged Jun 28, 2024

• feat: make it possible to create v2 fragments using the fragment API

  #2534 merged Jun 27, 2024

• fix: show underlying cause of DynamoDB service error

  #2541 merged Jun 27, 2024

• feat: change to snapshot isolation

  #2537 merged Jun 27, 2024

• refactor: flat search to use datafusion top k

  #2535 merged Jun 27, 2024

• fix: pytorch dot inference pass unwanted parameters

  #2539 merged Jun 27, 2024

• fix: return correct sub index statistics

  #2532 merged Jun 27, 2024

• feat: stable row id support in queries

  #2452 merged Jun 26, 2024

• perf: avoid allocating bitvec for each insertion while indexing HNSW

  #2527 merged Jun 26, 2024

• chore: reuse the scheduler during reading shuffled partitions

  #2528 merged Jun 26, 2024

• ci: fix stepWithCompactEncoding error on MacOS

  #2531 merged Jun 25, 2024

• chore: upgrade aws-config for aarch64-pc-windows support

  #2510 merged Jun 25, 2024

• docs: artefact_management_doc

  #2504 merged Jun 25, 2024

• refactor: move IVF_HNSW_SQ & IVF_FLAT to new buliding & search path

  #2469 merged Jun 25, 2024

• feat: enhance binary array encoding, make it the default

  #2521 merged Jun 24, 2024

• feat: expose DatasetPreFilter, PreFilter, and FilterLoader to the public API

  #2522 merged Jun 24, 2024

• refactor: combined capacity updates and decoding for all physical decoders

  #2508 merged Jun 24, 2024

• feat: enable aarch64-pc-windows builds

  #2512 merged Jun 24, 2024

• fix: allow row id only in v2

  #2481 merged Jun 24, 2024

• chore(java): enable separate java builds

  #2457 merged Jun 23, 2024

• perf: use faster kmean find partition routing for pq assignment

  #2515 merged Jun 23, 2024

• fix: dot distance so kmeans can converge

  #2509 merged Jun 22, 2024

• ci: fix pytorch CI failure in arm64 jobs

  #2507 merged Jun 21, 2024

• docs(python): note multiprocessing incompatibility

  #2506 merged Jun 21, 2024

• chore: make flat storage and hnsw build accept dot distance type

  #2499 merged Jun 21, 2024

• docs: flickr8k_dataset_creation_example

  #2489 merged Jun 21, 2024

• chore: remove unused code

  #2494 merged Jun 21, 2024

• feat: support stable row ids in Dataset::take_rows()

  #2447 merged Jun 21, 2024

• chore: replace legacy constants with new ones

  #2495 merged Jun 20, 2024

• feat: convert binary logical encoding/decoding to physical array encoding/page decoding

  #2426 merged Jun 20, 2024

• perf: don't write precomputed partitions to trace file

  #2493 merged Jun 20, 2024

• chore: remove unused distance type from HNSW

  #2485 merged Jun 19, 2024

• chore: replace the legacy numeric constants with new ones

  #2486 merged Jun 19, 2024

• fix: low recall if many partitions with only 1 row

  #2487 merged Jun 19, 2024

• feat: added some additional logging to the indexing process

  #2488 merged Jun 19, 2024

• chore: implement with_new_children correctly for index nodes

  #2483 merged Jun 18, 2024

• feat(rust): add support for loongarch

  #2406 merged Jun 17, 2024

• chore: provide a trait to decouple take rows from Dataset

  #2478 merged Jun 17, 2024

• docs: clip training example

  #2477 merged Jun 17, 2024

• feat: make IVF_HNSW_SQ work with dot distance

  #2472 merged Jun 15, 2024

• feat: add arguments to allow resuming gpu indexing

  #2473 merged Jun 15, 2024

• fix: correct with_new_children for knn flat

  #2471 merged Jun 14, 2024

• chore(java): remove the unneeded spark/v3 folder

  #2463 merged Jun 14, 2024

• refactor: impl the sub index trait for HNSW and clean code

  #2462 merged Jun 13, 2024

• fix: incorrect logic when decoding manifest from last block

  #2464 merged Jun 13, 2024

• fix: respect storage options on create_index

  #2460 merged Jun 12, 2024

• chore(java): enable lance-jni rust release

  #2369 merged Jun 12, 2024

• docs: add spark-lance-connector readme

  #2456 merged Jun 11, 2024

• feat: store IVF in global buffer

  #2449 merged Jun 11, 2024

• feat: pickle the manifest when pickling a dataset

  #2459 merged Jun 10, 2024

• feat(java): add lance spark connector for read

  #2429 merged Jun 8, 2024

25 Pull requests opened by 9 people

• refactor: move sq and hnsw builder to lance-index

  #2455 opened Jun 8, 2024

• feat: add outer retry loop to more object_store operations

  #2458 opened Jun 9, 2024

• feat: support binary & float vectors for IVF_FLAT index

  #2461 opened Jun 10, 2024

• feat: more flexible and sophisticated handling of non-null constraints

  #2467 opened Jun 13, 2024

• feat: add FSST string compression(draft, wip)

  #2470 opened Jun 14, 2024

• feat: add version tags

  #2482 opened Jun 17, 2024

• fix: build one partition at a time to avoid OOM

  #2484 opened Jun 18, 2024

• perf: use v2 to write the sorted IVF partition files

  #2492 opened Jun 20, 2024

• feat(java): support Lance Spark Batch Write

  #2500 opened Jun 21, 2024

• feat: support to query with vectors of another types

  #2502 opened Jun 21, 2024

• feat: object store registry for custom object store providers

  #2513 opened Jun 22, 2024

• ci(java): build JNI release package

  #2516 opened Jun 24, 2024

• chore: bump pyarrow version

  #2517 opened Jun 24, 2024

• [WIP] feat(java): enable java build

  #2524 opened Jun 25, 2024

• feat: move-stable row ids in compaction

  #2544 opened Jun 28, 2024

• feat: stable row id deletions and scalar indices

  #2550 opened Jul 1, 2024

• refactor: build & search IVF_HNSW_PQ with new indexer & IVF impl

  #2552 opened Jul 2, 2024

• feat: add standalone partition assignment operation

  #2556 opened Jul 3, 2024

• feat: add tqdm progress bar when training kmeans with CPU

  #2557 opened Jul 3, 2024

• feat: add a standalone method for calculating the PQ model

  #2562 opened Jul 3, 2024

• fix: spawn multiple encoding task inside a batch number of rows in one column when needed

  #2563 opened Jul 3, 2024

• feat: standalone vector transform stage

  #2566 opened Jul 5, 2024

• chore: bump aws package versions

  #2567 opened Jul 5, 2024

• feat: support to set ef for search from python

  #2569 opened Jul 6, 2024

• perf: improve inverted index performance

  #2574 opened Jul 8, 2024

13 Issues closed by 4 people

• ci: Python release fails without protoc

  #2551 closed Jul 3, 2024

• linker errors on arm64(macos m1)

  #1924 closed Jul 3, 2024

• [Rust] Do not use dynamic function to calculate distance

  #997 closed Jun 29, 2024

• Allow append and delete to not conflict

  #2143 closed Jun 27, 2024

• Use TopK node for KNN

  #1324 closed Jun 27, 2024

• lance `storage_options` doesn't work well for MinIO s3 backend

  #2166 closed Jun 26, 2024

• Optimistic concurrency control

  #951 closed Jun 26, 2024

• ci: Mac runner flaky test

  #2002 closed Jun 25, 2024

• Cannot read "just the row id" from a dataset using v2 files

  #2480 closed Jun 24, 2024

• Document Lance compatibility with python multiprocessing

  #2405 closed Jun 21, 2024

• Commit conflicts unavoidable running compaction when there are many files

  #2479 closed Jun 17, 2024

• "error: Unrecognized option: 'diagnostic-width'" when first run test in visual code on mac

  #2465 closed Jun 17, 2024

• chore: setup maven repo for java jni core SDK

  #2264 closed Jun 15, 2024

26 Issues opened by 10 people

• feat(encoding): row selection push down to decode

  #2572 opened Jul 7, 2024

• page compaction after encoding

  #2571 opened Jul 6, 2024

• bug: spawn multiple encoding tasks inside one batch number of rows of one column

  #2561 opened Jul 3, 2024

• LanceFragment.create does not coerce input to the target schema

  #2549 opened Jul 1, 2024

• Add support for bitmap indices

  #2548 opened Jul 1, 2024

• feat: support creating scalar index on dictionary type

  #2546 opened Jun 29, 2024

• Epic: Support integer vectors (int8)

  #2543 opened Jun 28, 2024

• discussion(encoding): The FastLanes Compression Layout

  #2542 opened Jun 27, 2024

• No. Row group?

  #2540 opened Jun 27, 2024

• Commit loop improvements

  #2538 opened Jun 26, 2024

• Unify `storage_options` and URI params

  #2536 opened Jun 26, 2024

• Commit conflict error in Python just gives module not found

  #2530 opened Jun 25, 2024

• Need python option in `crate_index` to save the IVF centroids

  #2520 opened Jun 24, 2024

• Shuffling stage is too slow

  #2519 opened Jun 24, 2024

• PQ assignment stage `HashMap` lookups are too slow at 1B rows

  #2518 opened Jun 24, 2024

• filtering on timestamp column not working when tried with duckdb

  #2514 opened Jun 23, 2024

• aws-* dependency maintenance

  #2511 opened Jun 22, 2024

• Segmentation fault training index on high-cpu system

  #2503 opened Jun 21, 2024

• Add regression tests for recall

  #2497 opened Jun 20, 2024

• lance run benchmark panic

  #2491 opened Jun 19, 2024

• remote server fail when run lance benchmark

  #2490 opened Jun 19, 2024

• feat(encoding): float point number compression

  #2476 opened Jun 17, 2024

• feat(encoding): autonomous encoding scheme selection

  #2475 opened Jun 17, 2024

• feat(encoding): encoding cascade

  #2474 opened Jun 17, 2024

• Native support for PIL Images in Lance

  #2468 opened Jun 14, 2024

• Failed to get AWS credentials: an error occurred while loading credentials

  #2466 opened Jun 13, 2024

11 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [bug] Writing a column of type list with nulls results in the nulls being replaced with []

  #1946 commented on Jun 17, 2024 • 4 new comments

• Android build fails due to `aarch64` in Rust

  #2411 commented on Jun 22, 2024 • 3 new comments

• head() is slower than take()

  #1927 commented on Jun 11, 2024 • 2 new comments

• Could not make proto path relative

  #1855 commented on Jun 25, 2024 • 2 new comments

• Enhance Lance Image Docs (DNC)

  #2212 commented on Jun 19, 2024 • 1 new comment

• import lancedb fails with illegal instruction on older intel CPU

  #2195 commented on Jul 5, 2024 • 1 new comment

• feat: add bitpack encoding for LanceV2

  #2333 commented on Jul 5, 2024 • 1 new comment

• Epic: Move-Stable Row Ids

  #2307 commented on Jul 1, 2024 • 0 new comments

• Add support for dictionary encoded fields to the v2 reader/writer

  #2347 commented on Jul 2, 2024 • 0 new comments

• add a metadata to manifest to indicate prior dynamoDB usage and generate warning when a version is created to off-board dynamodb or when the dynamodb table info changes.

  #1206 commented on Jul 3, 2024 • 0 new comments

• chore(deps): bump org.json:json from 20210307 to 20231013 in /java

  #2374 commented on Jul 2, 2024 • 0 new comments