Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor hashtable use linear probing #15

Merged
merged 89 commits into from
Jun 8, 2020
Merged

Refactor hashtable use linear probing #15

merged 89 commits into from
Jun 8, 2020

Commits on May 31, 2020

  1. Initial implementation of the hash search algorithm in the array with… 

    … 14 entries with AVX512, AVX2, SSE4 and a simple linear search support
    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    bfc66a6 View commit details
    Browse the repository at this point in the history

  2. Initial drop of the cachelines-based hashtable implementation

    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    55d89e2 View commit details
    Browse the repository at this point in the history

  3. Style fix

    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    03e881f View commit details
    Browse the repository at this point in the history

  4. Initial benchmarking support for the hashtable_support_hash_search_{a… 

    …vx512,avx2,sse4,loop} functions, not fully tested
    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    20f7a9d View commit details
    Browse the repository at this point in the history

  5. Prefix cmake custom variables with the projectname

    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    248be49 View commit details
    Browse the repository at this point in the history

  6. Drop more code related to the cacheline-based hashtable

    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    6722f40 View commit details
    Browse the repository at this point in the history

  7. Rename HASHTABLE_INLINE_KEY_MAX_SIZE to HASHTABLE_KEY_INLINE_MAX_LENG… 

    …TH to match the naming convention
    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    a80afd5 View commit details
    Browse the repository at this point in the history

  8. Add support to store a rpefix of the key into the KV structure to be … 

    …used during the search phase to speed up the lookup
    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    d224862 View commit details
    Browse the repository at this point in the history

  9. Import portable snippet cpu library ( https://github.com/nemequ/porta… 

    …ble-snippets )
    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    520370b View commit details
    Browse the repository at this point in the history

  10. Drop gcc builtins to make the code more portable, drop the preferred … 

    …instruction set variable to hint which instruction set should be selected
    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    3f4c92c View commit details
    Browse the repository at this point in the history

  11. Drop the AVX512 and the SSE linear search algorithm implementation

    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    0726c6b View commit details
    Browse the repository at this point in the history

  12. Replace avx2 search algorithm implementation with the branchless prov… 

    …ided by @chtz ( https://stackoverflow.com/a/62123631/169278 )
    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    30e13c2 View commit details
    Browse the repository at this point in the history

  13. Drop AVX512 and SEE benchs

    @danielealbano
    danielealbano committed May 31, 2020
    Configuration menu
    Copy the full SHA
    b0a76cb View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2020

  1. Drop unused header

    @danielealbano
    danielealbano committed Jun 1, 2020
    Configuration menu
    Copy the full SHA
    494e0f1 View commit details
    Browse the repository at this point in the history

  2. Refactor the code to better test the two search implementation, in th… 

    …e real world they don't get to perform exactly X searches on exactly the same data exactly sequentially, this approach regenerates the data each time and is more realistic
    @danielealbano
    danielealbano committed Jun 1, 2020
    Configuration menu
    Copy the full SHA
    1449d42 View commit details
    Browse the repository at this point in the history

  3. Split out the avx2/loop hash search algorithm implementation into the… 

    …ir own files, need to properly handle AVX2 compilation flags targetted per src file
    @danielealbano
    danielealbano committed Jun 1, 2020
    Configuration menu
    Copy the full SHA
    e300fcb View commit details
    Browse the repository at this point in the history

  4. Add the ad-hoc compilation flags for the avx2 hash serch algorithm im… 

    …plementation
    @danielealbano
    danielealbano committed Jun 1, 2020
    Configuration menu
    Copy the full SHA
    33bb3e9 View commit details
    Browse the repository at this point in the history

  5. Improve benchmark structure

    @danielealbano
    danielealbano committed Jun 1, 2020
    Configuration menu
    Copy the full SHA
    08341c0 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2020

  1. Rely on the build system to decide if this the avx2 variant has to be… 

    … compiled
    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    b196a46 View commit details
    Browse the repository at this point in the history

  2. Implement the AVX hash search algorithm variant

    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    dfe2963 View commit details
    Browse the repository at this point in the history

  3. Add the AVX search algorithm to the auto-selection function

    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    32103ac View commit details
    Browse the repository at this point in the history

  4. Rework the build system

    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    cd87975 View commit details
    Browse the repository at this point in the history

  5. Add the AVX version of the search algorithm to the benches and simpli… 

    …fy the define containing the main bench
    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    86c8716 View commit details
    Browse the repository at this point in the history

  6. Rename TARGET_ARCH in CACHEGRAND_ARCH_TARGET, rename version.* in cma… 

    …ke_config.*, rename version.cmake module in cmake_config.cmake, improve message logging, add a custom target to automatically update the cmake_config.c file on every build to correctly update the build date/time
    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    22c33d7 View commit details
    Browse the repository at this point in the history

  7. Expose the CMAKE_BUILD_TYPE variable

    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    71a0928 View commit details
    Browse the repository at this point in the history

  8. Reorganise the cmake_config.h file and fix cpp support (when included… 

    … via benches or testing)
    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    aed75ea View commit details
    Browse the repository at this point in the history

  9. Split the buildstep invoked as custom dependency target in its own cm… 

    …ake script
    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    bf482bf View commit details
    Browse the repository at this point in the history

  10. Update the src cmakefile script to use the new cmake_config_c exposed… 

    … variables to include the re-generated file at build time
    @danielealbano
    danielealbano committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    6343ad6 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2020

  1. Refactor the hash search functions to be able to ignore some matches … 

    …using a bitmask
    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    21f4898 View commit details
    Browse the repository at this point in the history

  2. Select the right hash search implementation when the hashtable is ini… 

    …tialized, add a static variable to ensure the code is executed only once
    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    fb5ad77 View commit details
    Browse the repository at this point in the history

  3. Drop some more cachelines related code

    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    8a1d00f View commit details
    Browse the repository at this point in the history

  4. Explicitly drop the volatile attribute

    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    7ef46fb View commit details
    Browse the repository at this point in the history

  5. Refactor the data structures to support chains of rings containing 14… 

    … slots (half hashes / key-values) in the buckets, add a mapping for the int128_t and uint128_t types and their related atomic version
    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    4b3ecf3 View commit details
    Browse the repository at this point in the history

  6. Refactor hashtable data initialization code to support the new data s… 

    …tructures
    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    0a6e118 View commit details
    Browse the repository at this point in the history

  7. Refactor the get operation and the support operations (slot hash / ke… 

    …y search) for the get operation
    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    472dcdb View commit details
    Browse the repository at this point in the history

  8. Refactor the hashtable get tests to check the new structures and impl… 

    …ementation
    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    c7da799 View commit details
    Browse the repository at this point in the history

  9. Drop more cachelines related code (cachelines_to_probe* / buckets_cou… 

    …nt_real)
    @danielealbano
    danielealbano committed Jun 6, 2020
    Configuration menu
    Copy the full SHA
    324f6c4 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2020

  1. Update the get op benchmark of the hashtable to support the new struc… 

    …ture and catch failures
    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    699fc98 View commit details
    Browse the repository at this point in the history

  2. Style fix

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    a09ae9a View commit details
    Browse the repository at this point in the history

  3. Drop clang support (it's necessary to stick with gcc to emit cmpxchg1… 

    …6b using the __sync builtins)
    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    ad2c718 View commit details
    Browse the repository at this point in the history

  4. Re-organise the bucket structure to have the write_lock using only 1 … 

    …byte
    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    efea617 View commit details
    Browse the repository at this point in the history

  5. Add a prefix_key struct in the hastable_bucket_key_value struct and u… 

    …pdate the code to use that instead of key_prefix in external_key
    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    4a3b82d View commit details
    Browse the repository at this point in the history

  6. Fix naming convention (atomic is appended after)

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    f99cb38 View commit details
    Browse the repository at this point in the history

  7. Rename skip_indexes in skip_indexes_mask

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    ac770ff View commit details
    Browse the repository at this point in the history

  8. When compiling code for debugging disable inlineing and any kind of o… 

    …ptimization - wrong file included by mistake (the benches for the hash search have been updated to support the skip_index_mask)
    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    3438f46 View commit details
    Browse the repository at this point in the history

  9. Fix benches dependencies

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    cafbfd7 View commit details
    Browse the repository at this point in the history

  10. Implement bucket locking, unlocking and fetching with write lock (use… 

    … __sync builtins for atomic to force gcc to emit the cmpxhg16 instruction in-line)
    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    100a585 View commit details
    Browse the repository at this point in the history

  11. Refactor the set implementation to rely on the new data structures an… 

    …d algorithm
    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    6ae7c03 View commit details
    Browse the repository at this point in the history

  12. Fix atomic invocation

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    05fa6ca View commit details
    Browse the repository at this point in the history

  13. Fix lock check

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    b830c4b View commit details
    Browse the repository at this point in the history

  14. Add thread yielding if bucket locked or unable to acquire bucket

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    0f2372e View commit details
    Browse the repository at this point in the history

  15. Expose if the bucket has been initialized upon the invocation of hash… 

    …table_support_op_bucket_fetch_and_write_lock
    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    a93fad5 View commit details
    Browse the repository at this point in the history

  16. Fix the mask implemented in the HASHTABLE_BUCKET_WRITE_LOCK_SET macro

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    a93b65b View commit details
    Browse the repository at this point in the history

  17. Add hashtable_support_op testing

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    352efbf View commit details
    Browse the repository at this point in the history

  18. Add testing for hashtable_support_hash_half

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    eced0ee View commit details
    Browse the repository at this point in the history

  19. Initialize found to false and pass back to the caller the found bucket

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    6adf743 View commit details
    Browse the repository at this point in the history

  20. Style fix

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    23049a8 View commit details
    Browse the repository at this point in the history

  21. Update the hashtable_op_set tests

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    1cf4cbc View commit details
    Browse the repository at this point in the history

  22. Fix hashtable_op_set "set 2 slots" test

    @danielealbano
    danielealbano committed Jun 7, 2020
    Configuration menu
    Copy the full SHA
    293ad18 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2020

  1. Update how the t1ha library is built, includes t1ha0, t1ha1 and t1ha… 

    …2, including the support for aesni/avx/avx2 accelerated algorithm
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    b60bfc6 View commit details
    Browse the repository at this point in the history

  2. Fix t1ha linking

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    b59fab5 View commit details
    Browse the repository at this point in the history

  3. Fix how the avx2/avx search algorithm gets build, if it's on x86_64 t… 

    …he compiler will always able to emit avx2/avx instructions
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    4cc7695 View commit details
    Browse the repository at this point in the history

  4. Implement the search algorithms for the 8 slots version and use ifunc… 

    … instead of function pointers to pick the best implementation option at runtime
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    d2e10e1 View commit details
    Browse the repository at this point in the history

  5. Switch to use t1ha0, after a number of tests the load factor almost d… 

    …oesn't change
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    5e6af66 View commit details
    Browse the repository at this point in the history

  6. Drop pre-check on the write lock

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    6276d49 View commit details
    Browse the repository at this point in the history

  7. Lock always before checkinf the chain_first_ring is null to avoid edg… 

    …e cases that would case slowdowns
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    2e4e6e8 View commit details
    Browse the repository at this point in the history

  8. Fix the search or create key (hashtable_support_op_search_key_or_crea… 

    …te_new) function
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    af18d87 View commit details
    Browse the repository at this point in the history

  9. Improve testing

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    1623cd1 View commit details
    Browse the repository at this point in the history

  10. Fix hashtable structures

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    78ee645 View commit details
    Browse the repository at this point in the history

  11. Update the hashtable support hash search benchmark to use the new nam… 

    …e convention for the methods and move the shared code to an external support c file
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    2a2ab70 View commit details
    Browse the repository at this point in the history

  12. Improve how the load factor is calculated

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    55bf660 View commit details
    Browse the repository at this point in the history

  13. Drop the code related to the cachelines / loadfactor calculation

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    11d9a70 View commit details
    Browse the repository at this point in the history

  14. Use the set_thread_affinity function from bench-support.c

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    cd9e3fc View commit details
    Browse the repository at this point in the history

  15. Fix headers

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    66d3c98 View commit details
    Browse the repository at this point in the history

  16. Drops keys pre-generation, better to generate random keys on every it… 

    …eration
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    2c4372e View commit details
    Browse the repository at this point in the history

  17. Add the ability to collect hashtable stats (load factor, buckets usag… 

    …e, etc)
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    ab5104b View commit details
    Browse the repository at this point in the history

  18. Check for set errors

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    c879f37 View commit details
    Browse the repository at this point in the history

  19. Fix index variable type and fix how the hashtable is prefilled with t… 

    …he keys for the update bench
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    561154f View commit details
    Browse the repository at this point in the history

  20. Run the benches at least 10 times

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    5151349 View commit details
    Browse the repository at this point in the history

  21. Allow a maximum of 4 threads per core on the benches

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    59d0640 View commit details
    Browse the repository at this point in the history

  22. Rename the support function in the hashtable set bench and share the … 

    …collect hashtable stats and update state code
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    3cb9501 View commit details
    Browse the repository at this point in the history

  23. Enforce -O3 compilation flag when building for non-debug and fix style

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    5248f3a View commit details
    Browse the repository at this point in the history

  24. Refactor the hashtable to use linear probing, add support to embed ke… 

    …y/values onto the bucket and add support disabling the locks (switch to atomic operations)
    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    33c17c5 View commit details
    Browse the repository at this point in the history

  25. Fix mmap result check

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    dd05e1e View commit details
    Browse the repository at this point in the history

  26. Drop useless fixtures

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    1fdbf4f View commit details
    Browse the repository at this point in the history

  27. Update the tests to test the linear probing

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    0b2b1e4 View commit details
    Browse the repository at this point in the history

  28. Update the defaults (no locks, key/values embedded)

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    be352e0 View commit details
    Browse the repository at this point in the history

  29. Update the tests to support the new data structure

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    0f1ef1a View commit details
    Browse the repository at this point in the history

  30. Re-enable locks, with atomic ops 6 times slower

    @danielealbano
    danielealbano committed Jun 8, 2020
    Configuration menu
    Copy the full SHA
    a891287 View commit details
    Browse the repository at this point in the history