Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flash Attention v2 MHA #17227

Merged
merged 70 commits into from
Aug 31, 2023
Merged

Flash Attention v2 MHA #17227

merged 70 commits into from
Aug 31, 2023

Commits on Aug 18, 2023

  1. add flash attention v2

    @tianleiwu @aciddelgado
    tianleiwu authored and aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    f39854d View commit details
    Browse the repository at this point in the history

  2. remove bfloat16 kernels

    @tianleiwu @aciddelgado
    tianleiwu authored and aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    aa1efa8 View commit details
    Browse the repository at this point in the history

  3. namespace

    @tianleiwu @aciddelgado
    tianleiwu authored and aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    1fe60ae View commit details
    Browse the repository at this point in the history

  4. remove commented code

    @tianleiwu @aciddelgado
    tianleiwu authored and aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    6cb6eba View commit details
    Browse the repository at this point in the history

  5. remove backward code

    @tianleiwu @aciddelgado
    tianleiwu authored and aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    893b77e View commit details
    Browse the repository at this point in the history

  6. path

    @tianleiwu @aciddelgado
    tianleiwu authored and aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    1195d3f View commit details
    Browse the repository at this point in the history

  7. flash v2 api and api call

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    4e59d95 View commit details
    Browse the repository at this point in the history

  8. api build fixes

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    b0f7f47 View commit details
    Browse the repository at this point in the history

  9. builds succesfully

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    0e79ec5 View commit details
    Browse the repository at this point in the history

  10. some fixes

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    2f61ac2 View commit details
    Browse the repository at this point in the history

  11. use varlen to run Attention_Mask1D_FP16_B2 test

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    372141d View commit details
    Browse the repository at this point in the history

  12. Packed MHA cleanup

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    95a7e35 View commit details
    Browse the repository at this point in the history

  13. flash attention runs

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    52553a7 View commit details
    Browse the repository at this point in the history

  14. update

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    018ecb6 View commit details
    Browse the repository at this point in the history

  15. pre clean

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    632451e View commit details
    Browse the repository at this point in the history

  16. clean PMHA

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    98a4e5d View commit details
    Browse the repository at this point in the history

  17. packed mha flash

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    12f8db8 View commit details
    Browse the repository at this point in the history

  18. remove extraneous changes

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    91c1ab8 View commit details
    Browse the repository at this point in the history

  19. reviewed changes

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    2fc7d86 View commit details
    Browse the repository at this point in the history

  20. reviewed changes

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    7cf0359 View commit details
    Browse the repository at this point in the history

  21. reviewed changes

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    f6554a1 View commit details
    Browse the repository at this point in the history

  22. reviewed changes

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    7a12487 View commit details
    Browse the repository at this point in the history

  23. reviewed changes

    @aciddelgado
    aciddelgado committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    33a43aa View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2023

  1. namespace and USE_FLASH_ATTENTION flag

    @aciddelgado
    aciddelgado committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    24524cc View commit details
    Browse the repository at this point in the history

  2. more compile flags flash

    @aciddelgado
    aciddelgado committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    d9002f7 View commit details
    Browse the repository at this point in the history

  3. lint

    @aciddelgado
    aciddelgado committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    3be4af2 View commit details
    Browse the repository at this point in the history

  4. gcc warnings in template

    @aciddelgado
    aciddelgado committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    3798446 View commit details
    Browse the repository at this point in the history

  5. address tianlei comments

    @aciddelgado
    aciddelgado committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    666d3bf View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2023

  1. clean up

    tlwu@microsoft.com committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    545851a View commit details
    Browse the repository at this point in the history

  2. cpplint

    tlwu@microsoft.com committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    5798e0b View commit details
    Browse the repository at this point in the history

  3. refactoring

    @tianleiwu
    tianleiwu committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    cad2901 View commit details
    Browse the repository at this point in the history

  4. workspace as buffer. copyright and cgmanifest.

    @aciddelgado
    aciddelgado committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    fdb189b View commit details
    Browse the repository at this point in the history

  5. undo cgmanifest

    @aciddelgado
    aciddelgado committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    14d7db2 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. enable flash attention in MultiHeadAttention op

    @tianleiwu
    tianleiwu committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    aba9ceb View commit details
    Browse the repository at this point in the history

  2. fix attention test error in A100

    @tianleiwu
    tianleiwu committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    a06da2e View commit details
    Browse the repository at this point in the history

  3. namespace from flash to onnxruntime::flash

    @tianleiwu
    tianleiwu committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    ab015e6 View commit details
    Browse the repository at this point in the history

  4. undo cgmanifest

    @tianleiwu
    tianleiwu committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    866414c View commit details
    Browse the repository at this point in the history

  5. add unit test

    @tianleiwu
    tianleiwu committed Aug 23, 2023
    1 Configuration menu
    Copy the full SHA
    6d682ab View commit details
    Browse the repository at this point in the history

  6. enable flash attention in Attention op

    @tianleiwu
    tianleiwu committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    300019e View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2023

  1. Merge branch 'main' into flash_v2_packed_mha

    @tianleiwu
    tianleiwu committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    7e50d9a View commit details
    Browse the repository at this point in the history

  2. set proper nvcc threads to avoid OOM

    @tianleiwu
    tianleiwu committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    7481ec3 View commit details
    Browse the repository at this point in the history

  3. --nvcc_threads=1 in build_cuda_c_api_package.sh

    @tianleiwu
    tianleiwu committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    2823476 View commit details
    Browse the repository at this point in the history

  4. test script segfaults

    @aciddelgado
    aciddelgado committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    c880147 View commit details
    Browse the repository at this point in the history

  5. pass cuda device prop to flash attention

    @tianleiwu
    tianleiwu committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    d2a040b View commit details
    Browse the repository at this point in the history

  6. add requirements for test_flash_attn.py

    @tianleiwu
    tianleiwu committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    8b92a9b View commit details
    Browse the repository at this point in the history

  7. remove nvcc_threads logic

    @tianleiwu
    tianleiwu committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    a0393e2 View commit details
    Browse the repository at this point in the history

  8. flash attn test, pmha works, mha crashes

    @aciddelgado
    aciddelgado committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    0830925 View commit details
    Browse the repository at this point in the history

  9. check head size for efficient attention

    @tianleiwu
    tianleiwu committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    1c96739 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2023

  1. lint except lambda assignment

    @aciddelgado
    aciddelgado committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    6d8e43d View commit details
    Browse the repository at this point in the history

  2. lint fix

    @aciddelgado
    aciddelgado committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    4ceca3b View commit details
    Browse the repository at this point in the history

  3. line length < 120

    @tianleiwu
    tianleiwu committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    3bfa3b5 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. flash v2 update

    @aciddelgado
    aciddelgado committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    bdb17d5 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2023

  1. formatting

    @aciddelgado
    aciddelgado committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    bfee28e View commit details
    Browse the repository at this point in the history

  2. flash benchmark script

    @aciddelgado
    aciddelgado committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    a54a7b9 View commit details
    Browse the repository at this point in the history

  3. merge with main

    @aciddelgado
    aciddelgado committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    b064a01 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. io binding

    @aciddelgado
    aciddelgado committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    f6927f7 View commit details
    Browse the repository at this point in the history

  2. update benchmark

    @tianleiwu
    tianleiwu committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    345f4e6 View commit details
    Browse the repository at this point in the history

  3. Add bert-base

    @tianleiwu
    tianleiwu committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    da8bc50 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into flash_v2_packed_mha 

    Nuget Fix
    @aciddelgado
    aciddelgado committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    40a1f61 View commit details
    Browse the repository at this point in the history

  5. merge main into branch for nuget fix

    @aciddelgado
    aciddelgado committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    f760123 View commit details
    Browse the repository at this point in the history

  6. update benchark to support more input formats

    @tianleiwu
    tianleiwu committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    92652c3 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'flash_v2_packed_mha' of https://github.com/microsoft/on… 

    …nxruntime into flash_v2_packed_mha
    @tianleiwu
    tianleiwu committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    ee2296f View commit details
    Browse the repository at this point in the history

  8. seq len threshold to trigger flash for packed qkv

    @tianleiwu
    tianleiwu committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    e998af7 View commit details
    Browse the repository at this point in the history

  9. add back 2 lines

    @tianleiwu
    tianleiwu committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    599d019 View commit details
    Browse the repository at this point in the history

  10. flash attention flag in packed attention op test and a few more bench… 

    …marks for roli
    @aciddelgado
    aciddelgado committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    492c59f View commit details
    Browse the repository at this point in the history

  11. flash attention flag in packed attention op test and a few more bench… 

    …marks for roli
    @aciddelgado
    aciddelgado committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    6400a02 View commit details
    Browse the repository at this point in the history

  12. Merge remote-tracking branch 'refs/remotes/origin/flash_v2_packed_mha… 

    …' into flash_v2_packed_mha

merge previous commit
    @aciddelgado
    aciddelgado committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    1929de5 View commit details
    Browse the repository at this point in the history

  13. specify TNLGv4 model for Turing Team in Benchmark

    @aciddelgado
    aciddelgado committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    c880f08 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2023

  1. remove env variable change from packed attention test

    @aciddelgado
    aciddelgado committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    30c2f79 View commit details
    Browse the repository at this point in the history

  2. python lint

    @aciddelgado
    aciddelgado committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    01443ef View commit details
    Browse the repository at this point in the history