fix attention mask collation

[winglian]

fixes #1597 @timpal0l

not sure even how the previous code worked 🤷

here's part of the yaml that worked for me

pretraining_dataset:
  - path: HuggingFaceTB/cosmopedia_6M
    split: train
    type: completion
max_steps: 10_000

pretrain_multipack_attn: true
pretrain_multipack_buffer_size: 10_000

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

[timpal0l]

@winglian I will try!

this should make the training faster if i understand correctly? since i understand that the attention mask was not set correctly with this set to false? (i guess this is only relevant when sample_packing: false also?)

This new error popped up now instead;

RuntimeErrorRuntimeError: : CUDA error: an illegal memory access was encountered

[ali-mosavian]

Not sure how this is done in Axolotl, but the idea is to use the attention mask to prohibit attention across samples. If correctly implemented.

[winglian]

Not sure how this is done in Axolotl, but the idea is to use the attention mask to prohibit attention across samples. If correctly implemented.

yes, this is the general philosophy of axolotl is to ensure the highest quality for SFT by preventing cross-attention. However, we enable this to be toggled on or off for (continued) pretraining b/c in those cases, you want to concatenate the samples to reach the full context length. Depending on the implementation, (FA2 vs SDPA vs eager) the model may never see position_ids up to the maximum context length. I don't have evidence one way or another that this is bad, but I would expect you want to maintain the existing context length during continued pretraining.