torch.backends.cuda.enable_cudnn_sdp(False)
truncate_dim option to the Sentence Transformer constructor. This also allows truncation when using HuggingFaceEmbeddings from LlamaIndex or LangChain.truncate_dim in evaluators to get the performance after truncation. (Hint: it's surprisingly good, even for models not trained with MatryoshkaLoss, and it can speed up e.g. clustering, retrieval, etc.)trust_remote_code to load models with custom modelling code.
I'm testing on it, with 32K(claimed at paper) and 1M seq len.
I'm training those models with minipile dataset and for now, it seems minimal continual training let model to adapt 'memory' could be sufficient.(less than <1B tokens)
Train is not finished yet, but after the loss converges then I could test haystack test or inference tests. it won't be take long :)
bash
git clone https://github.com/Beomi/InfiniTransformer
bash
pip install -r requirements.txt
pip install -e git+https://github.com/huggingface/transformers.git@b109257f4f#egg=transformers
./train.gemma.infini.noclm.1Mseq.sh.