11 Commits

Author	SHA1	Message	Date
Raushan Turganbay	ebbe8d8014	Cache docs: update (#32929) ... * some changes * more updates * fix cache copy * nits * nits * add tests	2024-09-04 15:05:31 +05:00
Gerben van V	5129671290	Add a static cache that offloads to the CPU or other device (#32161) ... * Add a static cache that offloads to the CPU or other device * Fix PR comments, add unit-tests	2024-08-29 11:51:09 +02:00
Joao Gante	cf32ee1753	Cache: use batch_size instead of max_batch_size (#32657) ... * more precise name * better docstrings * Update src/transformers/cache_utils.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>	2024-08-16 11:48:45 +01:00
Guang Yang	0164560353	Fixed test test_static_cache_exportability with torch 2.4.0 (#32516) ... Workaround the export issue in torch 2.4 Co-authored-by: Guang Yang <guangyang@fb.com>	2024-08-08 18:13:40 +01:00
OsamaS99	51ab25e293	Fixed Hybrid Cache Shape Initialization. (#32163) ... * fixed hybrid cache init, added test * Fix Test Typo --------- Co-authored-by: Aaron Haag <aaron.haag@siemens.com>	2024-08-01 13:57:42 +01:00
Nikos Karampatziakis	ca59d6f77c	Offloaded KV Cache (#31325) ... * Initial implementation of OffloadedCache * enable usage via cache_implementation * Address feedback, add tests, remove legacy methods. * Remove flash-attn, discover synchronization bugs, fix bugs * Prevent usage in CPU only mode * Add a section about offloaded KV cache to the docs * Fix typos in docs * Clarifications and better explanation of streams	2024-08-01 14:42:07 +02:00
Guang Yang	811a9caa21	Make static cache compatible with torch.export (#32168)	2024-07-29 18:19:15 +01:00
Joao Gante	7ffe25f2b9	Generate: end-to-end compilation (#30788) ... * mvp * added test (a few models need fixes) * fix a few test cases * test nits * harder test 😈 * revert changes in stablelm * test with improved condition * add todo * tmp commit * merged with main * nits * add todo * final corrections * add docs for generation compilation * docs nits * add tip * PR suggestions * add more details to the compilation docs * fix cache positions * cache is now init in generate; update docs * tag test as flaky * docs * post rebase make fixup and other nits * remove unintended changes * whisper (encoder-decoder) not supported * move token default updates to ; add tests for token defaults * push changes * manual rebase * chameleon doesn't support this * fix test_static_cache_mha_mqa_gqa (broken in another PR) * docs: dynamic is better with end-to-end compilation	2024-07-29 10:52:13 +01:00
Fanli Lin	27c7f971c0	[tests] fix static cache implementation is not compatible with attn_implementation==flash_attention_2 (#32039) ... * add flash attention check * fix * fix	2024-07-26 11:41:27 +02:00
Joao Gante	739a63166d	Generate: remove deprecated code due to Cache and cache_position being default (#31898) ... * tmp commit * shorter * nit * explicit kwargs * propagate changes * mass propagation with a few manual touches (let's see how CI behaves) * fix cacheless case * Update src/transformers/generation/utils.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * make fixup --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>	2024-07-14 15:16:58 +01:00
Yih-Dar	93cd94b79d	Move some test files (tets/test_xxx_utils.py) to tests/utils (#31730) ... * move * move * move * move * Update tests/utils/test_image_processing_utils.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>	2024-07-02 13:46:03 +02:00