transformers/tests/utils
Matt edc1dffd00
Chat Template support for function calling and RAG (#30621)
* First draft, still missing automatic function conversion

* First draft of the automatic schema generator

* Lots of small fixes

* the walrus has betrayed me

* please stop committing your debug breakpoints

* Lots of cleanup and edge cases, looking better now

* Comments and bugfixes for the type hint parser

* More cleanup

* Add tests, update schema generator

* Update tests, proper handling of return values

* Small docstring change

* More doc updates

* More doc updates

* Add json_schema decorator

* Clean up the TODOs and finish the docs

* self.maxDiff = None to see the whole diff for the nested list test

* add import for add_json_schema

* Quick test fix

* Fix something that was bugging me in the chat template docstring

* Less "anyOf" when unnecessary

* Support return types for the templates that need them

* Proper return type tests

* Switch to Google format docstrings

* Update chat templating docs to match new format

* Stop putting the return type in with the other parameters

* Add Tuple support

* No more decorator - we just do it implicitly!

* Add enum support to get_json_schema

* Update docstring

* Add copyright header

* Update src/transformers/tokenization_utils_base.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update docs/source/en/chat_templating.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update src/transformers/utils/chat_template_utils.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update src/transformers/utils/chat_template_utils.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Add copyright header

* make fixup

* Fix indentation

* Reformat chat_template_utils

* Correct return value

* Make regexes module-level

* Support more complex, multi-line arg docstrings

* Update error message for ...

* Update ruff

* Add document type validation

* Refactor docs

* Refactor docs

* Refactor docs

* Clean up Tuple error

* Add an extra test for very complex defs and docstrings and clean everything up for it

* Document enum block

* Quick test fixes

* Stop supporting type hints in docstring to fix bugs and simplify the regex

* Update docs for the regex change

* Clean up enum regex

* Wrap functions in {"type": "function", "function": ...}

* Update src/transformers/utils/chat_template_utils.py

Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com>

* Temporary tool calling commit

* Add type hints to chat template utils, partially update docs (incomplete!)

* Code cleanup based on @molbap's suggestion

* Add comments to explain regexes

* Fix up type parsing for unions and lists

* Add custom exception types and adjust tests to look for them

* Update docs with a demo!

* Docs cleanup

* Pass content as string

* Update tool call formatting

* Update docs with new function format

* Update docs

* Update docs with a second tool to show the model choosing correctly

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com>
2024-06-11 15:46:38 +01:00
..
__init__.py
test_activations_tf.py
test_activations.py Add the GeLU activation from pytorch with the tanh approximation (#21345) 2023-02-02 09:33:04 -05:00
test_add_new_model_like.py Remove static pretrained maps from the library's internals (#29112) 2024-03-25 10:33:38 +01:00
test_audio_utils.py Add MusicGen Melody (#28819) 2024-03-18 13:06:12 +00:00
test_backbone_utils.py 🚨 out_indices always a list (#30941) 2024-05-22 15:23:04 +01:00
test_chat_template_utils.py Chat Template support for function calling and RAG (#30621) 2024-06-11 15:46:38 +01:00
test_cli.py Safetensors serialization by default (#27064) 2023-10-31 19:16:49 +01:00
test_convert_slow_tokenizer.py Revert error back into warning for byte fallback conversion. (#22607) 2023-04-06 14:00:29 +02:00
test_deprecation.py Decorators for deprecation and named arguments validation (#30799) 2024-06-10 12:35:10 +01:00
test_doc_samples.py Fix the check of models supporting FA/SDPA not run (#28202) 2023-12-22 12:56:11 +01:00
test_dynamic_module_utils.py Fix the regex in get_imports to support multiline try blocks and excepts with specific exception types (#23725) 2023-05-24 15:40:19 -04:00
test_file_utils.py Inheritance-based framework detection (#21784) 2023-02-27 15:31:55 +00:00
test_generic.py Decorators for deprecation and named arguments validation (#30799) 2024-06-10 12:35:10 +01:00
test_hf_argparser.py Allow for str versions of dicts based on typing (#30227) 2024-04-16 08:15:09 -04:00
test_hub_utils.py Use HF_HUB_OFFLINE + fix has_file in offline mode (#31016) 2024-05-29 11:55:43 +01:00
test_image_processing_utils.py Add Image Processors (#19796) 2022-11-02 11:57:36 +00:00
test_image_utils.py load_image - decode b64encode and encodebytes strings (#30192) 2024-04-26 18:21:47 +01:00
test_logging.py Fix flaky test for log level (#21776) 2023-02-28 16:24:14 -05:00
test_model_card.py Black preview (#17217) 2022-05-12 16:25:55 -04:00
test_model_output.py fix / skip (for now) some tests before switch to torch 2.2 (#28838) 2024-02-02 14:11:50 +01:00
test_modeling_tf_core.py Add tf_keras imports to prepare for Keras 3 (#28588) 2024-01-30 17:26:36 +00:00
test_offline.py Use HF_HUB_OFFLINE + fix has_file in offline mode (#31016) 2024-05-29 11:55:43 +01:00
test_skip_decorators.py Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
test_versions_utils.py improve dev setup comments and hints (#28495) 2024-01-15 18:36:40 +00:00
tiny_model_summary.json Update tests/utils/tiny_model_summary.json (#29941) 2024-04-03 09:25:01 +02:00