use Codewithkyrian\Transformers\Tokenizers\AutoTokenizer;

[swanghao]

System Info

"codewithkyrian/transformers": "^0.6.2"

PHP Version

8.4.14

Environment/Platform

• Command-line application
• Web application
• Serverless
• Other (please specify)

Description

The Tokenizers directory does not contain AutoTokenizer; the provided example is located under the PreTrainedTokenizers directory.

Reproduction

use Codewithkyrian\Transformers\Models\Auto\AutoModel;
use Codewithkyrian\Transformers\Tokenizers\AutoTokenizer;

$tokenizer = AutoTokenizer::fromPretrained('Xenova/toxic-bert');
$model = AutoModel::fromPretrained('Xenova/toxic-bert');

$input = 'I hate you so much';
$encodedInput = $tokenizer($input, padding: true, truncation: true);

$output = $model($encodedInput);