Use accelerate for spacy-llm to offload to CPU via device_map

[marcg03]

I want to integrate an LLM into my spacy pipeline but my GPU doesn't have enough space.

I read here that accelerate could be used with the option device_map=auto to offload to the CPU but I don't know how to adapt a basic example to do that.

For example how would you do this for the following example (taken from here):

from spacy_llm.util import assemble

nlp = assemble("config.cfg")
doc = nlp("Jack and Jill rode up the hill in Les Deux Alpes")
print([(ent.text, ent.label_) for ent in doc.ents])

[Cyrille37]

Hi
I also try to split workload between GPU and CPU but I could not find how to use "accelerate" with "device_map=auto".

I can only get it works 100% GPU or 100% CPU 😞

[components.llm.model]
...
config_init = {"device_map": "auto"}

Perhaps the config file is not enough, and some Python instructions are mandatory ?