Is your enhancement request related to a problem? Please describe.
Yes, we have new conformance tests GatewayWeightedAcrossTwoInferencePools that fails in the pipeline since we do not allow multiple backend refs when Inference Pool is the backend type.

What would you like to be added:
Support multiple backend refs for Inference Pools

Why this is needed:
To be gateway API inference extension conformant

Additional context
Add any other context or screenshots about the enhancement request here.