run_task/blueapi hangs until it times out

[RJCD-Diamond]

minimal example to show behaviour of run_task hanging.

log_detector on b21 can be run without interrupting normal operations

Looks like it's probably some race condition

import os
import time
from pathlib import Path
from blueapi.client.client import BlueapiClient
from blueapi.config import (
    ApplicationConfig,
    ConfigLoader,
)
from blueapi.service.model import TaskRequest
import saxs_bluesky.blueapi_configs

blueapi_config_path = "/dls_sw/b21/software/blueapi/scratch/saxs_bluesky/src/saxs_bluesky/blueapi_configs/b21_blueapi_config.yaml"

config_loader = ConfigLoader(ApplicationConfig)
config_loader.use_values_from_yaml(Path(blueapi_config_path))
loaded_config = config_loader.load()
CLIENT = BlueapiClient.from_config(loaded_config)

task = TaskRequest(
    name="log_detectors",
    params={},
    instrument_session="cm40643-4",
)

for i in range(10):
    print(i)
    CLIENT.run_task(task)
    time.sleep(5)

[RJCD-Diamond]

Another minimal example with plans not specific to SCM beamline

import os
import time
from pathlib import Path
from blueapi.client.client import BlueapiClient
from blueapi.config import (
    ApplicationConfig,
    ConfigLoader,
)
from blueapi.service.model import TaskRequest
import saxs_bluesky.blueapi_configs
from dodal.plan_stubs.wrapped import set_absolute

blueapi_config_path = "ixx_blueapi_config.yaml"

config_loader = ConfigLoader(ApplicationConfig)
config_loader.use_values_from_yaml(Path(blueapi_config_path))
loaded_config = config_loader.load()
CLIENT = BlueapiClient.from_config(loaded_config)

task = TaskRequest(
    name="set_absolute",
    params={"movable": SOMETHING, "value": 1},
    instrument_session="cm12345-1",
)

for i in range(10):
    print(i)
    CLIENT.run_task(task)
    time.sleep(5)

It was hanging on the second iteration on b21 at the time, but is not consistent

[RJCD-Diamond]

It's possible this will be fixed by #1232

[RJCD-Diamond]

#1232 has not fixed it

[abbiemery]

Can I ask what you are trying to do/why you are implementing this using time.sleep ? I don't know exactly what is happening without digging into it, but since this is a blocking call on that thread its a good candidate for shenanigans. If I know what you are trying to achieve we can do it in a way that works better with the eco-system.

[abbiemery]

It's hanging on the dodal wrapped versions of bps.set_absolute and set_relative at the moment.

Putting your comment from slack here as notes.

[RJCD-Diamond]

Can I ask what you are trying to do/why you are implementing this using time.sleep ? I don't know exactly what is happening without digging into it, but since this is a blocking call on that thread its a good candidate for shenanigans. If I know what you are trying to achieve we can do it in a way that works better with the eco-system.

You get the same behaviour with from dodal.plan_stubs.wrapped import sleep - if that's what you're wondering.

The sleep is just there just to stop requests coming in too frequently, in case that was an issue.

I'm trying to give a blueapi scripting interface for the beamline scientist for commissioning purposes. If I can give them the building blocks of the system, then they should have everything they need to start using it in ernest

[RJCD-Diamond]

Also when looking at the logs/messages of blueapi - the TaskRequest that it fails on is actually executed and is completed successfully without error

[RJCD-Diamond]

The absolutely most minimal example

from pathlib import Path
from blueapi.client.client import BlueapiClient
from blueapi.config import (
    ApplicationConfig,
    ConfigLoader,
)
from blueapi.service.model import TaskRequest
from dodal.plan_stubs.wrapped import set_absolute

blueapi_config_path = "ixx_blueapi_config.yaml"

config_loader = ConfigLoader(ApplicationConfig)
config_loader.use_values_from_yaml(Path(blueapi_config_path))
loaded_config = config_loader.load()
CLIENT = BlueapiClient.from_config(loaded_config)

task = TaskRequest(
    name="set_absolute",
    params={"movable": SOMETHING, "value": 1},
    instrument_session="cm12345-1",
)

for i in range(10):
    print(i)
    CLIENT.run_task(task)

[RJCD-Diamond]

Possibly related:

#1160

[olliesilvester]

@RJCD-Diamond Do you have the logs for when this last happened? We saw another weird thing happen on i04 today, I'm wondering if it's all related