gh-142418 Enhance markcoroutinefunction to support functools.partial

[chaope]

Enhance markcoroutinefunction to support functools.partial and added test to cover the corner case.

Based on the PR (#99247) that added markcoroutinefunction, the purpose is to

be able to continue to identify sync functions returning awaitables, without calling them to see that.

Given that, I think this issue indeed reports a bug. The root cause is that markcoroutinefunction sets the mark on partial function object (joke), while iscoroutinefunction looks at the underlying function object of partial function object (manufacturer_of_jokes). The deeper issue is that markcoroutinefunction and iscoroutinefunction uses different code path to find the function object it operates on.

In order to fix this bug, we can do it in two ways: fix the problem with minimum change or fix it more thoroughly, but riskier.

• minimum change: add functools._unwrap_partial to markcoroutinefunction

  • This PR implements minimum change.

• fix it more thoroughly: move the logic in _has_code_flag to a separate function and reuse it in markcoroutinefunction and _has_coroutine_mark

  • I do think if not (isfunction(f) or _signature_is_functionlike(f)): is appropriate to be applied to markcoroutinefunction and iscoroutinefunction as well, if the check fails, we raise a ValueError, but it may break user's code, open for discussion.
  • If we do want to check isfunction in iscoroutinefunction, then we don't need gh-120200: Fix inspect.iscoroutinefunction(inspect) is True corner case #120214

def _find_underlying_function(f):
    f = functools._unwrap_partialmethod(f)
    while ismethod(f):
        f = f.__func__
    f = functools._unwrap_partial(f)
    return f

def _has_code_flag(f, flag):
    f = _find_underlying_function(f)
    if not (isfunction(f) or _signature_is_functionlike(f)):
        return False
    return bool(f.__code__.co_flags & flag)

def markcoroutinefunction(f):
    f = _find_underlying_function(f)
    f._is_coroutine_marker = _is_coroutine_mark
    return f

def _has_coroutine_mark(f):
    f = _find_underlying_function(f)
    return getattr(f, "_is_coroutine_marker", None) is _is_coroutine_mark

• Issue: inspect.iscoroutinefunction() does not detect marked partial objects #142418

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[x42005e1f]

Please read the example in the original issue carefully. The original function may not always return a coroutine object depending on the parameters passed. Marking it will break the expected behavior because the return type of the partial function and the original function are different.

[chaope]

Please read the example in the original issue carefully. The original function may not always return a coroutine object depending on the parameters passed. Marking it will break the expected behavior because the return type of the partial function and the original function are different.

My understanding of the design of markcoroutinefunction and iscoroutinefunction is that it gives you a way to mark a function as coroutine, but it doesn't do any check to prevent you from falsely mark a random object as coroutine. We can have another issue discussing the design of it and if we do want to change the design of it, maybe we need a PEP.

But the issue you reported indeed reveals a bug that after a function is marked as coroutine, it is not correctly recognized by iscoroutinefunction (which is against what markcoroutinefunction and iscoroutinefunction are designed for).

To demonstrate that markcoroutinefunction can be applied to random object in python, you can try the following code, the object doesn't even need to be a callable.

>>> from inspect import iscoroutinefunction, markcoroutinefunction
>>> 
>>> class Object(object):
...     pass
... 
>>> o = Object()
>>> o = markcoroutinefunction(o)
>>> iscoroutinefunction(o)
True

[x42005e1f]

My understanding of the design of markcoroutinefunction and iscoroutinefunction is that it gives you a way to mark a function as coroutine, but it doesn't do any check to prevent you from falsely mark a random object as coroutine. We can have another issue discussing the design of it and if we do want to change the design of it, maybe we need a PEP.

How does this relate to my comment? If you unwrap an object in markcoroutinefunction(), you do not mark the passed object (instead, you mark the one it references). And this violates both the expected and current behavior.

[chaope]

As markcoroutinefunction already has

    if hasattr(func, '__func__'):
        func = func.__func__

I don't think marking the passed object is the expected behavior.

When we do markcoroutinefunction for object A, the expected behavior is that iscoroutinefunction returns True for object A.

current iscoroutinefunction code does the unwrap, which means that all of the partial function sharing the same base function should return same result for iscoroutinefunction, I think this is the right behavior, and I want to preserve it, so that my change is done on markcoroutinefunction.

[x42005e1f]

Current behavior:

>>> def f(): ...
>>> g = markcoroutinefunction(partial(f))
>>> iscoroutinefunction(g)
False
>>> iscoroutinefunction(f)
False

Expected behavior:

>>> def f(): ...
>>> g = markcoroutinefunction(partial(f))
>>> iscoroutinefunction(g)
True
>>> iscoroutinefunction(f)
False

Your behavior:

>>> def f(): ...
>>> g = markcoroutinefunction(partial(f))
>>> iscoroutinefunction(g)
True
>>> iscoroutinefunction(f)
True

[x42005e1f]

Please note that the main difference between method objects and partial objects is that the former cannot change the return value (they expect the same parameters, except for self), while the latter can narrow it down (since they can directly affect the parameters).

(However, in special cases, method objects can also narrow the return type, but these are very exotic cases, which cannot be said about partial objects).

[x42005e1f]

current iscoroutinefunction code does the unwrap, which means that all of the partial function sharing the same base function should return same result for iscoroutinefunction, I think this is the right behavior, and I want to preserve it, so that my change is done on markcoroutinefunction.

Let me clarify this point. The reason why iscoroutinefunction() returns True for any wrapping partial object may be that the marked function already has CoroutineType as its return type (you can see this even in the typeshed annotations). No matter how much we narrow this type, it will still remain a coroutine. And this rule must not be broken, otherwise we may get type errors that cannot be detected statically.

[chaope]

I understand your concern and the expected behavior you provided. Using the variable name in your issue, ideally, iscoroutinefunction(manufacturer_of_jokes) should not be True.

But then it's a question of if we want to unwrap partial function in iscoroutinefunction.

• If we don't unwrap, then the specific issue can be resolved by only looking at the object itself. But that will almost defeat the purpose of iscoroutinefunction.
• If we unwrap, then your joke can again be wrapped in partial function, what should the wrapper function return then?

from functools import partial
from inspect import iscoroutinefunction, markcoroutinefunction

async def wedonotlikesnakecase():
    return "the_funniest_joke_in_the_world"

def sync_func():
    return "sync_func"

def manufacturer_of_jokes(somefunc, another_func=None):
    global manufacturer_of_jokes
    del manufacturer_of_jokes
    if another_func is not None:
        return another_func()
    return somefunc()

joke = partial(manufacturer_of_jokes, wedonotlikesnakecase)
joke = markcoroutinefunction(joke)
joke1 = partial(joke, wedonotlikesnakecase) # This will be coroutine.
joke2 = partial(joke, sync_func)            # This will not be coroutine.
print(iscoroutinefunction(joke1), iscoroutinefunction(joke2))

[x42005e1f]

If we don't unwrap, then the specific issue can be resolved by only looking at the object itself. But that will almost defeat the purpose of iscoroutinefunction.

A solution is to check the marker for each unwrapped object (including the passed object itself, if it is not a method object). You can see this in the code attached to the original issue and in the parallel PR. Once the marker is found, any subsequent narrowed type is guaranteed to be CoroutineType (as long as the function is marked correctly).

If we unwrap, then your joke can again be wrapped in partial function, what should the wrapper function return then?

See the answer above. All explained via the type narrowing principle. Your joke() has an arbitrary return type (Any) because it can accept an arbitrary function whose value it returns when passed (applying markcoroutinefunction() to it is a big mistake). joke1() is a coroutine function and should be marked accordingly. joke2() is a string function and should not be considered a coroutine function. Your behavior violates the latter.

[x42005e1f]

If iscoroutinefunction() returns True, this should be read as "the return type is always CoroutineType (or one of its subclasses)" rather than "the return type may be CoroutineType (or one of its subclasses)". Imagine you are doing the following:

if iscoroutinefunction(func):
    await func(*args, **kwargs)

This is error-prone if the return value is not always a coroutine (and in general, what is the point of different behavior, why would iscoroutinefunction() be needed at all if it does not provide such a guarantee as long as functions are marked correctly?).