Unbound type variables in Callable type alias are substituted with Any

[Drino]

Bug Report

A Callable type alias is handled like usual generic, not as Callable.
This does not allow to define a type alias for a decorator.

It can be handled via callback Protocol, but in this case it is not possible to support signature transformation via ParamSpec.

To Reproduce

Run mypy on following code:

from typing import Any, Callable, TypeAlias, TypeVar
from typing_extensions import reveal_type

TCallable = TypeVar('TCallable', bound=Callable[..., Any])

TDecorator: TypeAlias = Callable[[TCallable], TCallable]

def factory_with_unbound_type_alias() -> TDecorator:
    ...

def factory_with_bound_type_alias() -> TDecorator[TCallable]:
    ...

def some_function(request: Any) -> str:
    return 'Hello world'


reveal_type(factory_with_unbound_type_alias()(some_function)) 
# note: Revealed type is "Any"
reveal_type(factory_with_bound_type_alias()(some_function))
# note: error: Argument 1 has incompatible type "Callable[[Any], str]"; expected <nothing>
# note: Revealed type is "<nothing>"

Expected Behavior

According to docs:

A parameterized generic alias is treated simply as an original type with the corresponding type variables substituted.

So def factory_with_bound_type_alias() -> TDecorator[TCallable]: should work.

But to be honest this is pretty counterintuitive.

I'd expect:

reveal_type(factory_with_unbound_type_alias()(some_function)) 
# Revealed type is "def (request: Any) -> builtins.str"

Generally speaking I'd expect unbound type variables in Callable type alias to be bound to its call scope, not filled with Any.

It already works this way with Callable itself:

from typing import Any, Callable, Dict, TypeVar
from typing_extensions import reveal_type

T = TypeVar('T')

def callable_factory() -> Callable[[T], T]:
    ...
    
def dict_factory() -> Dict[T, T]:
    ...
    
reveal_type(callable_factory())
# note: Revealed type is "def [T] (T`-1) -> T`-1"
reveal_type(dict_factory())
# note: Revealed type is "builtins.dict[<nothing>, <nothing>]"

I'd expect it to work this way until alias has another non-callable generic depending on this variable (out of this Callable scope), e.g. current behavior in this snippet is fine:

from typing import Any, Callable, List, Union, TypeVar
from typing_extensions import reveal_type

T = TypeVar('T')

TListOrFactory = Union[List[T], Callable[[], List[T]]]

def make_list_or_factory() -> TListOrFactory:
    ...

reveal_type(make_list_or_factory()) 
# note: Revealed type is "Union[builtins.list[Any], def () -> builtins.list[Any]]"

Your Environment

gist
mypy-playground

• Mypy version used: mypy 0.971
• Python version used: 3.10

Related discussion
I've created a similar ticket in Pyright repo:
microsoft/pyright#3803

It appears that the right way to handle Callable in Pyright is by passing it a type variable:

def factory_with_bound_type_alias() -> TDecorator[TCallable]:
    ...

reveal_type(factory_with_bound_type_alias()(some_function))
# pyright: Type of "factory_with_bound_type_alias()(some_function)" is "(request: Any) -> str"

I've searched for any discussions on semantics of Callable aliases, but didn't manage to find anything.

So, after all I've created a (dead) discussion in typing repo:
python/typing#1236

[sobolevn]

So, basically these two functions produce different results, but must produce the same thing:

from typing import Any, Callable, TypeAlias, TypeVar
from typing_extensions import reveal_type

TCallable = TypeVar('TCallable', bound=Callable[..., Any])
TDecorator: TypeAlias = Callable[[TCallable], TCallable]

def a() -> TDecorator: ...
def b() -> Callable[[TCallable], TCallable]: ...

reveal_type(a())  # Revealed type is "Any"
reveal_type(b())  # Revealed type is "def [TCallable <: def (*Any, **Any) -> Any] (TCallable`-1) -> TCallable`-1"

I will take a look!

[sobolevn]

It is not just with Callable, the same problem also affects other cases:

from typing import TypeVar, List
from typing_extensions import reveal_type, TypeAlias

T = TypeVar('T')
TAlias: TypeAlias = List[T]

def a() -> TAlias: ...
def b() -> List[T]: ...

reveal_type(a())  # "builtins.list[Any]"
reveal_type(b())  # "builtins.list[<nothing>]"

[sobolevn]

One more thing: function b produces an error: Type variable "ex.T" is unbound, while a does not

[sobolevn]

This is quite hard, at the moment - I have no idea how to do that properly.
This is the place I've debug for the most amount of time:

mypy/mypy/typeanal.py

Lines 1635 to 1663 in 2ba6451

	def visit_unbound_type(self, t: UnboundType) -> TypeVarLikeList:
	name = t.name
	node = None
	# Special case P.args and P.kwargs for ParamSpecs only.
	if name.endswith("args"):
	if name.endswith(".args") or name.endswith(".kwargs"):
	base = ".".join(name.split(".")[:-1])
	n = self.lookup(base, t)
	if n is not None and isinstance(n.node, ParamSpecExpr):
	node = n
	name = base
	if node is None:
	node = self.lookup(name, t)
	if (
	node
	and isinstance(node.node, TypeVarLikeExpr)
	and (self.include_bound_tvars or self.scope.get_binding(node) is None)
	):
	assert isinstance(node.node, TypeVarLikeExpr)
	return [(name, node.node)]
	elif not self.include_callables and self._seems_like_callable(t):
	return []
	elif node and node.fullname in LITERAL_TYPE_NAMES:
	return []
	elif node and node.fullname in ANNOTATED_TYPE_NAMES and t.args:
	# Don't query the second argument to Annotated for TypeVars
	return self.query_types([t.args[0]])
	else:
	return super().visit_unbound_type(t)

[KotlinIsland]

in basedmypy typevars are allowed in the bound of other typevars.

[sobolevn]

@KotlinIsland sorry, I don't understand. Can you please clarify?

[KotlinIsland]

Oh, my mistake, this is a TypeVar in an alias, not a TypeVar in the bound of a TypeVar, sorry

[hauntsaninja]

(I believe everyone here knows this but) for what it's worth, I do not consider any of this to be a bug. Just generic type aliases going unapplied (think MyList = List[T]), as Akuli explains: python/typing#1236

Note that if you use --strict (or --disallow-any-generics), you'll get a nice error message here: error: Missing type parameters for generic type "TDecorator"

[Drino]

Thanks for the thoughtful discussion! :)

@sobolevn In your example with List you mean it should be <nothing> not Any? I believe in other means this is fine, as List doesn't possess it's own type var scope which can be resolved on call (while Callable does).

@hauntsaninja

I do not consider any of this to be a bug.

The documentation on type alias says:

A parameterized generic alias is treated simply as an original type with the corresponding type variables substituted.

Thus,

def factory_with_bound_type_alias() -> TDecorator[TCallable]:
    ...

supposed to be treated as

def factory_with_bound_type_alias() ->  Callable[[TCallable], TCallable]:
    ...

But they are different.

I believe that either documentation or implementation should be updated.

Apart from this there is an awesome docs section on decorator factories - It's probably worth to put an advice to use Protocol instead of Callable type alias as decorator shortcut (as the latter does not work). Decorator factories look like the case where people attempt to create a shortcut for a generic Callable type and it will be really nice to have some explanation there.

I also wanted to notice that this behavior (mandatory usage of type variable) is quite counterintuitive - from my experience on their first attempt nobody manages to write correct annotation using Callable type alias. Though, it seems that --strict highlight it, so this is probably not an issue.

[sirosen]

I believe that this is the explanation of this issue which came up with the click 8.1.4 release.

If we define

F = TypeVar("F", bound=Callable[..., Any])
_Decorator: TypeAlias = Callable[[F], F]

Then these two functions are treated differently

def a() -> _Decorator[F]: ...
def b() -> Callable[[F], F]: ...

I don't quite understand how the two are different from mypy's perspective -- mechanically they may have differences, but they look semantically the same to me.