MDEV-38877: Unnecessary filesort on derived table materialization

[OmarGamal10]

• Fixes MDEV-38877

Why it happens?

The optimizer flags outer references for subqueries as constants, so that for every re-execution for the subquery, index is skipped, as filtering on a constant makes all rows have the same value already.

• Consider this example
  SELECT * FROM t2 JOIN (SELECT groups_20, MAX(b) FROM t1 GROUP BY groups_20) DT ON t2.a = groups_20;

• After hours of investigation, I found that the index is bypassed because table->const_key_parts incorrectly flags the [GROUP/ORDER] BY column as a constant. This optimization is correct for Nested Loop / Lateral joins since the subquery is re-executed for each outer row, the join column is a literal constant in this context, making index usage/sorting redundant.
  However, if the optimizer decides to materialize the subquery, the subquery is executed once to build a table. In this context, the column is a variable, not a constant.

  • The constant flag tricks the optimizer into assuming the data is already sorted.
  • The optimizer skips the index.
  • The optimizer finds out that data needs grouping/sorting and is not constant, falling back to a full table scan and a filesort.

• The fix is a guard condition to prevent treating an outer reference as a constant in case of derived tables.

• Before

• After