Alembic migration fails when moving from 2.10.5 to 3.1.5

[ladandu]

I'm facing this issue
when running migration for upgrade from 2.10.5 to 3.1.5 using airflow db migrate I see this issue

File "/usr/local/airflow/venv/lib/python3.12/site-packages/airflow/migrations/versions/0041_3_0_0_rename_dataset_as_asset.py", line 171, in upgrade
    op.rename_table(original_name, new_name)
  File "<string>", line 8, in rename_table
  File "<string>", line 3, in rename_table
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/alembic/operations/ops.py", line 1486, in rename_table
    return operations.invoke(op)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/alembic/operations/base.py", line 441, in invoke
    return fn(self, operation)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/alembic/operations/toimpl.py", line 143, in rename_table
    operations.impl.rename_table(
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/alembic/ddl/impl.py", line 415, in rename_table
    self._exec(
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/alembic/ddl/impl.py", line 246, in _exec
    return conn.execute(construct, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/future/engine.py", line 286, in execute
    return self._execute_20(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
    return connection._execute_ddl(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/engine/base.py", line 1477, in _execute_ddl
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/airflow/venv/lib64/python3.12/site-packages/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.ProgrammingError: (MySQLdb.ProgrammingError) (1146, "Table 'autonoc_airflow.dataset_alias_dataset' doesn't exist")
[SQL: ALTER TABLE dataset_alias_dataset RENAME TO asset_alias_asset]
(Background on this error at: https://sqlalche.me/e/14/f405)

Key Observation

After the failure:
✓ asset_alias_asset table EXISTS (renamed version)
✗ dataset_alias_dataset table DOES NOT EXIST (old version)
✓ Alembic version shows 3a8972ecb8f9 (migration marked complete)
This indicates: The rename succeeded, but post-rename operations fail because they reference the old table name.

Environment

Airflow: 2.10.5 → 3.1.5
Database: MySQL 9.x
Deployment: Kubernetes single pod
Alembic revision: 3a8972ecb8f9

[raphaelauv]

try to re-run the 2.10.5 database migration first