core.upgrade

Attributes

_T

Classes

UpgradeState	Keeps the state of all upgrade steps over all modules.
upgrade_task	Marks the decorated function as an upgrade task. Upgrade tasks should
UpgradeTransaction	Holds the session and the alembic operations connection together and
UpgradeContext	Holdes the context of the upgrade. An instance of this is passed
RawUpgradeRunner	Runs the given raw tasks.
UpgradeRunner	Runs the given basic tasks.

Functions

get_distributions_with_entry_map(...)	Iterates through all distributions with entry_maps and yields
get_upgrade_modules(→ Iterator[tuple[str, ModuleType]])	Returns all modules that registered themselves for onegov.core
is_task(→ TypeGuard[_Task[_P, _T]])	Returns True if the given function is an uprade task.
get_module_tasks(→ Iterator[_Task[..., Any]])	Goes through a module or class and returns all upgrade tasks.
get_tasks_by_id(→ dict[str, _Task[..., Any]])	Takes a list of upgrade modules or classes and returns the tasks
get_module_order_key(→ Callable[[str], ...)	Returns a sort order key which orders task_ids in order of their
get_tasks(→ list[tuple[str, _Task[..., Any]]])	Takes a list of upgrade modules or classes and returns the
register_modules(→ None)	Sets up the state tracking for all modules. Initially, all tasks
register_all_modules_and_tasks(→ None)	Registers all the modules and all the tasks.

Module Contents

core.upgrade._T

class core.upgrade.UpgradeState

Bases: onegov.core.orm.Base, onegov.core.orm.mixins.TimestampMixin

Keeps the state of all upgrade steps over all modules.

__tablename__ = 'upgrades'

module

state

property executed_tasks: set[str]

was_already_executed(task: _Task[..., Any]) → bool

mark_as_executed(task: _Task[..., Any]) → None

core.upgrade.get_distributions_with_entry_map(key: str) → Iterator[tuple[Distribution, dict[str, EntryPoint]]]

Iterates through all distributions with entry_maps and yields each distribution along side the entry map with the given key.

core.upgrade.get_upgrade_modules() → Iterator[tuple[str, ModuleType]]

Returns all modules that registered themselves for onegov.core upgrades like this:

entry_points={
    'onegov': [
        'upgrade = onegov.mypackage.upgrade'
    ]
}

To add multiple upgrades in a single setup.py file, the following syntax may be used. This will become the default in the future:

entry_points= {
    'onegov_upgrades': [
        'onegov.mypackage = onegov.mypackage.upgrade'
    ]
}

Note that the part before the =-sign should be kept the same, even if the location changes. Otherwise completed updates may be run again!

class core.upgrade.upgrade_task(name: str, always_run: bool = False, requires: str | None = None, raw: bool = False)

Marks the decorated function as an upgrade task. Upgrade tasks should be defined outside classes (except for testing) - that is in the root of the module (directly in onegov/form/upgrades.py for example).

Each upgrade task has the following properties:

Name:

The name of the task in readable English. Do not change the name of the task once you pushed your changes. This name is used as an id!

Always_run:

By default, tasks are run once on if they haven’t been run yet, or if the module was not yet under upgrade control (it just added itself through the entry_points mechanism).

There are times where this is not wanted.

If you enable upgrades in your module and you plan to run an upgrade task in the same release, then you need to ‘always_run’ and detect yourself if you need to do any changes.

If you need to enable upgrades and run a task at the same time which can only be run once and is impossible to make idempotent, then you need to make two releases. One that enables upgade control and a second one that

Also, if you want to run an upgrade task every time you upgrade (for example, for maintenance), then you should also use it.

The easiest way to use upgrade control is to enable it from the first release on.

Another way to tackle this is to write update methods idempotent as often as possible and to always run them.

Requires:

A reference to another projects upgrade task that should be run first.

For example:

requires='onegov.form:Add new form field'

If the reference cannot be found, the upgrade fails.

Raw:

A raw task is run without setting up a request. Instead a database connection and a list of targeted schemas is passed. It’s up to the upgrade function to make sure the right schemas are targeted through that connection!

This is useful for upgrades which need to execute some raw SQL that cannot be executed in a request context. For example, this is a way to migrate the upgrade states table of the upgrade itself.

Raw tasks are always run and there state is not recorded. So the raw task needs to be idempotent and return False if there was no change.

Only use this if you have no other way. This is a very blunt instrument only used under special circumstances.

Note, raw tasks may be normal function OR generators. If they are generators, then with each yield the transaction is either commited or rolled-back depending on the dry-run cli argument.

Always run tasks may return False if they wish to be considered as not run (and therefore not shown in the upgrade runner).

name

always_run

requires

raw

__call__(fn: UpgradeFunc) → UpgradeTask

__call__(fn: RawFunc) → RawTask

core.upgrade.is_task(function: Callable[_P, _T]) → TypeGuard[_Task[_P, _T]]

Returns True if the given function is an uprade task.

core.upgrade.get_module_tasks(module: types.ModuleType) → Iterator[_Task[..., Any]]

Goes through a module or class and returns all upgrade tasks.

core.upgrade.get_tasks_by_id(upgrade_modules: Iterable[tuple[str, ModuleType]]) → dict[str, _Task[..., Any]]

Takes a list of upgrade modules or classes and returns the tasks keyed by id.

core.upgrade.get_module_order_key(tasks: Mapping[str, _Task[..., Any]]) → Callable[[str], SupportsRichComparison]

Returns a sort order key which orders task_ids in order of their module dependencies. That is a task from onegov.core is sorted before a task in onegov.user, because onegov.user depends on onegov.core.

This is used to order unrelated tasks in a sane way.

core.upgrade.get_tasks(upgrade_modules: Iterable[tuple[str, ModuleType]] | None = None) → list[tuple[str, _Task[..., Any]]]

Takes a list of upgrade modules or classes and returns the tasks that should be run in the order they should be run.

The order takes dependencies into account.

core.upgrade.register_modules(session: sqlalchemy.orm.Session, modules: Iterable[tuple[str, ModuleType]], tasks: Collection[tuple[str, _Task[..., Any]]]) → None

Sets up the state tracking for all modules. Initially, all tasks are marekd as executed, because we assume tasks to upgrade older deployments to a new deployment.

If this is a new deployment we do not need to execute these tasks.

Tasks where this is not desired should be marked as ‘always_run’. They will then manage their own state (i.e. check if they need to run or not).

This function is idempotent.

core.upgrade.register_all_modules_and_tasks(session: sqlalchemy.orm.Session) → None

Registers all the modules and all the tasks.

class core.upgrade.UpgradeTransaction(context: UpgradeContext)

Holds the session and the alembic operations connection together and commits/aborts both at the same time.

Not really a two-phase commit solution. That could be accomplished but for now this suffices.

operations_transaction

session

flush() → None

commit() → None

abort() → None

class core.upgrade.UpgradeContext(request: core.request.CoreRequest)

Holdes the context of the upgrade. An instance of this is passed to each upgrade task.

operations: Any

request

app

schema

engine

operations_connection: sqlalchemy.engine.Connection

begin() → UpgradeTransaction

has_column(table: str, column: str) → bool

has_enum(enum: str) → bool

has_table(table: str) → bool

models(table: str) → Iterator[Any]

records_per_table(table: str, columns: Iterable[Column[Any]] | None = None) → Iterator[Any]

stop_search_updates() → Iterator[None]

is_empty_table(table: str) → bool

add_column_with_defaults(table: str, column: Column[_T], default: _T | Callable[[Any], _T]) → None

class core.upgrade.RawUpgradeRunner(tasks: Sequence[tuple[str, RawTask]], commit: bool = True, on_task_success: TaskCallback | None = None, on_task_fail: TaskCallback | None = None)

Runs the given raw tasks.

tasks

commit

_on_task_success

_on_task_fail

run_upgrade(dsn: str, schemas: Sequence[str]) → int

class core.upgrade.UpgradeRunner(modules: Sequence[tuple[str, ModuleType]], tasks: Sequence[tuple[str, UpgradeTask]], commit: bool = True, on_task_success: TaskCallback | None = None, on_task_fail: TaskCallback | None = None)

Runs the given basic tasks.

states: dict[str, UpgradeState]

modules

tasks

commit

_on_task_success

_on_task_fail

on_task_success(task: UpgradeTask) → None

on_task_fail(task: UpgradeTask) → None

get_state(context: UpgradeContext, module: str) → UpgradeState

register_modules(request: core.request.CoreRequest) → None

get_module_from_task_id(task_id: str) → str

run_upgrade(request: core.request.CoreRequest) → int