core.upgrade

Attributes

_T

Classes

UpgradeState

Keeps the state of all upgrade steps over all modules.

upgrade_task

Marks the decorated function as an upgrade task. Upgrade tasks should

UpgradeTransaction

Holds the session and the alembic operations connection together and

UpgradeContext

Holdes the context of the upgrade. An instance of this is passed

RawUpgradeRunner

Runs the given raw tasks.

UpgradeRunner

Runs the given basic tasks.

Functions

get_distributions_with_entry_map(...)

Iterates through all distributions with entry_maps and yields

get_upgrade_modules(→ Iterator[tuple[str, ModuleType]])

Returns all modules that registered themselves for onegov.core

is_task(→ TypeGuard[_Task[_P, _T]])

Returns True if the given function is an uprade task.

get_module_tasks(→ Iterator[_Task[..., Any]])

Goes through a module or class and returns all upgrade tasks.

get_tasks_by_id(→ dict[str, _Task[..., Any]])

Takes a list of upgrade modules or classes and returns the tasks

get_module_order_key(→ Callable[[str], ...)

Returns a sort order key which orders task_ids in order of their

get_tasks(→ list[tuple[str, _Task[..., Any]]])

Takes a list of upgrade modules or classes and returns the

register_modules(→ None)

Sets up the state tracking for all modules. Initially, all tasks

register_all_modules_and_tasks(→ None)

Registers all the modules and all the tasks.

Module Contents

core.upgrade._T[source]
class core.upgrade.UpgradeState[source]

Bases: onegov.core.orm.Base, onegov.core.orm.mixins.TimestampMixin

Keeps the state of all upgrade steps over all modules.

__tablename__ = 'upgrades'[source]
module[source]
state[source]
property executed_tasks: set[str][source]
was_already_executed(task: _Task[..., Any]) bool[source]
mark_as_executed(task: _Task[..., Any]) None[source]
core.upgrade.get_distributions_with_entry_map(key: str) Iterator[tuple[Distribution, dict[str, EntryPoint]]][source]

Iterates through all distributions with entry_maps and yields each distribution along side the entry map with the given key.

core.upgrade.get_upgrade_modules() Iterator[tuple[str, ModuleType]][source]

Returns all modules that registered themselves for onegov.core upgrades like this:

entry_points={
    'onegov': [
        'upgrade = onegov.mypackage.upgrade'
    ]
}

To add multiple upgrades in a single setup.py file, the following syntax may be used. This will become the default in the future:

entry_points= {
    'onegov_upgrades': [
        'onegov.mypackage = onegov.mypackage.upgrade'
    ]
}

Note that the part before the =-sign should be kept the same, even if the location changes. Otherwise completed updates may be run again!

class core.upgrade.upgrade_task(name: str, always_run: bool = False, requires: str | None = None, raw: bool = False)[source]

Marks the decorated function as an upgrade task. Upgrade tasks should be defined outside classes (except for testing) - that is in the root of the module (directly in onegov/form/upgrades.py for example).

Each upgrade task has the following properties:

Name:

The name of the task in readable English. Do not change the name of the task once you pushed your changes. This name is used as an id!

Always_run:

By default, tasks are run once on if they haven’t been run yet, or if the module was not yet under upgrade control (it just added itself through the entry_points mechanism).

There are times where this is not wanted.

If you enable upgrades in your module and you plan to run an upgrade task in the same release, then you need to ‘always_run’ and detect yourself if you need to do any changes.

If you need to enable upgrades and run a task at the same time which can only be run once and is impossible to make idempotent, then you need to make two releases. One that enables upgade control and a second one that

Also, if you want to run an upgrade task every time you upgrade (for example, for maintenance), then you should also use it.

The easiest way to use upgrade control is to enable it from the first release on.

Another way to tackle this is to write update methods idempotent as often as possible and to always run them.

Requires:

A reference to another projects upgrade task that should be run first.

For example:

requires='onegov.form:Add new form field'

If the reference cannot be found, the upgrade fails.

Raw:

A raw task is run without setting up a request. Instead a database connection and a list of targeted schemas is passed. It’s up to the upgrade function to make sure the right schemas are targeted through that connection!

This is useful for upgrades which need to execute some raw SQL that cannot be executed in a request context. For example, this is a way to migrate the upgrade states table of the upgrade itself.

Raw tasks are always run and there state is not recorded. So the raw task needs to be idempotent and return False if there was no change.

Only use this if you have no other way. This is a very blunt instrument only used under special circumstances.

Note, raw tasks may be normal function OR generators. If they are generators, then with each yield the transaction is either commited or rolled-back depending on the dry-run cli argument.

Always run tasks may return False if they wish to be considered as not run (and therefore not shown in the upgrade runner).

name[source]
always_run[source]
requires[source]
raw[source]
__call__(fn: UpgradeFunc) UpgradeTask[source]
__call__(fn: RawFunc) RawTask
core.upgrade.is_task(function: Callable[_P, _T]) TypeGuard[_Task[_P, _T]][source]

Returns True if the given function is an uprade task.

core.upgrade.get_module_tasks(module: types.ModuleType) Iterator[_Task[..., Any]][source]

Goes through a module or class and returns all upgrade tasks.

core.upgrade.get_tasks_by_id(upgrade_modules: Iterable[tuple[str, ModuleType]]) dict[str, _Task[..., Any]][source]

Takes a list of upgrade modules or classes and returns the tasks keyed by id.

core.upgrade.get_module_order_key(tasks: Mapping[str, _Task[..., Any]]) Callable[[str], SupportsRichComparison][source]

Returns a sort order key which orders task_ids in order of their module dependencies. That is a task from onegov.core is sorted before a task in onegov.user, because onegov.user depends on onegov.core.

This is used to order unrelated tasks in a sane way.

core.upgrade.get_tasks(upgrade_modules: Iterable[tuple[str, ModuleType]] | None = None) list[tuple[str, _Task[..., Any]]][source]

Takes a list of upgrade modules or classes and returns the tasks that should be run in the order they should be run.

The order takes dependencies into account.

core.upgrade.register_modules(session: sqlalchemy.orm.Session, modules: Iterable[tuple[str, ModuleType]], tasks: Collection[tuple[str, _Task[..., Any]]]) None[source]

Sets up the state tracking for all modules. Initially, all tasks are marekd as executed, because we assume tasks to upgrade older deployments to a new deployment.

If this is a new deployment we do not need to execute these tasks.

Tasks where this is not desired should be marked as ‘always_run’. They will then manage their own state (i.e. check if they need to run or not).

This function is idempotent.

core.upgrade.register_all_modules_and_tasks(session: sqlalchemy.orm.Session) None[source]

Registers all the modules and all the tasks.

class core.upgrade.UpgradeTransaction(context: UpgradeContext)[source]

Holds the session and the alembic operations connection together and commits/aborts both at the same time.

Not really a two-phase commit solution. That could be accomplished but for now this suffices.

operations_transaction[source]
session[source]
flush() None[source]
commit() None[source]
abort() None[source]
class core.upgrade.UpgradeContext(request: core.request.CoreRequest)[source]

Holdes the context of the upgrade. An instance of this is passed to each upgrade task.

operations: Any[source]
request[source]
app[source]
schema[source]
engine[source]
conn[source]
operations_connection: sqlalchemy.engine.Connection[source]
operations[source]
begin() UpgradeTransaction[source]
has_column(table: str, column: str) bool[source]
has_enum(enum: str) bool[source]
has_table(table: str) bool[source]
models(table: str) Iterator[Any][source]
records_per_table(table: str, columns: Iterable[Column[Any]] | None = None) Iterator[Any][source]
stop_search_updates() Iterator[None][source]
is_empty_table(table: str) bool[source]
add_column_with_defaults(table: str, column: Column[_T], default: _T | Callable[[Any], _T]) None[source]
class core.upgrade.RawUpgradeRunner(tasks: Sequence[tuple[str, RawTask]], commit: bool = True, on_task_success: TaskCallback | None = None, on_task_fail: TaskCallback | None = None)[source]

Runs the given raw tasks.

tasks[source]
commit[source]
_on_task_success[source]
_on_task_fail[source]
run_upgrade(dsn: str, schemas: Sequence[str]) int[source]
class core.upgrade.UpgradeRunner(modules: Sequence[tuple[str, ModuleType]], tasks: Sequence[tuple[str, UpgradeTask]], commit: bool = True, on_task_success: TaskCallback | None = None, on_task_fail: TaskCallback | None = None)[source]

Runs the given basic tasks.

states: dict[str, UpgradeState][source]
modules[source]
tasks[source]
commit[source]
states[source]
_on_task_success[source]
_on_task_fail[source]
on_task_success(task: UpgradeTask) None[source]
on_task_fail(task: UpgradeTask) None[source]
get_state(context: UpgradeContext, module: str) UpgradeState[source]
register_modules(request: core.request.CoreRequest) None[source]
get_module_from_task_id(task_id: str) str[source]
run_upgrade(request: core.request.CoreRequest) int[source]