[DEV-6356] added table installation_progress by Ildyakov · Pull Request #3547 · athenianco/athenian-api

Ildyakov · 2023-04-04T18:48:16Z

This change is

Ildyakov · 2023-04-04T18:48:54Z

working with athenianco/cloud-common#138
and https://github.com/athenianco/metadata/pull/1586

lwsanty · 2023-04-04T18:55:21Z

+        ),
+        sa.Column(
+            "account_created",
+            sa.TIMESTAMP(timezone=True),


can we maybe already populate this column for existing accounts? or we don't have to?
also, do we need to fill it during the account creation in accounts table?
cc @dennwc

probably we don`t need an existing accounts at all. the point of this feature is just for tracking new installations

Agree with Olek: we should create records for existing accounts. This is where the current_status will help - we can set it, but leave timestamps empty. Maybe we can update the table later to reflect actual installations that happened in the past.

Yep, we usually do this manually with a data post-migration. It's worth setting a separate task to backfill the history.

lwsanty · 2023-04-04T18:58:56Z

+        sa.Column(
+            "account_id",
+            sa.Integer(),
+            sa.ForeignKey("accounts.id", name="fk_installation_progress_account"),
+            primary_key=True,
+        ),
+        sa.Column(
+            "account_created",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "fetch_started",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "fetch_completed",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "consistency_started",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "consistency_completed",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "precompute_started",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "precompute_completed",
+            sa.TIMESTAMP(timezone=True),
+        ),
+        sa.Column(
+            "current_status",
+            sa.Text(),
+        ),


can all these timestamps be null?

yep and they will be

dennwc · 2023-04-05T09:52:38Z

+
+    account_id = Column(
+        Integer(),
+        ForeignKey("accounts.id", name="fk_installation_progress_account"),


This FK assumes that there must be an Athenian account associated with the record. This is not the case for GitHub installations: they usually start without Athenian account. So we won't be able to insert most of the records here, unless we force Athenian account creation from MD side.

So what's your plan for the account_id column? Is it still an Athenian account ID or GitHub account ID?

this have to be a Github ID as it is in the Event

dennwc · 2023-04-05T09:54:04Z

+        ),
+        sa.Column(
+            "account_created",
+            sa.TIMESTAMP(timezone=True),


Agree with Olek: we should create records for existing accounts. This is where the current_status will help - we can set it, but leave timestamps empty. Maybe we can update the table later to reflect actual installations that happened in the past.

vmarkovtsev · 2023-04-11T07:45:58Z

+
+    __tablename__ = "installation_progress"
+
+    account_id = Column(


Could you please format every field in one line for better readability and consistency with the surrounding models 🙏 E.g.

Suggested change

account_id = Column(

account_id = Column(Integer(), primary_key=True)

Also, note that primary_key=True implies nullable=False under the hood.

vmarkovtsev · 2023-04-11T07:46:52Z

+        primary_key=True,
+    )
+    account_created = Column(
+        "account_created",


We don't have to set the name for each column, luckily. This is enough:

account_created = Column(TIMESTAMP(timezone=True))

vmarkovtsev · 2023-04-11T07:47:13Z

+        sa.Column(
+            "account_id",
+            sa.Integer(),
+            nullable=False,


Suggested change

nullable=False,

vmarkovtsev · 2023-04-11T07:48:19Z

+        ),
+        sa.Column(
+            "account_created",
+            sa.TIMESTAMP(timezone=True),


Yep, we usually do this manually with a data post-migration. It's worth setting a separate task to backfill the history.

…loyment_labels

Bumps [morcilla[postgresql,sqlite]](https://github.com/athenianco/morcilla) from 0.5.35 to 0.5.37. - [Release notes](https://github.com/athenianco/morcilla/releases) - [Commits](athenianco/morcilla@v0.5.35...v0.5.37) --- updated-dependencies: - dependency-name: morcilla[postgresql,sqlite] dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

…ackage

Bumps [sentry-sdk[pure_eval]](https://github.com/getsentry/sentry-python) from 1.17.0 to 1.19.1. - [Release notes](https://github.com/getsentry/sentry-python/releases) - [Changelog](https://github.com/getsentry/sentry-python/blob/master/CHANGELOG.md) - [Commits](getsentry/sentry-python@1.17.0...1.19.1) --- updated-dependencies: - dependency-name: sentry-sdk[pure_eval] dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [gcloud-aio-auth](https://github.com/talkiq/gcloud-aio) from 4.2.0 to 4.2.1. - [Release notes](https://github.com/talkiq/gcloud-aio/releases) - [Commits](talkiq/gcloud-aio@kms-4.2.0...auth-4.2.1) --- updated-dependencies: - dependency-name: gcloud-aio-auth dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [gcloud-aio-pubsub](https://github.com/talkiq/gcloud-aio) from 5.3.0 to 5.4.0. - [Release notes](https://github.com/talkiq/gcloud-aio/releases) - [Commits](talkiq/gcloud-aio@pubsub-5.3.0...pubsub-5.4.0) --- updated-dependencies: - dependency-name: gcloud-aio-pubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [types-python-dateutil](https://github.com/python/typeshed) from 2.8.19.11 to 2.8.19.12. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-python-dateutil dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.2.2 to 7.2.3. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.2.2...7.2.3) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [faker](https://github.com/joke2k/faker) from 18.3.1 to 18.4.0. - [Release notes](https://github.com/joke2k/faker/releases) - [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md) - [Commits](joke2k/faker@v18.3.1...v18.4.0) --- updated-dependencies: - dependency-name: faker dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.2.2 to 7.3.0. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.2.2...7.3.0) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

vmarkovtsev · 2023-04-12T19:46:57Z

@Ildyakov merge conflicts, let's rebase

vmarkovtsev · 2023-04-13T09:53:40Z

I think that there is a new clean PR: #3564

[DEV-6356] added table installation_progress

b992d89

Ildyakov requested a review from lwsanty April 4, 2023 18:48

lwsanty requested a review from dennwc April 4, 2023 18:52

lwsanty reviewed Apr 4, 2023

View reviewed changes

aleksei added 2 commits April 4, 2023 14:29

[DEV-6356] update for migration

6955cab

[DEV-6356] flake update

dea89ff

dennwc requested changes Apr 5, 2023

View reviewed changes

aleksei added 2 commits April 5, 2023 08:17

[DEV-6356] update instaprogress table

6fa6add

[DEV-6356] migration update

0da39ac

vmarkovtsev requested changes Apr 11, 2023

View reviewed changes

aleksei and others added 18 commits April 11, 2023 09:18

[DEV-6356] openapi update

48d6158

[DEV-6356] models restore

0b55fdf

[DEV-6356] fixes for migration

5634945

[DEV-6264] Unbuffered output for show_logged_query.py gcloud subproc

3f00fdd

[DEV-6264] Rename deployment_labels controllers module

375f12d

[DEV-6264] Implement internal function get_deployment_labels

d1ffa7b

[DEV-6264] Define DeploymentLabelsResponse web model

5abad5a

[DEV-6264] Implement get_deployment_labels controller

1356611

[DEV-6264] Implement internal upsert_deployment_labels and delete_dep…

a49ced2

…loyment_labels

[DEV-6264] Define DeploymentModifyLabelsRequest web model

7093240

[DEV-6264] Implement modify_deployment_labels controller

4ae7d01

[DEV-6264] Move huge test data to a separate module

c756fe4

[DEV-6264] Move a couple of misplaced tests

1768fbe

[DEV-6264] Implement invalidate_precomputed_on_labels_change()

22afbc6

[DEV-6264] Invalidate precomputed facts on deployment labels change

9e76d64

Bump version: 0.20.40 → 0.20.41

0e722cc

[DEV-6387] Add jira-acknowledge-time-below-threshold-ratio metric

0f0959c

[DEV-6387] Validate jira-ack-time threshold metric in goal update

6dd8683

gkwillie and others added 23 commits April 11, 2023 10:40

Bump version: 0.20.41 → 0.20.42

c75df89

[DEV-6403] Fix mismatching columns in extended pr deployments

dd1397c

[DEV-6368] Move invalidate_precomputed_on_labels_change() to miners p…

f447930

…ackage

[DEV-6368] Expose all_labels property in LogicalDeploymentSettings

6b4c157

[DEV-6230] Drop GraphQL endpoint and dependencies

2ab0a34

Bump version: 0.20.42 → 0.20.43

82ba63f

[DEV-6264] Fix alembic conf for rdb

fedac7e

[DEV-6264] Fix __table_args__ typo in some rdb models

f223233

Optimize _delete_deployments

4e9aa26

Bump version: 0.20.43 → 0.20.44

23ed10a

[DEV-6399] Optimize load_precomputed_done_candidates()

f26d5e9

Bump version: 0.20.44 → 0.20.45

69be81d

[DEV-6399] Solve performance bugs in query_jira_raw()

a0ab1ce

Bump version: 0.20.45 → 0.20.46

ca798c4

[DEV-6356] openapi update

4e5711e

Ildyakov requested a review from a team as a code owner April 11, 2023 15:40

vmarkovtsev closed this Apr 13, 2023

vmarkovtsev deleted the DEV-6356-migration-installation-table-create branch April 13, 2023 09:53

	account_id = Column(
	account_id = Column(Integer(), primary_key=True)

Conversation

Ildyakov commented Apr 4, 2023 • edited by eiso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ildyakov commented Apr 4, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vmarkovtsev commented Apr 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vmarkovtsev commented Apr 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Ildyakov commented Apr 4, 2023 •

edited by eiso

Loading

vmarkovtsev commented Apr 12, 2023 •

edited

Loading