Sequencing monitoring by RyandenOtter · Pull Request #5 · jaketf/bigquery-utils

RyandenOtter · 2021-01-13T00:03:35Z

Here are the updates for the monitoring views.

... on multiple projects fixup!

Add an environment variable to support overriding the default project for the BigQuery Client. By default this will be the project in which the cloud function is deployed.

Change the tests to use new utils module.

Change import pattern

Move utility methods into a utils module

* Restructures code into constants and exception modules * Implements Backlog Publisher / Subscriber algorithm for ordering incrementals * Implements basic integration tests for Publisher / Subscriber

jaketf · 2021-01-13T00:09:58Z

you need to sign the first two commits.

git commit --amend --author="Author Name <email@address.com>"

jaketf

I'd like if the SQL had more structured place holders for dataset / project ids and a script to render them.
For example you could use envsubst
This way we could render them and dry run them in the cloud build so we'd have some basic level of CI on these queries.

tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest/README.md

tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest/monitoring/gcf_ingest_log.sql

tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest/monitoring/monitored_tables.sql

Adding the views and tables for ingest monitoring Remove unneeded sql Update README.md Fix formatting issues on the tables.

…/README.md Re-Wording Co-authored-by: Jacob Ferriero <jferriero@google.com>

…/README.md Co-authored-by: Jacob Ferriero <jferriero@google.com>

…query-utils into sequencing-monitoring

Adding additional information to the READMEmd

tools/cloud_functions/gcs_event_based_ingest/cloudbuild.yaml

tools/cloud_functions/gcs_event_based_ingest/tests/gcs_ocn_bq_ingest/test_monitoring_tables.sh

tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest/README.md

...unctions/gcs_event_based_ingest/gcs_ocn_bq_ingest/monitoring/monitoring_tables_and_views.sql

tools/cloud_functions/gcs_event_based_ingest/tests/gcs_ocn_bq_ingest/test_monitoring_tables.sh

Co-authored-by: Jacob Ferriero <jferriero@google.com>

…ingest/test_monitoring_tables.sh Co-authored-by: Jacob Ferriero <jferriero@google.com>

Move the monitoring.sql files up a directory Split out the monitoring sql into its own files (Note the numbering is to ensure the files are listed in the correct order because they are all dependent on one another) test_monitoring_tables.sh Made some alterations to test_monitoring_tables.sh to ensure dataset gets cleaned up even if the sqls error. removed the dry run argument from the bq query to ensure the sql is actually applied. Readme: Moved the Monitoring section out to the higher-level README cloudbuild.yaml Updated the cloudbuild to point to the new location of the tests.

Fix 03_gcf_ingest_latest_by_table.sql Some more README fixes.

RyandenOtter · 2021-01-23T19:02:11Z

Hey! It looks like I don't have permissions to delete datasets in your test project, which is causing my local tests to fail.
It also looks like the gcs_event_based_ingest_ci image doesn't have envsubst installed, which is causing the build to fail.

jaketf · 2021-01-27T23:48:22Z

tools/cloud_functions/gcs_event_based_ingest/monitoring/01_monitored_tables.sql

@@ -0,0 +1,28 @@
+/*


is there a reasoning for the numbering 01, 02, 03 in these query filenames? can we remove that?

I see you need to run these sql in a certain order because they create metadata entries that you depend on existing and you're doing this with sort. If there's a way to just have an ordered list in the shell script that might be better than confusing filenames IMO.

jaketf · 2021-01-27T23:49:06Z

tools/cloud_functions/gcs_event_based_ingest/monitoring/02_gcf_ingest_log.sql

+  destination_table.project_id,
+  destination_table.dataset_id,
+  destination_table.table_id,
+  start_time;


jaketf · 2021-01-27T23:50:21Z

tools/cloud_functions/gcs_event_based_ingest/tests/monitoring/test_monitoring_tables.sh

+  fi
+done <  <(find monitoring -path "*.sql" | sort)
+echo "removing dataset $DATASET"
+trap "bq rm -r -f -d $DATASET" EXIT


nit. this line will never get executed if there is a failure above. Trap statement should be moved to like 29 (see suggestion above).

Suggested change

trap "bq rm -r -f -d $DATASET" EXIT

jaketf · 2021-01-27T23:50:29Z

tools/cloud_functions/gcs_event_based_ingest/tests/monitoring/test_monitoring_tables.sh

+    exit "$result"
+  fi
+done <  <(find monitoring -path "*.sql" | sort)
+echo "removing dataset $DATASET"


Suggested change

echo "removing dataset $DATASET"

jaketf · 2021-01-27T23:56:45Z

tools/cloud_functions/gcs_event_based_ingest/tests/monitoring/test_monitoring_tables.sh

+
+export DATASET="test_monitoring_dataset_${BUILD_ID}"
+echo "Creating ${DATASET}"
+bq --location=US mk -f -d "$DATASET"


Suggested change

bq --location=US mk -f -d "$DATASET"

bq --location=US mk -f -d "$DATASET"

trap 'echo "removing dataset $DATASET" && bq rm -r -f -d $DATASET' EXIT

jaketf · 2021-01-27T23:57:18Z

tools/cloud_functions/gcs_event_based_ingest/tests/monitoring/test_monitoring_tables.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# Copyright 2019 Google Inc.


Suggested change

# Copyright 2019 Google Inc.

# Copyright 2021 Google Inc.

jaketf · 2021-01-27T23:57:34Z

tools/cloud_functions/gcs_event_based_ingest/monitoring/03_gcf_ingest_latest_by_table.sql

@@ -0,0 +1,51 @@
+/*
+# Copyright 2020 Google LLC


Suggested change

# Copyright 2020 Google LLC

# Copyright 2021 Google LLC

jaketf · 2021-01-27T23:57:41Z

tools/cloud_functions/gcs_event_based_ingest/monitoring/02_gcf_ingest_log.sql

@@ -0,0 +1,57 @@
+/*
+# Copyright 2020 Google LLC


Suggested change

# Copyright 2020 Google LLC

# Copyright 2021 Google LLC

jaketf · 2021-01-27T23:57:50Z

tools/cloud_functions/gcs_event_based_ingest/monitoring/01_monitored_tables.sql

@@ -0,0 +1,28 @@
+/*
+# Copyright 2020 Google LLC


Suggested change

# Copyright 2020 Google LLC

# Copyright 2021 Google LLC

jaketf · 2021-02-04T23:51:41Z

You can just install envsubst in Dockerfile.ci

Jacob Ferriero and others added 30 commits November 23, 2020 15:38

feat: tf gcs-bq-ingest module sets bq permissions

6fd7db9

... on multiple projects fixup!

fixup roles

d2f00ce

fixup dockerfil ci check

65d9515

docs: add note on unicode delimiters

7fdffd7

fix: update nested values in configs

537f05d

fix: external configs not found in parent dirs

02458b8

feat: bq project env-var

d951ebf

Add an environment variable to support overriding the default project for the BigQuery Client. By default this will be the project in which the cloud function is deployed.

Merge branch 'feat/bq-project-env-var'

c77908f

Merge branch 'fix/configDiscovery'

1768414

Merge branch 'docs/delimiters'

ddf3391

Merge branch 'tf/multi-project'

abbf5ea

Move utility methods into a utils module

cf23f1b

Change the tests to use new utils module.

Fix sorting issues

b52d291

Move out constants into their own file

3276214

Change import pattern

fixup! pylint

117d91b

fixup! fixup! gcb pylint issue

3770c84

Merge pull request #1 from jaketf/feat/move-utils

8b1982b

Move utility methods into a utils module

feat: sequencing with backlog publisher / subscriber

81bb167

* Restructures code into constants and exception modules * Implements Backlog Publisher / Subscriber algorithm for ordering incrementals * Implements basic integration tests for Publisher / Subscriber

fixup! mypy pylint

3c798f7

fixup! flake8

c631150

fixup! mypy tests

d5fabfa

support _config/*.sql for bq tranform sql

1c26e23

improve performance of wait_on_bq_job

d16fb1b

wip

c627af0

fixup! handle race condition

35e26d9

ordering docs and isort single line rule

8c97f5a

docs

70d2d2b

fixup linters

6ec3625

fixup import style

2d0e5a8

typing isort single line exclusion

7cb00e4

Jacob Ferriero added 5 commits January 7, 2021 18:53

chore: add test for child job failing behavior

61d2c14

fixup flake8

fb69a6a

fixup flake8

1aec908

fixup flake8

0490217

feat: separate bq storage and compute project env vars

3c3bd3d

jaketf requested changes Jan 13, 2021

View reviewed changes

There was a missing comma in the example SQL given.

41ebdbd

Adding the views and tables for ingest monitoring Remove unneeded sql Update README.md Fix formatting issues on the tables.

jaketf force-pushed the sequencing-monitoring branch from f8c6ed9 to 41ebdbd Compare January 13, 2021 00:46

RyandenOtter and others added 10 commits January 13, 2021 07:10

Update tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest…

776773c

…/README.md Re-Wording Co-authored-by: Jacob Ferriero <jferriero@google.com>

Update tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest…

f5f98e7

…/README.md Co-authored-by: Jacob Ferriero <jferriero@google.com>

Update tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest…

6bab306

…/README.md Co-authored-by: Jacob Ferriero <jferriero@google.com>

Update tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest…

6d10343

…/README.md Co-authored-by: Jacob Ferriero <jferriero@google.com>

Update tools/cloud_functions/gcs_event_based_ingest/gcs_ocn_bq_ingest…

ccf35bf

…/README.md Co-authored-by: Jacob Ferriero <jferriero@google.com>

Merge branch 'sequencing-monitoring' of https://github.com/jaketf/big…

a02a5f4

…query-utils into sequencing-monitoring

Refinement of the Views.

80ee643

Adding additional information to the READMEmd

Fix Cloudbuild

d960c25

Fix Cloudbuild

ff94cfb

Fix Cloudbuild

a623158

jaketf requested changes Jan 14, 2021

View reviewed changes

RyandenOtter and others added 6 commits January 14, 2021 14:04

Update tools/cloud_functions/gcs_event_based_ingest/cloudbuild.yaml

1e42b33

Co-authored-by: Jacob Ferriero <jferriero@google.com>

Update tools/cloud_functions/gcs_event_based_ingest/tests/gcs_ocn_bq_…

bd3a3c5

…ingest/test_monitoring_tables.sh Co-authored-by: Jacob Ferriero <jferriero@google.com>

Add 'IF NOT EXISTS' to sql CREATE statements.

e655f80

Add instructions on deploying the files. [skip ci]

c7bf900

Add more information about what each table/view does.

ca16804

Fix 03_gcf_ingest_latest_by_table.sql Some more README fixes.

jaketf requested changes Feb 4, 2021

View reviewed changes

jaketf force-pushed the sequencing-develop branch from 21bd43c to 98e7863 Compare March 31, 2021 19:45

	bq --location=US mk -f -d "$DATASET"
	bq --location=US mk -f -d "$DATASET"
	trap 'echo "removing dataset $DATASET" && bq rm -r -f -d $DATASET' EXIT

Conversation

RyandenOtter commented Jan 13, 2021

Uh oh!

jaketf commented Jan 13, 2021

Uh oh!

jaketf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RyandenOtter commented Jan 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaketf commented Feb 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

RyandenOtter commented Jan 23, 2021 •

edited

Loading