-
Notifications
You must be signed in to change notification settings - Fork 774
Flyte 2 WIP #6583
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Flyte 2 WIP #6583
Conversation
* Add flyteidl and minimal publishing workflows Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * pretend version for buf python Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
|
Bito Automatic Review Skipped - Branch Excluded |
* python version fix and use OIDC to authenticate crates Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix python versioning Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Set version before setting up env Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * echo version Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Write to GITHUB_ENV Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Use env: Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf generate Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
|
Bito Automatic Review Skipped - Branch Excluded |
* Mark go package as /v2 to be able to import both at the same time Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Upgrade to go 1.24 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Rename python, rust and npm to flyteidl2 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
|
Bito Automatic Review Skipped - Branch Excluded |
…6610) * trying idl2 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update trace def Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename idl2 to flyteidl2 Signed-off-by: machichima <nary12321@gmail.com> * generate -> buf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename idl2 to flyteidl2 Signed-off-by: machichima <nary12321@gmail.com> * build: re-gen Signed-off-by: machichima <nary12321@gmail.com> * fix: exclude python venv Signed-off-by: machichima <nary12321@gmail.com> * Add translator service Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf gen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * remove node execution service proto Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Breakup task service into a separate package Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Help strings for make file targets Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update TaskLog with read bool Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Trying the updated protovalidate plugin Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * protovalidate with semver Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try claude :shrug Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add buf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update deps Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * build: remove generate validate for python Signed-off-by: machichima <nary12321@gmail.com> * build: use managed mode on buf.gen.yaml Signed-off-by: machichima <nary12321@gmail.com> * Update validate go plugin Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update imagebuild Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix oneof buf lint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add deps to generated python code Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Be explicit about the index Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * updated idl Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Catch up to main Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Generate mocks for workflow package Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * generate mocks for other packages Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf improvements Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * updates Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * feat: generate mock for common Signed-off-by: machichima <nary12321@gmail.com> * use make gen in GH Actions Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Do not setup python env in generation CI Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * setup uv Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * set working directory Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Try to reference files by path Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Explicitly move .venv to root Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try exclude in buf.yaml Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * set UV_PROJECT_ENVIRONMENT Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * download tooling Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add which mockery Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com>
|
Bito Automatic Review Skipped - Branch Excluded |
* Add Trigger APIs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update triggers Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
|
Bito Automatic Review Skipped - Branch Excluded |
* Fix cargo build and add CI check Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint lib.rs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update Readme and contributing guides Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * signed? Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build-crate target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix included ts files in npm Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
|
Bito Automatic Review Skipped - Branch Excluded |
* Add device class to `GpuAccelerator` Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * make gen Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add HABANA_GAUDI as supported device class Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
|
Bito Automatic Review Skipped - Branch Excluded |
* Clean up triggers api Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * make gen Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> --------- Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Kevin Su <pingsutw@apache.org>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Kevin Su <pingsutw@apache.org>
This reverts commit ab2f36b.
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: Kevin Su <pingsutw@apache.org>
|
Bito Automatic Review Skipped - Branch Excluded |
* refactor: split out action and general interfaces Signed-off-by: machichima <nary12321@gmail.com> * refactor: update state service Signed-off-by: machichima <nary12321@gmail.com> * chore: remove cmd in gitignore Signed-off-by: machichima <nary12321@gmail.com> * fix: add repository.go as entry point for repository inerface Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
|
Bito Automatic Review Skipped - Branch Excluded |
Signed-off-by: machichima <nary12321@gmail.com>
|
Bito Automatic Review Skipped - Branch Excluded |
* Migrate CoPilot Signed-off-by: Barry Wu <a0987818905@gmail.com> * Update: Add cmd/ and delete go.mod and go.sum in copilot Signed-off-by: Barry Wu <a0987818905@gmail.com> * Fix CI error Signed-off-by: Barry Wu <a0987818905@gmail.com> * Update make gen changes Signed-off-by: Barry Wu <a0987818905@gmail.com> --------- Signed-off-by: Barry Wu <a0987818905@gmail.com>
|
Bito Automatic Review Skipped - Branch Excluded |
* feat: add task model and use db&gorm tag together Signed-off-by: machichima <nary12321@gmail.com> * docs: update development guidance Signed-off-by: machichima <nary12321@gmail.com> * feat: init moving task service Signed-off-by: machichima <nary12321@gmail.com> * feat: add filter and sort for task service Signed-off-by: machichima <nary12321@gmail.com> * feat: mount task service Signed-off-by: machichima <nary12321@gmail.com> * refactor: clean up Signed-off-by: machichima <nary12321@gmail.com> * docs: add task service api endpoint docs Signed-off-by: machichima <nary12321@gmail.com> * test: add unit test for repository/impl Signed-off-by: machichima <nary12321@gmail.com> * test: unit test for task transformers Signed-off-by: machichima <nary12321@gmail.com> * feat: add mocks for repo interfaces Signed-off-by: machichima <nary12321@gmail.com> * test: unit test for runs/service Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: add migration for task and task spec Signed-off-by: machichima <nary12321@gmail.com> * refactor: format Signed-off-by: machichima <nary12321@gmail.com> * docs: add example script for create and list task Signed-off-by: machichima <nary12321@gmail.com> * fix: remove unused impl/constant.go Signed-off-by: machichima <nary12321@gmail.com> * fix: prevent overwrite filter Signed-off-by: machichima <nary12321@gmail.com> * refactor: move To... function to transformers Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
|
Bito Automatic Review Skipped - Branch Excluded |
* refactor: rename tests/ to test/ Signed-off-by: machichima <nary12321@gmail.com> * feat: add test setup and cleanp Signed-off-by: machichima <nary12321@gmail.com> * test: add deploy task api test Signed-off-by: machichima <nary12321@gmail.com> * docs: update make commands and docs Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
* added timestamps Signed-off-by: “Alex <alexchien130@gmail.com> * added test and match CI Signed-off-by: “Alex <alexchien130@gmail.com> * cleaner testing Signed-off-by: “Alex <alexchien130@gmail.com> * better readability Signed-off-by: “Alex <alexchien130@gmail.com> * remove empty line Signed-off-by: “Alex <alexchien130@gmail.com> --------- Signed-off-by: “Alex <alexchien130@gmail.com>
* feat: remove phase and message Signed-off-by: machichima <nary12321@gmail.com> * feat: add condition type and reason constant Signed-off-by: machichima <nary12321@gmail.com> * feat: use condition instead of phase Signed-off-by: machichima <nary12321@gmail.com> * feat: add read task spec Signed-off-by: machichima <nary12321@gmail.com> * fix: update go mod to build executor correctly Signed-off-by: machichima <nary12321@gmail.com> * feat: add printcolumn for taskaction crd Signed-off-by: machichima <nary12321@gmail.com> * build: update dockerfile and makefile to build successfully Signed-off-by: machichima <nary12321@gmail.com> * docs: add development guide and readme Signed-off-by: machichima <nary12321@gmail.com> * feat: add mockery for workflowconnect Signed-off-by: machichima <nary12321@gmail.com> * test: update taskaction controller test to make it pass Signed-off-by: machichima <nary12321@gmail.com> * feat: add validation for ParentActionName Signed-off-by: machichima <nary12321@gmail.com> * fix: revert changes in kustomization yaml Signed-off-by: machichima <nary12321@gmail.com> * docs: update typo Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
* add RawOutputDataConfig Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * add ExecutionEnv idl Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * add Node related idl Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove datacatalog Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * re-add mocks Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove dynamic job spec Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * update cache client Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen storage metadata mock Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * set v2 scope for prom request metrics Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over grpcutils metrics Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * move secret manager impl to executor Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * copy over garbage collectable changes Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove v1 proto defs Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * remove execution_envs def Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * tidy Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> * gen Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> --------- Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
* feat: Add test for TaskService.ListVersions and test script Signed-off-by: Kevin Liao <q85292542000@gmail.com> * Update generated files Signed-off-by: Kevin Liao <q85292542000@gmail.com> * Revert "Update generated files" This reverts commit 4e7bb31. Signed-off-by: Kevin Liao <q85292542000@gmail.com> * make gen Signed-off-by: Kevin Liao <q85292542000@gmail.com> * Revert "make gen" This reverts commit e590fed. Signed-off-by: Kevin Liao <q85292542000@gmail.com> * make gen Signed-off-by: Kevin Liao <q85292542000@gmail.com> * Remove cleanup db Signed-off-by: Kevin Liao <q85292542000@gmail.com> * update uv.lock Signed-off-by: Kevin Liao <q85292542000@gmail.com> --------- Signed-off-by: Kevin Liao <q85292542000@gmail.com>
* refactor: remove unused function Signed-off-by: machichima <nary12321@gmail.com> * feat: fail directly if test server failed to start Signed-off-by: machichima <nary12321@gmail.com> * refactor: fix typo Signed-off-by: machichima <nary12321@gmail.com> * test: loop through model slice for cleanup Signed-off-by: machichima <nary12321@gmail.com> * test: clean up to defer block Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
* refactor: define all mockery in .mockery.yaml - Centralize mockery configuration in .mockery.yaml - Remove //go:generate mockery comments from interface files - Add configurations for flyteplugins and flytestdlib packages - Add global filename, inpackage, and outpkg configurations - Remove all 'case: underscore' settings (use global filename template instead) This change consolidates all mock generation configuration into a single location, making it easier to maintain and ensuring consistency across the codebase. This ensures all mock files: - Are named in snake_case via filename template - Are placed in mocks/ directories with package mocks - Follow consistent naming conventions across the codebase Fixes #6815 Signed-off-by: majiayu000 <1835304752@qq.com> * Run make gen Signed-off-by: majiayu000 <1835304752@qq.com> * Update tests for new mockery output Signed-off-by: majiayu000 <1835304752@qq.com> * Regenerate mocks after rebase Signed-off-by: majiayu000 <1835304752@qq.com> * build: move with-expecter to global settings Signed-off-by: machichima <nary12321@gmail.com> * fix: build error in flyteplugins Signed-off-by: machichima <nary12321@gmail.com> * test: convert On to EXPECT in flyteplugins tests Signed-off-by: machichima <nary12321@gmail.com> * test: more On to EXPECT in flyteplugins Signed-off-by: machichima <nary12321@gmail.com> * test: more On to EXPECT in flyteplugins Signed-off-by: machichima <nary12321@gmail.com> * test: On to EXPECT in executor/ Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: majiayu000 <1835304752@qq.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com>
* test: add API test for ListTasks endpoint Signed-off-by: Ryan Lo <wenchih@apache.org> * make gen Signed-off-by: Ryan Lo <wenchih@apache.org> * Revert "make gen" This reverts commit 9ae4c78. Signed-off-by: Ryan Lo <wenchih@apache.org> * chore: make gen & add test tasks Signed-off-by: Ryan Lo <wenchih@apache.org> --------- Signed-off-by: Ryan Lo <wenchih@apache.org>
* build: ignore Cargo.lock Signed-off-by: machichima <nary12321@gmail.com> * build: update Makefile Signed-off-by: machichima <nary12321@gmail.com> * fix: ignore Cargo.lock in dockerignore Signed-off-by: machichima <nary12321@gmail.com> * build: git rm cargo lock Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
* feat: add data proxy proto for CreateUploadLocationRequest Signed-off-by: machichima <nary12321@gmail.com> * feat: add skeleton of data proxy service Signed-off-by: machichima <nary12321@gmail.com> * feat: update file exists and create storage path logic Signed-off-by: machichima <nary12321@gmail.com> * build: gen mockery for metadata Signed-off-by: machichima <nary12321@gmail.com> * feat: move out validation function Signed-off-by: machichima <nary12321@gmail.com> * test: add unit test Signed-off-by: machichima <nary12321@gmail.com> * build: add main.go to run dataproxy Signed-off-by: machichima <nary12321@gmail.com> * docs: add config example for data proxy and data storage Signed-off-by: machichima <nary12321@gmail.com> * docs: add development/readme and minio setup for develop locally Signed-off-by: machichima <nary12321@gmail.com> * docs: add example script Signed-off-by: machichima <nary12321@gmail.com> * docs: update config example YAML Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: add AddContentMd5Metadata back and make compatible with flyteidl Signed-off-by: machichima <nary12321@gmail.com> * fix: AddContentMD5Metadata and base64 encode ContentMD5 Signed-off-by: machichima <nary12321@gmail.com> * feat: run all service (include dataproxy) in same port Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: update mockery Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: WangWang0226 <eeha8834@gmail.com>
Tracking issue
Why are the changes needed?
What changes were proposed in this pull request?
How was this patch tested?
Labels
Please add one or more of the following labels to categorize your PR:
This is important to improve the readability of release notes.
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link