Refactor OmniSci to Heavy by tupui · Pull Request #454 · heavyai/rbc

tupui · 2022-03-15T13:46:27Z

Closes #451

pearu

Thanks, @tupui for this work!

I have a couple of nits.

In addition, at some point, we should filter out heavydb dependent code from test_externals_cmath.py and test_externals_libdevice.py as well and move these under tests/heavyai/

rbc/heavyai/__init__.py

rbc/omniscidb.py

pearu · 2022-03-17T10:06:36Z

rbc/remotejit.py

 # we support Arrays even without omnisci, so it must be renamed and moved
 # somewhere elsef
-from .omnisci_backend import OmnisciCompilerPipeline
+from .heavyai import OmnisciCompilerPipeline


At some point (not necessarily here to keep this PR simple), we'll need to rename OmnisciCompilerPipeline as well.

@tupui , could you open an issue for renaming OmnisciCompilerPipeline?

pearu · 2022-03-17T10:21:02Z

https://github.com/xnd-project/rbc/blob/master/.github/workflows/rbc_test.yml requires an update as well. For instance, currently, CI does not run any of the heavyai tests..

Co-authored-by: Pearu Peterson <pearu.peterson@gmail.com>

…i_heavy

tupui · 2022-03-18T12:10:37Z

Most tests seem to be passing except: rbc/tests/heavyai/test_math.py::test_numpy_function[np.positive-cpu]
There is a core dump when evaluating: nb.njit(fn)

tupui · 2022-03-22T14:38:08Z

@pearu do you have a hint?

tupui · 2022-03-22T15:15:10Z

ok we debugged with @guilhermeleobas and #445 fixes the segfault.

@pearu Once this is green, I propose to merge maybe with the test skip, and then @guilhermeleobas would remove it in his PR.

(just seeing some other unrelated things I will fix)

pearu · 2022-03-22T15:21:33Z

Considering that this PR is mostly about renaming omnisci to heavydb, @tupui , @guilhermeleobas could you explain how #445 that deals with numba internals is related to the changes in this PR?

tupui · 2022-03-22T15:35:25Z

Considering that this PR is mostly about renaming omnisci to heavydb, @tupui , @guilhermeleobas could you explain how #445 that deals with numba internals is related to the changes in this PR?

AFAIU, @guilhermeleobas noticed it elsewhere.

guilhermeleobas · 2022-03-22T15:37:35Z

Considering that this PR is mostly about renaming omnisci to heavydb, @tupui , @guilhermeleobas could you explain how #445 that deals with numba internals is related to the changes in this PR?

It is a combination of factors that in the end involving Numba calling a C function that doesn't exist. I've seeing this failure before and what triggers it is running a tests that raises an exception followed by rbc/tests/heavyai/test_math.py::test_numpy_function[np.positive-cpu]. #445 solves by not generating or attempting to call the cfunc.

I had the same issue in #452

tupui · 2022-03-22T16:16:10Z

Ok seems like all CI is passing. There is one workflow failing one test but seems unrelated test_add_ints_rjit. I tried to relaunch the failed workflow but GitHub is having issues currently.

tupui · 2022-03-22T16:17:03Z

See here for to view the previous run: https://github.com/xnd-project/rbc/actions/runs/2023309018/attempts/1

tupui · 2022-03-22T17:44:18Z

@pearu @guilhermeleobas the CI is green. It was just a hiccup (maybe even related to GH's current outage).

pearu

LGTM! There are a couple of discussion items that should be addressed in a follow-up but let's land this now.

Thanks, @tupui!

pearu · 2022-03-23T07:55:50Z

.github/workflows/rbc_test.yml

          RBC_TESTS_FULL: TRUE
        run: |
-          mamba run -n rbc pytest -sv -r A rbc/tests/ -x -k omnisci
+          mamba run -n rbc pytest -sv -r A rbc/tests/ -x -k heavyai


Suggested change

mamba run -n rbc pytest -sv -r A rbc/tests/ -x -k heavyai

mamba run -n rbc pytest -sv -r A rbc/tests/heavyai/ -x

However, let's leave this change to a follow-up PR when heavydb stuff from tests/test_externals_xyz.py is moved under tests/heavyai. @tupui could you open an issue for this task?

pearu · 2022-03-23T07:59:36Z

rbc/remotejit.py

 # we support Arrays even without omnisci, so it must be renamed and moved
 # somewhere elsef
-from .omnisci_backend import OmnisciCompilerPipeline
+from .heavyai import OmnisciCompilerPipeline


@tupui , could you open an issue for renaming OmnisciCompilerPipeline?

tupui added 4 commits March 15, 2022 13:58

Refactor omniscidb.py to havydb.py

e46307c

Refactor omnisci_backend to heavyai and move tests to heavyai folder

6e23d05

Refactor omnisci_xxx to heavyai/xxx

09233b7

Refactor externals/omniscidb to heavydb

9a5e61d

tupui added heavydb Related to heavydb server deprecation To deprecating something labels Mar 15, 2022

tupui self-assigned this Mar 15, 2022

Refactor omnisci fixture

86b2888

tupui force-pushed the tupui/rebrand_omni_heavy branch from f411e24 to 86b2888 Compare March 15, 2022 13:54

tupui added 3 commits March 15, 2022 15:50

Fix heavydbdb

d5fb5ac

Add omniscidb.py for smoother transition

8e20f59

Revert wrong edit

7468cae

tupui force-pushed the tupui/rebrand_omni_heavy branch 2 times, most recently from 8f2ed88 to b3d6bfe Compare March 15, 2022 17:58

Fix circular imports

4285033

tupui force-pushed the tupui/rebrand_omni_heavy branch from b3d6bfe to 4285033 Compare March 16, 2022 09:32

pearu suggested changes Mar 17, 2022

View reviewed changes

tupui and others added 7 commits March 17, 2022 16:53

Run heavyai tests in CI

f6a3afc

Relative imports and improve warning message

e48b5a3

Co-authored-by: Pearu Peterson <pearu.peterson@gmail.com>

Merge branch 'master' into tupui/rebrand_omni_heavy

d911331

Fix relative imports

b442990

Fix csv path in tests

be419e7

Merge remote-tracking branch 'upstream/master' into tupui/rebrand_omn…

2ae1cac

…i_heavy

Fix tests

60c60c1

tupui requested a review from pearu March 22, 2022 07:39

[DO NOT MERGE] disable test that raises exception

7293693

guilhermeleobas force-pushed the tupui/rebrand_omni_heavy branch from 8f5f7fa to 7293693 Compare March 22, 2022 15:05

Fix global singleton

379d8d1

tupui marked this pull request as ready for review March 22, 2022 16:15

guilhermeleobas approved these changes Mar 22, 2022

View reviewed changes

pearu approved these changes Mar 23, 2022

View reviewed changes

tupui merged commit 2044a3a into heavyai:master Mar 23, 2022

tupui deleted the tupui/rebrand_omni_heavy branch March 23, 2022 08:12

	mamba run -n rbc pytest -sv -r A rbc/tests/ -x -k heavyai
	mamba run -n rbc pytest -sv -r A rbc/tests/heavyai/ -x

Conversation

tupui commented Mar 15, 2022

Uh oh!

pearu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pearu Mar 17, 2022

Choose a reason for hiding this comment

Uh oh!

pearu Mar 23, 2022

Choose a reason for hiding this comment

Uh oh!

pearu commented Mar 17, 2022

Uh oh!

tupui commented Mar 18, 2022

Uh oh!

tupui commented Mar 22, 2022

Uh oh!

tupui commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pearu commented Mar 22, 2022

Uh oh!

tupui commented Mar 22, 2022

Uh oh!

guilhermeleobas commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tupui commented Mar 22, 2022

Uh oh!

tupui commented Mar 22, 2022

Uh oh!

tupui commented Mar 22, 2022

Uh oh!

pearu left a comment

Choose a reason for hiding this comment

Uh oh!

pearu Mar 23, 2022

Choose a reason for hiding this comment

Uh oh!

pearu Mar 23, 2022

Choose a reason for hiding this comment

Uh oh!

tupui Mar 23, 2022

Choose a reason for hiding this comment

Uh oh!

pearu Mar 23, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tupui commented Mar 22, 2022 •

edited

Loading

guilhermeleobas commented Mar 22, 2022 •

edited

Loading