tqdm style progress bar for dagster integration #337

geoHeil · 2025-11-22T14:41:48Z

resolves: #333

geoHeil · 2025-11-22T14:42:06Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

tqdm style progress bar for dagster integration #337 👈 (View in Graphite)
MetaxyIOManager #330
MetaxyMetadataStoreResource #329
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2025-11-22T14:48:14Z

Coverage Report (Python 3.10)

File	Stmts	Miss	Cover	Missing
__init__.py	17	3	82%	79, 83–84
_packaging.py	28	8	71%	17–18, 40, 43–44, 46, 48, 96
_utils.py	19	2	89%	26, 32
config.py	193	24	87%	49, 55, 60, 73–74, 222–225, 258, 265–266, 268, 271, 428, 453, 461–462, 464–465, 467, 495, 533, 562
entrypoints.py	66	10	84%	156, 171, 215–216, 218–219, 221–222, 225–226
_testing
metaxy_project.py	249	30	87%	93, 129, 136, 199, 306, 308–314, 353, 418, 439, 474, 488, 520, 533, 586, 719, 732, 734–740, 746
models.py	22	1	95%	40
runbook.py	159	16	89%	89, 114, 151, 243, 245, 247–248, 333, 368, 384, 413, 436–437, 444, 463, 533
_testing/parametric
metadata.py	125	31	75%	121–122, 127, 130, 132–133, 135, 138, 144–146, 148, 152–153, 155–156, 161–162, 165–166, 168, 171, 173, 322, 354, 594–596, 601, 610, 638
cli
context.py	50	27	46%	44–45, 49–50, 52–54, 66–68, 72, 80–82, 86, 99, 111, 113, 121–122, 124, 128, 156–157, 159, 164, 167
ext/dagster
helpers.py	277	139	49%	48–50, 70, 75, 81, 88–103, 114, 118–123, 131, 133–134, 136–137, 148–151, 154, 156, 176, 327, 332, 350–351, 353–358, 361–363, 365, 368–369, 372–375, 378, 555–556, 558–560, 562–564, 566–569, 571–576, 578–582, 584–589, 591–600, 602–610, 612–622, 624–627, 629–630, 632–634, 972, 981, 984, 986–987, 989, 992–993, 996
io_manager.py	177	34	80%	212–213, 292, 347, 376, 382–383, 438, 442, 516, 529, 533, 562, 589–592, 594, 597–599, 601–603, 607, 616, 622, 624, 628, 631, 636, 640–641, 646
resource.py	71	18	74%	55–58, 62–64, 68–69, 73, 77–78, 82, 86–87, 91, 153, 197
ext/sqlalchemy
plugin.py	64	1	98%	148
ext/sqlmodel
plugin.py	60	9	85%	93–95, 148–149, 151–152, 154, 156
graph
describe.py	136	130	4%	46–47, 53, 56, 61–62, 64–65, 67, 69–71, 73–75, 77–80, 82, 85–88, 91, 96–98, 100–111, 114–117, 120, 130–132, 134, 159–161, 164–167, 169, 173, 175, 180–181, 183–184, 186–187, 189, 191–193, 195–197, 199–201, 206, 208, 214–215, 218, 220–225, 227–228, 230, 256–263, 265, 269, 271, 276–277, 279–280, 282–283, 285, 288–294, 299–300, 302–303, 309–310, 313, 315–320, 322–323, 325
utils.py	11	1	90%	17
graph/diff
diff_models.py	143	16	88%	70, 82, 154, 171, 182, 200, 214, 228, 242, 299, 319, 333, 357, 378, 399, 420
differ.py	278	23	91%	42, 74, 127–128, 133, 152, 173–174, 179, 192, 194–196, 198, 454, 573, 607, 705, 709–710, 740–741, 755
models.py	164	86	47%	110, 121, 152, 182, 186, 209, 227, 274, 276–277, 280–282, 285, 288–290, 292–293, 295, 301, 304–306, 309, 312, 320, 323–324, 326, 342, 344–345, 348, 350, 353–361, 363, 366–367, 370–373, 376–377, 379–380, 382, 385–398, 400–402, 405–407, 409, 415, 418, 424, 432, 435–438, 440
traversal.py	96	85	11%	24, 41–42, 44–45, 47–50, 52, 55, 59–61, 63, 66–67, 69, 83–85, 87–89, 91–92, 94–95, 97–98, 100–102, 105–108, 110, 133–135, 138, 141–144, 147–150, 153, 157, 164, 183, 185–187, 189, 191–194, 196–197, 199–200, 215–221, 223, 225–227, 229–231, 233–234, 236–237, 245
graph/diff/rendering
base.py	93	56	39%	99–100, 103–104, 106, 111, 123, 135, 172–173, 176–177, 180–181, 183–185, 196–198, 209–216, 218, 232–234, 250, 261, 269, 272–274, 276–277, 280–281, 285–286, 289–295, 298, 306–307, 309, 315
cards.py	89	82	7%	23–25, 27, 30, 33, 35–36, 38–40, 43–46, 50, 53–61, 66–68, 71, 78–80, 91–92, 94, 97–98, 101, 104–106, 108, 111–112, 114–116, 118–120, 122–124, 127–131, 133, 144, 147–148, 153–155, 157–158, 160–161, 164–166, 168, 179–186, 188
formatter.py	294	51	82%	62, 64, 66, 82–87, 89, 104, 125–127, 130, 134, 137, 139, 144, 149, 155, 160–163, 165–166, 171, 174–175, 180, 183–184, 189, 194, 198, 210, 222–223, 438–439, 444–445, 454, 461, 465, 555–556, 597–598, 730
graphviz.py	114	104	8%	24, 27, 30–32, 35–36, 38–43, 46, 48–51, 54–55, 57, 61, 64–69, 71, 74–76, 78–79, 81–83, 86, 88, 94, 98, 100, 112–113, 116–119, 121, 132–139, 141, 152–159, 161, 172, 175–177, 179–180, 182–184, 186–187, 189–190, 192, 203, 206–208, 210–211, 216–218, 220–221, 223–224, 226, 237–244, 246
mermaid.py	137	123	10%	25, 28, 31, 33–35, 37–38, 41, 44, 46–48, 51–59, 62, 64, 71, 74, 77–79, 82–84, 87, 92–94, 96, 99–100, 102, 113, 125–128, 130, 141, 155, 158, 160, 163–165, 167, 169, 171, 174, 176, 179, 181, 184, 186, 191–192, 194, 205, 208–209, 212–213, 218–220, 223–224, 226–227, 229–231, 233–234, 236–237, 239, 242, 244–247, 250–251, 262, 265–267, 269–270, 272–273, 278–280, 282–283, 285–286, 288, 290, 301–306, 308, 319–324, 326
rich.py	72	65	9%	23–24, 26, 29, 32–34, 38, 41, 43–45, 48–50, 60, 63, 68–69, 72–73, 75, 78, 81–82, 84–85, 88–90, 92–93, 96–99, 102–106, 118, 120, 125–126, 131, 134, 137–138, 140–141, 144–146, 148–149, 160–167, 169
theme.py	16	1	93%	48
metadata_store
_ducklake_support.py	225	34	84%	41, 43, 46, 55, 62, 75, 81, 89, 96–97, 110–111, 129, 132, 140, 150, 178–180, 182, 207, 261, 269, 271, 279, 285, 288, 377, 405–408, 411, 414
base.py	375	41	89%	273, 293, 296, 301, 303–304, 343, 366, 467, 498–499, 544, 546, 686, 703, 708, 917, 1016, 1074, 1081, 1089, 1095, 1195, 1389, 1409–1410, 1414, 1419, 1426, 1431–1432, 1437, 1444, 1446, 1455, 1457, 1460, 1463, 1485–1486, 1489
bigquery.py	66	11	83%	201, 227–228, 266, 271, 276, 281, 286, 292, 297, 302
clickhouse.py	41	9	78%	78, 113, 118, 123, 128, 136, 141, 146, 151
delta.py	108	16	85%	77, 93, 96, 130–131, 190, 234, 258, 261–262, 265, 273, 313, 319–321
duckdb.py	150	29	80%	52–58, 60, 171–172, 176, 244, 254–256, 258, 266–267, 288, 293, 298, 312–313, 324, 329, 387, 394, 400, 405
ibis.py	122	10	91%	158, 177, 328–329, 331, 356, 370, 388, 444, 471
memory.py	87	13	85%	79, 82, 84–85, 88, 122, 124, 132, 134, 158, 161–162, 203
warnings.py	22	1	95%	29
metadata_store/system
models.py	44	3	93%	92–93, 108
storage.py	164	43	73%	143, 189, 379, 381–383, 385–386, 389–390, 392–396, 401, 403, 406, 422, 425, 429–430, 436, 438, 445, 465–466, 468, 471, 473–474, 477, 479–480, 483, 486, 497, 505, 518, 730, 816, 827, 832
migrations
detector.py	57	9	84%	69, 79, 94–95, 100, 102, 117, 123, 129
executor.py	151	25	83%	54, 59–60, 62, 111–112, 117, 178–180, 194–195, 250–251, 260–261, 293–295, 406, 409–410, 413–414, 422
generator.py	124	124	0%	3–4, 6, 8–13, 20, 33, 35–36, 39–41, 44–46, 48, 51, 106, 108, 110, 112–113, 117, 120–122, 124, 128–129, 134, 137, 140–145, 149, 157, 159–160, 166–167, 170–173, 182, 185, 190–191, 194, 197–199, 201–202, 205, 208, 216–218, 221–224, 226, 228–230, 233–235, 238–241, 244–245, 248, 250–251, 255–257, 260–261, 268–269, 271–273, 275–276, 279–280, 282–283, 285–286, 289–290, 292, 297, 299, 302, 304, 310, 312–314, 316, 319–321, 323, 328, 332, 334, 343
loader.py	92	78	15%	26, 28, 30–31, 33–34, 37–40, 42, 58–59, 61–62, 68–73, 75, 78–84, 86, 101–102, 104–105, 107–108, 126, 129, 134–135, 141, 144, 146, 148, 153–154, 161, 178, 180–181, 183–184, 187–190, 192–193, 196–198, 201–202, 204–207, 209–210, 215–218, 221, 224–226, 231
models.py	106	8	92%	129, 238, 288, 321–322, 324–326
ops.py	100	41	59%	54–58, 210–211, 223–224, 236, 246, 252–253, 257–258, 261–262, 268, 270, 273–274, 276, 278–279, 282, 292, 296, 299, 304–306, 309, 312–313, 315, 319–320, 325–327, 329
models
feature.py	395	53	86%	72–73, 127, 132, 136, 138–139, 146, 171, 176, 221, 230, 309, 318, 405, 408–410, 412, 540, 642, 753–754, 757–760, 772, 776, 791, 793–794, 797–801, 804, 806, 829, 884, 991–992, 996, 999, 1025–1027, 1270, 1277–1278, 1280, 1286
feature_spec.py	86	2	97%	113, 138
field.py	53	6	88%	41, 43, 56–57, 59, 163
fields_mapping.py	88	10	88%	55, 100–101, 194, 267–270, 273, 275
filter_expression.py	162	40	75%	45, 75, 79–81, 84, 99, 117, 172, 180–184, 187–189, 210, 220–225, 233, 242, 251–255, 270–274, 282–283, 285, 289
lineage.py	44	1	97%	74
plan.py	98	17	82%	37, 41, 45, 75, 77–78, 80, 84, 86–88, 90, 135, 168, 188, 222–223
types.py	141	14	90%	89–93, 103–104, 106, 131, 167, 181, 187, 193, 199
utils
exceptions.py	6	1	83%	10
hashing.py	55	6	89%	45, 139, 185, 191, 197, 207
versioning
engine.py	183	10	94%	45, 66, 69, 83, 289, 515, 518, 608, 613, 650
ibis.py	58	14	75%	72, 160–161, 164, 167, 170, 173, 178–179, 183–184, 189, 192, 227
polars.py	43	2	95%	58, 177
types.py	22	1	95%	41
TOTAL	7052	1848	73%

Tests	Skipped	Failures	Errors	Time
1119	22 💤	0 ❌	0 🔥	3m 51s ⏱️

github-actions · 2025-11-22T14:48:23Z

Coverage Report (Python 3.13)

File	Stmts	Miss	Cover	Missing
__init__.py	17	3	82%	79, 83–84
_packaging.py	28	8	71%	17–18, 40, 43–44, 46, 48, 96
_utils.py	19	2	89%	26, 32
config.py	193	24	87%	49, 55, 60, 73–74, 222–225, 258, 265–266, 268, 271, 428, 453, 461–462, 464–465, 467, 495, 533, 562
entrypoints.py	66	10	84%	156, 171, 215–216, 218–219, 221–222, 225–226
_testing
metaxy_project.py	249	30	87%	93, 129, 136, 199, 306, 308–314, 353, 418, 439, 474, 488, 520, 533, 586, 719, 732, 734–740, 746
models.py	22	1	95%	40
runbook.py	159	16	89%	89, 114, 151, 243, 245, 247–248, 333, 368, 384, 413, 436–437, 444, 463, 533
_testing/parametric
metadata.py	125	31	75%	121–122, 127, 130, 132–133, 135, 138, 144–146, 148, 152–153, 155–156, 161–162, 165–166, 168, 171, 173, 322, 354, 594–596, 601, 610, 638
cli
context.py	50	27	46%	44–45, 49–50, 52–54, 66–68, 72, 80–82, 86, 99, 111, 113, 121–122, 124, 128, 156–157, 159, 164, 167
ext/dagster
helpers.py	277	139	49%	48–50, 70, 75, 81, 88–103, 114, 118–123, 131, 133–134, 136–137, 148–151, 154, 156, 176, 327, 332, 350–351, 353–358, 361–363, 365, 368–369, 372–375, 378, 555–556, 558–560, 562–564, 566–569, 571–576, 578–582, 584–589, 591–600, 602–610, 612–622, 624–627, 629–630, 632–634, 972, 981, 984, 986–987, 989, 992–993, 996
io_manager.py	177	34	80%	212–213, 292, 347, 376, 382–383, 438, 442, 516, 529, 533, 562, 589–592, 594, 597–599, 601–603, 607, 616, 622, 624, 628, 631, 636, 640–641, 646
resource.py	71	18	74%	55–58, 62–64, 68–69, 73, 77–78, 82, 86–87, 91, 153, 197
ext/sqlalchemy
plugin.py	64	1	98%	148
ext/sqlmodel
plugin.py	60	9	85%	93–95, 148–149, 151–152, 154, 156
graph
describe.py	136	130	4%	46–47, 53, 56, 61–62, 64–65, 67, 69–71, 73–75, 77–80, 82, 85–88, 91, 96–98, 100–111, 114–117, 120, 130–132, 134, 159–161, 164–167, 169, 173, 175, 180–181, 183–184, 186–187, 189, 191–193, 195–197, 199–201, 206, 208, 214–215, 218, 220–225, 227–228, 230, 256–263, 265, 269, 271, 276–277, 279–280, 282–283, 285, 288–294, 299–300, 302–303, 309–310, 313, 315–320, 322–323, 325
utils.py	11	1	90%	17
graph/diff
diff_models.py	143	16	88%	70, 82, 154, 171, 182, 200, 214, 228, 242, 299, 319, 333, 357, 378, 399, 420
differ.py	278	23	91%	42, 74, 127–128, 133, 152, 173–174, 179, 192, 194–196, 198, 454, 573, 607, 705, 709–710, 740–741, 755
models.py	164	86	47%	110, 121, 152, 182, 186, 209, 227, 274, 276–277, 280–282, 285, 288–290, 292–293, 295, 301, 304–306, 309, 312, 320, 323–324, 326, 342, 344–345, 348, 350, 353–361, 363, 366–367, 370–373, 376–377, 379–380, 382, 385–398, 400–402, 405–407, 409, 415, 418, 424, 432, 435–438, 440
traversal.py	96	85	11%	24, 41–42, 44–45, 47–50, 52, 55, 59–61, 63, 66–67, 69, 83–85, 87–89, 91–92, 94–95, 97–98, 100–102, 105–108, 110, 133–135, 138, 141–144, 147–150, 153, 157, 164, 183, 185–187, 189, 191–194, 196–197, 199–200, 215–221, 223, 225–227, 229–231, 233–234, 236–237, 245
graph/diff/rendering
base.py	93	56	39%	99–100, 103–104, 106, 111, 123, 135, 172–173, 176–177, 180–181, 183–185, 196–198, 209–216, 218, 232–234, 250, 261, 269, 272–274, 276–277, 280–281, 285–286, 289–295, 298, 306–307, 309, 315
cards.py	89	82	7%	23–25, 27, 30, 33, 35–36, 38–40, 43–46, 50, 53–61, 66–68, 71, 78–80, 91–92, 94, 97–98, 101, 104–106, 108, 111–112, 114–116, 118–120, 122–124, 127–131, 133, 144, 147–148, 153–155, 157–158, 160–161, 164–166, 168, 179–186, 188
formatter.py	294	51	82%	62, 64, 66, 82–87, 89, 104, 125–127, 130, 134, 137, 139, 144, 149, 155, 160–163, 165–166, 171, 174–175, 180, 183–184, 189, 194, 198, 210, 222–223, 438–439, 444–445, 454, 461, 465, 555–556, 597–598, 730
graphviz.py	114	104	8%	24, 27, 30–32, 35–36, 38–43, 46, 48–51, 54–55, 57, 61, 64–69, 71, 74–76, 78–79, 81–83, 86, 88, 94, 98, 100, 112–113, 116–119, 121, 132–139, 141, 152–159, 161, 172, 175–177, 179–180, 182–184, 186–187, 189–190, 192, 203, 206–208, 210–211, 216–218, 220–221, 223–224, 226, 237–244, 246
mermaid.py	137	123	10%	25, 28, 31, 33–35, 37–38, 41, 44, 46–48, 51–59, 62, 64, 71, 74, 77–79, 82–84, 87, 92–94, 96, 99–100, 102, 113, 125–128, 130, 141, 155, 158, 160, 163–165, 167, 169, 171, 174, 176, 179, 181, 184, 186, 191–192, 194, 205, 208–209, 212–213, 218–220, 223–224, 226–227, 229–231, 233–234, 236–237, 239, 242, 244–247, 250–251, 262, 265–267, 269–270, 272–273, 278–280, 282–283, 285–286, 288, 290, 301–306, 308, 319–324, 326
rich.py	72	65	9%	23–24, 26, 29, 32–34, 38, 41, 43–45, 48–50, 60, 63, 68–69, 72–73, 75, 78, 81–82, 84–85, 88–90, 92–93, 96–99, 102–106, 118, 120, 125–126, 131, 134, 137–138, 140–141, 144–146, 148–149, 160–167, 169
theme.py	16	1	93%	48
metadata_store
_ducklake_support.py	225	34	84%	41, 43, 46, 55, 62, 75, 81, 89, 96–97, 110–111, 129, 132, 140, 150, 178–180, 182, 207, 261, 269, 271, 279, 285, 288, 377, 405–408, 411, 414
base.py	375	41	89%	273, 293, 296, 301, 303–304, 343, 366, 467, 498–499, 544, 546, 686, 703, 708, 917, 1016, 1074, 1081, 1089, 1095, 1195, 1389, 1409–1410, 1414, 1419, 1426, 1431–1432, 1437, 1444, 1446, 1455, 1457, 1460, 1463, 1485–1486, 1489
bigquery.py	66	11	83%	201, 227–228, 266, 271, 276, 281, 286, 292, 297, 302
clickhouse.py	41	9	78%	78, 113, 118, 123, 128, 136, 141, 146, 151
delta.py	108	16	85%	77, 93, 96, 130–131, 190, 234, 258, 261–262, 265, 273, 313, 319–321
duckdb.py	150	29	80%	52–58, 60, 171–172, 176, 244, 254–256, 258, 266–267, 288, 293, 298, 312–313, 324, 329, 387, 394, 400, 405
ibis.py	122	10	91%	158, 177, 328–329, 331, 356, 370, 388, 444, 471
memory.py	87	13	85%	79, 82, 84–85, 88, 122, 124, 132, 134, 158, 161–162, 203
warnings.py	22	1	95%	29
metadata_store/system
models.py	44	3	93%	92–93, 108
storage.py	164	43	73%	143, 189, 379, 381–383, 385–386, 389–390, 392–396, 401, 403, 406, 422, 425, 429–430, 436, 438, 445, 465–466, 468, 471, 473–474, 477, 479–480, 483, 486, 497, 505, 518, 730, 816, 827, 832
migrations
detector.py	57	9	84%	69, 79, 94–95, 100, 102, 117, 123, 129
executor.py	151	25	83%	54, 59–60, 62, 111–112, 117, 178–180, 194–195, 250–251, 260–261, 293–295, 406, 409–410, 413–414, 422
generator.py	124	124	0%	3–4, 6, 8–13, 20, 33, 35–36, 39–41, 44–46, 48, 51, 106, 108, 110, 112–113, 117, 120–122, 124, 128–129, 134, 137, 140–145, 149, 157, 159–160, 166–167, 170–173, 182, 185, 190–191, 194, 197–199, 201–202, 205, 208, 216–218, 221–224, 226, 228–230, 233–235, 238–241, 244–245, 248, 250–251, 255–257, 260–261, 268–269, 271–273, 275–276, 279–280, 282–283, 285–286, 289–290, 292, 297, 299, 302, 304, 310, 312–314, 316, 319–321, 323, 328, 332, 334, 343
loader.py	92	78	15%	26, 28, 30–31, 33–34, 37–40, 42, 58–59, 61–62, 68–73, 75, 78–84, 86, 101–102, 104–105, 107–108, 126, 129, 134–135, 141, 144, 146, 148, 153–154, 161, 178, 180–181, 183–184, 187–190, 192–193, 196–198, 201–202, 204–207, 209–210, 215–218, 221, 224–226, 231
models.py	106	8	92%	129, 238, 288, 321–322, 324–326
ops.py	100	41	59%	54–58, 210–211, 223–224, 236, 246, 252–253, 257–258, 261–262, 268, 270, 273–274, 276, 278–279, 282, 292, 296, 299, 304–306, 309, 312–313, 315, 319–320, 325–327, 329
models
feature.py	395	53	86%	72–73, 127, 132, 136, 138–139, 146, 171, 176, 221, 230, 309, 318, 405, 408–410, 412, 540, 642, 753–754, 757–760, 772, 776, 791, 793–794, 797–801, 804, 806, 829, 884, 991–992, 996, 999, 1025–1027, 1270, 1277–1278, 1280, 1286
feature_spec.py	86	2	97%	113, 138
field.py	53	6	88%	41, 43, 56–57, 59, 163
fields_mapping.py	88	10	88%	55, 100–101, 194, 267–270, 273, 275
filter_expression.py	162	40	75%	45, 75, 79–81, 84, 99, 117, 172, 180–184, 187–189, 210, 220–225, 233, 242, 251–255, 270–274, 282–283, 285, 289
lineage.py	44	1	97%	74
plan.py	98	17	82%	37, 41, 45, 75, 77–78, 80, 84, 86–88, 90, 135, 168, 188, 222–223
types.py	141	14	90%	89–93, 103–104, 106, 131, 167, 181, 187, 193, 199
utils
exceptions.py	6	1	83%	10
hashing.py	55	6	89%	45, 139, 185, 191, 197, 207
versioning
engine.py	183	10	94%	45, 66, 69, 83, 289, 515, 518, 608, 613, 650
ibis.py	58	14	75%	72, 160–161, 164, 167, 170, 173, 178–179, 183–184, 189, 192, 227
polars.py	43	2	95%	58, 177
types.py	22	1	95%	41
TOTAL	7052	1848	73%

Tests	Skipped	Failures	Errors	Time
1119	22 💤	0 ❌	0 🔥	4m 14s ⏱️

github-actions · 2025-11-22T14:48:31Z

Coverage Report (Python 3.12)

File	Stmts	Miss	Cover	Missing
__init__.py	17	3	82%	79, 83–84
_packaging.py	28	8	71%	17–18, 40, 43–44, 46, 48, 96
_utils.py	19	2	89%	26, 32
config.py	193	24	87%	49, 55, 60, 73–74, 222–225, 258, 265–266, 268, 271, 428, 453, 461–462, 464–465, 467, 495, 533, 562
entrypoints.py	66	10	84%	156, 171, 215–216, 218–219, 221–222, 225–226
_testing
metaxy_project.py	249	30	87%	93, 129, 136, 199, 306, 308–314, 353, 418, 439, 474, 488, 520, 533, 586, 719, 732, 734–740, 746
models.py	22	1	95%	40
runbook.py	159	16	89%	89, 114, 151, 243, 245, 247–248, 333, 368, 384, 413, 436–437, 444, 463, 533
_testing/parametric
metadata.py	125	31	75%	121–122, 127, 130, 132–133, 135, 138, 144–146, 148, 152–153, 155–156, 161–162, 165–166, 168, 171, 173, 322, 354, 594–596, 601, 610, 638
cli
context.py	50	27	46%	44–45, 49–50, 52–54, 66–68, 72, 80–82, 86, 99, 111, 113, 121–122, 124, 128, 156–157, 159, 164, 167
ext/dagster
helpers.py	277	139	49%	48–50, 70, 75, 81, 88–103, 114, 118–123, 131, 133–134, 136–137, 148–151, 154, 156, 176, 327, 332, 350–351, 353–358, 361–363, 365, 368–369, 372–375, 378, 555–556, 558–560, 562–564, 566–569, 571–576, 578–582, 584–589, 591–600, 602–610, 612–622, 624–627, 629–630, 632–634, 972, 981, 984, 986–987, 989, 992–993, 996
io_manager.py	177	34	80%	212–213, 292, 347, 376, 382–383, 438, 442, 516, 529, 533, 562, 589–592, 594, 597–599, 601–603, 607, 616, 622, 624, 628, 631, 636, 640–641, 646
resource.py	71	18	74%	55–58, 62–64, 68–69, 73, 77–78, 82, 86–87, 91, 153, 197
ext/sqlalchemy
plugin.py	64	1	98%	148
ext/sqlmodel
plugin.py	60	9	85%	93–95, 148–149, 151–152, 154, 156
graph
describe.py	136	130	4%	46–47, 53, 56, 61–62, 64–65, 67, 69–71, 73–75, 77–80, 82, 85–88, 91, 96–98, 100–111, 114–117, 120, 130–132, 134, 159–161, 164–167, 169, 173, 175, 180–181, 183–184, 186–187, 189, 191–193, 195–197, 199–201, 206, 208, 214–215, 218, 220–225, 227–228, 230, 256–263, 265, 269, 271, 276–277, 279–280, 282–283, 285, 288–294, 299–300, 302–303, 309–310, 313, 315–320, 322–323, 325
utils.py	11	1	90%	17
graph/diff
diff_models.py	143	16	88%	70, 82, 154, 171, 182, 200, 214, 228, 242, 299, 319, 333, 357, 378, 399, 420
differ.py	278	23	91%	42, 74, 127–128, 133, 152, 173–174, 179, 192, 194–196, 198, 454, 573, 607, 705, 709–710, 740–741, 755
models.py	164	86	47%	110, 121, 152, 182, 186, 209, 227, 274, 276–277, 280–282, 285, 288–290, 292–293, 295, 301, 304–306, 309, 312, 320, 323–324, 326, 342, 344–345, 348, 350, 353–361, 363, 366–367, 370–373, 376–377, 379–380, 382, 385–398, 400–402, 405–407, 409, 415, 418, 424, 432, 435–438, 440
traversal.py	96	85	11%	24, 41–42, 44–45, 47–50, 52, 55, 59–61, 63, 66–67, 69, 83–85, 87–89, 91–92, 94–95, 97–98, 100–102, 105–108, 110, 133–135, 138, 141–144, 147–150, 153, 157, 164, 183, 185–187, 189, 191–194, 196–197, 199–200, 215–221, 223, 225–227, 229–231, 233–234, 236–237, 245
graph/diff/rendering
base.py	93	56	39%	99–100, 103–104, 106, 111, 123, 135, 172–173, 176–177, 180–181, 183–185, 196–198, 209–216, 218, 232–234, 250, 261, 269, 272–274, 276–277, 280–281, 285–286, 289–295, 298, 306–307, 309, 315
cards.py	89	82	7%	23–25, 27, 30, 33, 35–36, 38–40, 43–46, 50, 53–61, 66–68, 71, 78–80, 91–92, 94, 97–98, 101, 104–106, 108, 111–112, 114–116, 118–120, 122–124, 127–131, 133, 144, 147–148, 153–155, 157–158, 160–161, 164–166, 168, 179–186, 188
formatter.py	294	51	82%	62, 64, 66, 82–87, 89, 104, 125–127, 130, 134, 137, 139, 144, 149, 155, 160–163, 165–166, 171, 174–175, 180, 183–184, 189, 194, 198, 210, 222–223, 438–439, 444–445, 454, 461, 465, 555–556, 597–598, 730
graphviz.py	114	104	8%	24, 27, 30–32, 35–36, 38–43, 46, 48–51, 54–55, 57, 61, 64–69, 71, 74–76, 78–79, 81–83, 86, 88, 94, 98, 100, 112–113, 116–119, 121, 132–139, 141, 152–159, 161, 172, 175–177, 179–180, 182–184, 186–187, 189–190, 192, 203, 206–208, 210–211, 216–218, 220–221, 223–224, 226, 237–244, 246
mermaid.py	137	123	10%	25, 28, 31, 33–35, 37–38, 41, 44, 46–48, 51–59, 62, 64, 71, 74, 77–79, 82–84, 87, 92–94, 96, 99–100, 102, 113, 125–128, 130, 141, 155, 158, 160, 163–165, 167, 169, 171, 174, 176, 179, 181, 184, 186, 191–192, 194, 205, 208–209, 212–213, 218–220, 223–224, 226–227, 229–231, 233–234, 236–237, 239, 242, 244–247, 250–251, 262, 265–267, 269–270, 272–273, 278–280, 282–283, 285–286, 288, 290, 301–306, 308, 319–324, 326
rich.py	72	65	9%	23–24, 26, 29, 32–34, 38, 41, 43–45, 48–50, 60, 63, 68–69, 72–73, 75, 78, 81–82, 84–85, 88–90, 92–93, 96–99, 102–106, 118, 120, 125–126, 131, 134, 137–138, 140–141, 144–146, 148–149, 160–167, 169
theme.py	16	1	93%	48
metadata_store
_ducklake_support.py	225	34	84%	41, 43, 46, 55, 62, 75, 81, 89, 96–97, 110–111, 129, 132, 140, 150, 178–180, 182, 207, 261, 269, 271, 279, 285, 288, 377, 405–408, 411, 414
base.py	375	41	89%	273, 293, 296, 301, 303–304, 343, 366, 467, 498–499, 544, 546, 686, 703, 708, 917, 1016, 1074, 1081, 1089, 1095, 1195, 1389, 1409–1410, 1414, 1419, 1426, 1431–1432, 1437, 1444, 1446, 1455, 1457, 1460, 1463, 1485–1486, 1489
bigquery.py	66	11	83%	201, 227–228, 266, 271, 276, 281, 286, 292, 297, 302
clickhouse.py	41	9	78%	78, 113, 118, 123, 128, 136, 141, 146, 151
delta.py	108	16	85%	77, 93, 96, 130–131, 190, 234, 258, 261–262, 265, 273, 313, 319–321
duckdb.py	150	29	80%	52–58, 60, 171–172, 176, 244, 254–256, 258, 266–267, 288, 293, 298, 312–313, 324, 329, 387, 394, 400, 405
ibis.py	122	10	91%	158, 177, 328–329, 331, 356, 370, 388, 444, 471
memory.py	87	13	85%	79, 82, 84–85, 88, 122, 124, 132, 134, 158, 161–162, 203
warnings.py	22	1	95%	29
metadata_store/system
models.py	44	3	93%	92–93, 108
storage.py	164	43	73%	143, 189, 379, 381–383, 385–386, 389–390, 392–396, 401, 403, 406, 422, 425, 429–430, 436, 438, 445, 465–466, 468, 471, 473–474, 477, 479–480, 483, 486, 497, 505, 518, 730, 816, 827, 832
migrations
detector.py	57	9	84%	69, 79, 94–95, 100, 102, 117, 123, 129
executor.py	151	25	83%	54, 59–60, 62, 111–112, 117, 178–180, 194–195, 250–251, 260–261, 293–295, 406, 409–410, 413–414, 422
generator.py	124	124	0%	3–4, 6, 8–13, 20, 33, 35–36, 39–41, 44–46, 48, 51, 106, 108, 110, 112–113, 117, 120–122, 124, 128–129, 134, 137, 140–145, 149, 157, 159–160, 166–167, 170–173, 182, 185, 190–191, 194, 197–199, 201–202, 205, 208, 216–218, 221–224, 226, 228–230, 233–235, 238–241, 244–245, 248, 250–251, 255–257, 260–261, 268–269, 271–273, 275–276, 279–280, 282–283, 285–286, 289–290, 292, 297, 299, 302, 304, 310, 312–314, 316, 319–321, 323, 328, 332, 334, 343
loader.py	92	78	15%	26, 28, 30–31, 33–34, 37–40, 42, 58–59, 61–62, 68–73, 75, 78–84, 86, 101–102, 104–105, 107–108, 126, 129, 134–135, 141, 144, 146, 148, 153–154, 161, 178, 180–181, 183–184, 187–190, 192–193, 196–198, 201–202, 204–207, 209–210, 215–218, 221, 224–226, 231
models.py	106	8	92%	129, 238, 288, 321–322, 324–326
ops.py	100	41	59%	54–58, 210–211, 223–224, 236, 246, 252–253, 257–258, 261–262, 268, 270, 273–274, 276, 278–279, 282, 292, 296, 299, 304–306, 309, 312–313, 315, 319–320, 325–327, 329
models
feature.py	395	53	86%	72–73, 127, 132, 136, 138–139, 146, 171, 176, 221, 230, 309, 318, 405, 408–410, 412, 540, 642, 753–754, 757–760, 772, 776, 791, 793–794, 797–801, 804, 806, 829, 884, 991–992, 996, 999, 1025–1027, 1270, 1277–1278, 1280, 1286
feature_spec.py	86	2	97%	113, 138
field.py	53	6	88%	41, 43, 56–57, 59, 163
fields_mapping.py	88	10	88%	55, 100–101, 194, 267–270, 273, 275
filter_expression.py	162	40	75%	45, 75, 79–81, 84, 99, 117, 172, 180–184, 187–189, 210, 220–225, 233, 242, 251–255, 270–274, 282–283, 285, 289
lineage.py	44	1	97%	74
plan.py	98	17	82%	37, 41, 45, 75, 77–78, 80, 84, 86–88, 90, 135, 168, 188, 222–223
types.py	141	14	90%	89–93, 103–104, 106, 131, 167, 181, 187, 193, 199
utils
exceptions.py	6	1	83%	10
hashing.py	55	6	89%	45, 139, 185, 191, 197, 207
versioning
engine.py	183	10	94%	45, 66, 69, 83, 289, 515, 518, 608, 613, 650
ibis.py	58	14	75%	72, 160–161, 164, 167, 170, 173, 178–179, 183–184, 189, 192, 227
polars.py	43	2	95%	58, 177
types.py	22	1	95%	41
TOTAL	7052	1848	73%

Tests	Skipped	Failures	Errors	Time
1119	22 💤	0 ❌	0 🔥	4m 3s ⏱️

github-actions · 2025-11-22T14:49:18Z

Coverage Report (Python 3.11)

File	Stmts	Miss	Cover	Missing
__init__.py	17	3	82%	79, 83–84
_packaging.py	28	8	71%	17–18, 40, 43–44, 46, 48, 96
_utils.py	19	2	89%	26, 32
config.py	193	24	87%	49, 55, 60, 73–74, 222–225, 258, 265–266, 268, 271, 428, 453, 461–462, 464–465, 467, 495, 533, 562
entrypoints.py	66	10	84%	156, 171, 215–216, 218–219, 221–222, 225–226
_testing
metaxy_project.py	249	30	87%	93, 129, 136, 199, 306, 308–314, 353, 418, 439, 474, 488, 520, 533, 586, 719, 732, 734–740, 746
models.py	22	1	95%	40
runbook.py	159	16	89%	89, 114, 151, 243, 245, 247–248, 333, 368, 384, 413, 436–437, 444, 463, 533
_testing/parametric
metadata.py	125	31	75%	121–122, 127, 130, 132–133, 135, 138, 144–146, 148, 152–153, 155–156, 161–162, 165–166, 168, 171, 173, 322, 354, 594–596, 601, 610, 638
cli
context.py	50	27	46%	44–45, 49–50, 52–54, 66–68, 72, 80–82, 86, 99, 111, 113, 121–122, 124, 128, 156–157, 159, 164, 167
ext/dagster
helpers.py	277	139	49%	48–50, 70, 75, 81, 88–103, 114, 118–123, 131, 133–134, 136–137, 148–151, 154, 156, 176, 327, 332, 350–351, 353–358, 361–363, 365, 368–369, 372–375, 378, 555–556, 558–560, 562–564, 566–569, 571–576, 578–582, 584–589, 591–600, 602–610, 612–622, 624–627, 629–630, 632–634, 972, 981, 984, 986–987, 989, 992–993, 996
io_manager.py	177	34	80%	212–213, 292, 347, 376, 382–383, 438, 442, 516, 529, 533, 562, 589–592, 594, 597–599, 601–603, 607, 616, 622, 624, 628, 631, 636, 640–641, 646
resource.py	71	18	74%	55–58, 62–64, 68–69, 73, 77–78, 82, 86–87, 91, 153, 197
ext/sqlalchemy
plugin.py	64	1	98%	148
ext/sqlmodel
plugin.py	60	9	85%	93–95, 148–149, 151–152, 154, 156
graph
describe.py	136	130	4%	46–47, 53, 56, 61–62, 64–65, 67, 69–71, 73–75, 77–80, 82, 85–88, 91, 96–98, 100–111, 114–117, 120, 130–132, 134, 159–161, 164–167, 169, 173, 175, 180–181, 183–184, 186–187, 189, 191–193, 195–197, 199–201, 206, 208, 214–215, 218, 220–225, 227–228, 230, 256–263, 265, 269, 271, 276–277, 279–280, 282–283, 285, 288–294, 299–300, 302–303, 309–310, 313, 315–320, 322–323, 325
utils.py	11	1	90%	17
graph/diff
diff_models.py	143	16	88%	70, 82, 154, 171, 182, 200, 214, 228, 242, 299, 319, 333, 357, 378, 399, 420
differ.py	278	23	91%	42, 74, 127–128, 133, 152, 173–174, 179, 192, 194–196, 198, 454, 573, 607, 705, 709–710, 740–741, 755
models.py	164	86	47%	110, 121, 152, 182, 186, 209, 227, 274, 276–277, 280–282, 285, 288–290, 292–293, 295, 301, 304–306, 309, 312, 320, 323–324, 326, 342, 344–345, 348, 350, 353–361, 363, 366–367, 370–373, 376–377, 379–380, 382, 385–398, 400–402, 405–407, 409, 415, 418, 424, 432, 435–438, 440
traversal.py	96	85	11%	24, 41–42, 44–45, 47–50, 52, 55, 59–61, 63, 66–67, 69, 83–85, 87–89, 91–92, 94–95, 97–98, 100–102, 105–108, 110, 133–135, 138, 141–144, 147–150, 153, 157, 164, 183, 185–187, 189, 191–194, 196–197, 199–200, 215–221, 223, 225–227, 229–231, 233–234, 236–237, 245
graph/diff/rendering
base.py	93	56	39%	99–100, 103–104, 106, 111, 123, 135, 172–173, 176–177, 180–181, 183–185, 196–198, 209–216, 218, 232–234, 250, 261, 269, 272–274, 276–277, 280–281, 285–286, 289–295, 298, 306–307, 309, 315
cards.py	89	82	7%	23–25, 27, 30, 33, 35–36, 38–40, 43–46, 50, 53–61, 66–68, 71, 78–80, 91–92, 94, 97–98, 101, 104–106, 108, 111–112, 114–116, 118–120, 122–124, 127–131, 133, 144, 147–148, 153–155, 157–158, 160–161, 164–166, 168, 179–186, 188
formatter.py	294	51	82%	62, 64, 66, 82–87, 89, 104, 125–127, 130, 134, 137, 139, 144, 149, 155, 160–163, 165–166, 171, 174–175, 180, 183–184, 189, 194, 198, 210, 222–223, 438–439, 444–445, 454, 461, 465, 555–556, 597–598, 730
graphviz.py	114	104	8%	24, 27, 30–32, 35–36, 38–43, 46, 48–51, 54–55, 57, 61, 64–69, 71, 74–76, 78–79, 81–83, 86, 88, 94, 98, 100, 112–113, 116–119, 121, 132–139, 141, 152–159, 161, 172, 175–177, 179–180, 182–184, 186–187, 189–190, 192, 203, 206–208, 210–211, 216–218, 220–221, 223–224, 226, 237–244, 246
mermaid.py	137	123	10%	25, 28, 31, 33–35, 37–38, 41, 44, 46–48, 51–59, 62, 64, 71, 74, 77–79, 82–84, 87, 92–94, 96, 99–100, 102, 113, 125–128, 130, 141, 155, 158, 160, 163–165, 167, 169, 171, 174, 176, 179, 181, 184, 186, 191–192, 194, 205, 208–209, 212–213, 218–220, 223–224, 226–227, 229–231, 233–234, 236–237, 239, 242, 244–247, 250–251, 262, 265–267, 269–270, 272–273, 278–280, 282–283, 285–286, 288, 290, 301–306, 308, 319–324, 326
rich.py	72	65	9%	23–24, 26, 29, 32–34, 38, 41, 43–45, 48–50, 60, 63, 68–69, 72–73, 75, 78, 81–82, 84–85, 88–90, 92–93, 96–99, 102–106, 118, 120, 125–126, 131, 134, 137–138, 140–141, 144–146, 148–149, 160–167, 169
theme.py	16	1	93%	48
metadata_store
_ducklake_support.py	225	34	84%	41, 43, 46, 55, 62, 75, 81, 89, 96–97, 110–111, 129, 132, 140, 150, 178–180, 182, 207, 261, 269, 271, 279, 285, 288, 377, 405–408, 411, 414
base.py	375	41	89%	273, 293, 296, 301, 303–304, 343, 366, 467, 498–499, 544, 546, 686, 703, 708, 917, 1016, 1074, 1081, 1089, 1095, 1195, 1389, 1409–1410, 1414, 1419, 1426, 1431–1432, 1437, 1444, 1446, 1455, 1457, 1460, 1463, 1485–1486, 1489
bigquery.py	66	11	83%	201, 227–228, 266, 271, 276, 281, 286, 292, 297, 302
clickhouse.py	41	9	78%	78, 113, 118, 123, 128, 136, 141, 146, 151
delta.py	108	16	85%	77, 93, 96, 130–131, 190, 234, 258, 261–262, 265, 273, 313, 319–321
duckdb.py	150	29	80%	52–58, 60, 171–172, 176, 244, 254–256, 258, 266–267, 288, 293, 298, 312–313, 324, 329, 387, 394, 400, 405
ibis.py	122	10	91%	158, 177, 328–329, 331, 356, 370, 388, 444, 471
memory.py	87	13	85%	79, 82, 84–85, 88, 122, 124, 132, 134, 158, 161–162, 203
warnings.py	22	1	95%	29
metadata_store/system
models.py	44	3	93%	92–93, 108
storage.py	164	43	73%	143, 189, 379, 381–383, 385–386, 389–390, 392–396, 401, 403, 406, 422, 425, 429–430, 436, 438, 445, 465–466, 468, 471, 473–474, 477, 479–480, 483, 486, 497, 505, 518, 730, 816, 827, 832
migrations
detector.py	57	9	84%	69, 79, 94–95, 100, 102, 117, 123, 129
executor.py	151	25	83%	54, 59–60, 62, 111–112, 117, 178–180, 194–195, 250–251, 260–261, 293–295, 406, 409–410, 413–414, 422
generator.py	124	124	0%	3–4, 6, 8–13, 20, 33, 35–36, 39–41, 44–46, 48, 51, 106, 108, 110, 112–113, 117, 120–122, 124, 128–129, 134, 137, 140–145, 149, 157, 159–160, 166–167, 170–173, 182, 185, 190–191, 194, 197–199, 201–202, 205, 208, 216–218, 221–224, 226, 228–230, 233–235, 238–241, 244–245, 248, 250–251, 255–257, 260–261, 268–269, 271–273, 275–276, 279–280, 282–283, 285–286, 289–290, 292, 297, 299, 302, 304, 310, 312–314, 316, 319–321, 323, 328, 332, 334, 343
loader.py	92	78	15%	26, 28, 30–31, 33–34, 37–40, 42, 58–59, 61–62, 68–73, 75, 78–84, 86, 101–102, 104–105, 107–108, 126, 129, 134–135, 141, 144, 146, 148, 153–154, 161, 178, 180–181, 183–184, 187–190, 192–193, 196–198, 201–202, 204–207, 209–210, 215–218, 221, 224–226, 231
models.py	106	8	92%	129, 238, 288, 321–322, 324–326
ops.py	100	41	59%	54–58, 210–211, 223–224, 236, 246, 252–253, 257–258, 261–262, 268, 270, 273–274, 276, 278–279, 282, 292, 296, 299, 304–306, 309, 312–313, 315, 319–320, 325–327, 329
models
feature.py	395	53	86%	72–73, 127, 132, 136, 138–139, 146, 171, 176, 221, 230, 309, 318, 405, 408–410, 412, 540, 642, 753–754, 757–760, 772, 776, 791, 793–794, 797–801, 804, 806, 829, 884, 991–992, 996, 999, 1025–1027, 1270, 1277–1278, 1280, 1286
feature_spec.py	86	2	97%	113, 138
field.py	53	6	88%	41, 43, 56–57, 59, 163
fields_mapping.py	88	10	88%	55, 100–101, 194, 267–270, 273, 275
filter_expression.py	162	40	75%	45, 75, 79–81, 84, 99, 117, 172, 180–184, 187–189, 210, 220–225, 233, 242, 251–255, 270–274, 282–283, 285, 289
lineage.py	44	1	97%	74
plan.py	98	17	82%	37, 41, 45, 75, 77–78, 80, 84, 86–88, 90, 135, 168, 188, 222–223
types.py	141	14	90%	89–93, 103–104, 106, 131, 167, 181, 187, 193, 199
utils
exceptions.py	6	1	83%	10
hashing.py	55	6	89%	45, 139, 185, 191, 197, 207
versioning
engine.py	183	10	94%	45, 66, 69, 83, 289, 515, 518, 608, 613, 650
ibis.py	58	14	75%	72, 160–161, 164, 167, 170, 173, 178–179, 183–184, 189, 192, 227
polars.py	43	2	95%	58, 177
types.py	22	1	95%	41
TOTAL	7052	1848	73%

Tests	Skipped	Failures	Errors	Time
1119	22 💤	0 ❌	0 🔥	3m 46s ⏱️

Copilot

Pull request overview

This PR adds a tqdm-style progress bar helper function for the Dagster integration, enabling progress tracking during dataframe processing in Metaxy assets.

Key Changes:

Added iter_dataframe_with_progress() function that yields dataframe chunks while displaying progress via tqdm (if available) and logging progress messages
Reorganized example files, replacing minimal/subsampled/branch_subset examples with non_partitioned/partitioned/branch_subsampled examples that demonstrate the new progress functionality
Updated exports in __init__.py and __all__ in helpers.py to include the new function

Reviewed changes

Copilot reviewed 7 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/metaxy/ext/dagster/helpers.py	Added `iter_dataframe_with_progress` function for chunked dataframe iteration with tqdm-style progress bars and logging
src/metaxy/ext/dagster/init.py	Exported the new `iter_dataframe_with_progress` function
examples/example-integration-dagster/src/example_integration_dagster/partitioned.py	New example demonstrating partitioned processing with progress tracking
examples/example-integration-dagster/src/example_integration_dagster/non_partitioned.py	New example showing non-partitioned data processing with progress logging
examples/example-integration-dagster/src/example_integration_dagster/branch_subsampled.py	New example combining branch stores and subsampling with progress tracking
examples/example-integration-dagster/src/example_integration_dagster/subsampled.py	Removed older example file
examples/example-integration-dagster/src/example_integration_dagster/minimal.py	Removed older example file
examples/example-integration-dagster/src/example_integration_dagster/branch_subset.py	Removed older example file
examples/example-integration-dagster/README.md	Updated documentation to reflect new example structure

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-23T20:17:15Z

src/metaxy/ext/dagster/helpers.py

+        if hasattr(log_fn, "__self__"):
+            logger_obj = getattr(log_fn, "__self__", None)
+            target = getattr(logger_obj, log_level, None)
+            if callable(target):
+                target(message)
+            else:
+                log_fn(message)  # Fallback to callable
+        elif callable(log_fn):
+            log_fn(message)


The logic for handling logger objects is incorrect. When log_fn=context.log is passed (as shown in the examples), the code checks hasattr(log_fn, "__self__") which would be False for a logger object, causing it to fall through to the elif callable(log_fn) branch at line 585. This means the log_level parameter is effectively ignored because the code never reaches lines 579-582 where it would be used. The logger detection logic needs to be fixed to properly detect logger objects and respect the log_level parameter.

Copilot · 2025-11-23T20:17:15Z

src/metaxy/ext/dagster/helpers.py

+    elapsed = time.monotonic() - start_time
+    failed_total = failed_count() if callable(failed_count) else failed_count or 0
+    _emit(
+        f"{desc} completed: processed={processed}, failed={failed_total}, "
+        f"elapsed={_format_duration(elapsed)}"
+    )


The final summary is emitted even when the function returns early due to total == 0 or chunk_size <= 0 at line 560. When the early return happens, the code at lines 632-637 is never executed, but there's no log message indicating why processing was skipped. This could be confusing for users. Consider adding a log message before the early return to explain why no chunks were processed.

Copilot · 2025-11-23T20:17:16Z

src/metaxy/ext/dagster/helpers.py

+    log_level: str = "info",
+    failed_count: int | Callable[[], int] | None = None,
+    show_eta: bool = True,
+    echo_to_stderr: bool = False,


The echo_to_stderr parameter is not documented in the docstring. This parameter controls whether messages are echoed to stderr in addition to the log_fn, but it's missing from the Args section.

Copilot · 2025-11-23T20:17:16Z

src/metaxy/ext/dagster/helpers.py

+        if show_eta:
+            elapsed = time.monotonic() - start_time
+            rate = processed / elapsed if elapsed > 0 else 0
+            remaining = total - processed
+            eta = remaining / rate if rate > 0 else None
+            eta_str = _format_duration(eta) if eta is not None else "n/a"
+            message = f"{desc}: {processed}/{total} rows (ETA {eta_str})"
+        else:
+            message = f"{desc}: {processed}/{total} rows"
+        _emit(message)


Progress messages are emitted for every chunk iteration, which could create excessive logging. For large dataframes with small chunk sizes, this could generate thousands of log messages. Consider adding a parameter to control logging frequency (e.g., emit logs only every N chunks or every X seconds).

Copilot · 2025-11-23T20:17:16Z

examples/example-integration-dagster/README.md

+### Progress/Logging Example
+
+Process in chunks with a tqdm-style progress bar (in TTY) and log updates:
+
+```bash
+dagster dev -m example_integration_dagster.progress_logging
+```


The README references a progress_logging example module that doesn't exist. Based on the files in the directory, there is no progress_logging.py file. The progress bar functionality is demonstrated in the existing examples (non_partitioned.py, partitioned.py, and branch_subsampled.py), so this section should either be removed or updated to reference one of those files.

Suggested change

### Progress/Logging Example

Process in chunks with a tqdm-style progress bar (in TTY) and log updates:

```bash

dagster dev -m example_integration_dagster.progress_logging

```

### Progress/Logging

Progress bar functionality (tqdm-style) and logging are demonstrated in the existing examples:

- `non_partitioned.py`

- `partitioned.py`

- `branch_subsampled.py`

Run any of these examples to see chunked processing with progress bars and log updates.

Copilot · 2025-11-23T20:17:16Z

src/metaxy/ext/dagster/helpers.py

+    import time
+
+    total = len(df)
+    if total == 0 or chunk_size <= 0:


When chunk_size <= 0, the function returns early without yielding any chunks. However, this could lead to unexpected behavior if the caller expects to iterate over at least the data in some form. Consider raising a ValueError instead to make the invalid input explicit, or default to processing the entire dataframe as a single chunk.

Suggested change

if total == 0 or chunk_size <= 0:

if chunk_size <= 0:

raise ValueError(f"chunk_size must be a positive integer, got {chunk_size}")

if total == 0:

danielgafni

I'm sorry, never mind, I got the title wrong. I'll take a look at this later again.

geoHeil mentioned this pull request Nov 22, 2025

MetaxyMetadataStoreResource #329

Closed

geoHeil mentioned this pull request Nov 22, 2025

MetaxyIOManager #330

Closed

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from ec0955e to 6bc1222 Compare November 22, 2025 15:02

geoHeil force-pushed the 11-21-metaxyiomanager branch from b2a25af to 7aa614f Compare November 22, 2025 15:02

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from 6bc1222 to 93e8129 Compare November 22, 2025 15:21

geoHeil force-pushed the 11-21-metaxyiomanager branch 2 times, most recently from fe25f56 to 3b67f97 Compare November 22, 2025 15:59

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from 93e8129 to e3c15e2 Compare November 22, 2025 15:59

geoHeil force-pushed the 11-21-metaxyiomanager branch from 3b67f97 to 93573f1 Compare November 22, 2025 16:06

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from e3c15e2 to c0e935c Compare November 22, 2025 16:06

geoHeil self-assigned this Nov 22, 2025

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from c0e935c to 395eefb Compare November 22, 2025 19:38

geoHeil force-pushed the 11-21-metaxyiomanager branch 2 times, most recently from 7730994 to d6d4b1b Compare November 22, 2025 21:13

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from 395eefb to eabf512 Compare November 22, 2025 21:13

geoHeil force-pushed the 11-21-metaxyiomanager branch from d6d4b1b to cb4ee96 Compare November 22, 2025 22:04

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch 2 times, most recently from 1cd7cc6 to 71533ae Compare November 23, 2025 13:42

geoHeil force-pushed the 11-21-metaxyiomanager branch from cb4ee96 to 4549f62 Compare November 23, 2025 13:42

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from 71533ae to bb80037 Compare November 23, 2025 13:54

geoHeil force-pushed the 11-21-metaxyiomanager branch from 4549f62 to 9c9ac27 Compare November 23, 2025 13:54

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from bb80037 to 9ea652d Compare November 23, 2025 14:29

geoHeil force-pushed the 11-21-metaxyiomanager branch from 9c9ac27 to 70be0bf Compare November 23, 2025 16:50

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from 9ea652d to 90d60d8 Compare November 23, 2025 16:50

geoHeil force-pushed the 11-21-metaxyiomanager branch from 70be0bf to 2421ec2 Compare November 23, 2025 17:02

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from 90d60d8 to 39ef66e Compare November 23, 2025 17:02

geoHeil force-pushed the 11-21-metaxyiomanager branch from 2421ec2 to b2e3e09 Compare November 23, 2025 19:33

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch 2 times, most recently from 34e5568 to 46ef8ea Compare November 23, 2025 19:44

geoHeil force-pushed the 11-21-metaxyiomanager branch from b2e3e09 to 4fbf469 Compare November 23, 2025 19:44

tqdm style progress bar for dagster integration

c6988aa

geoHeil force-pushed the 11-21-metaxyiomanager branch from 4fbf469 to 3189d9c Compare November 23, 2025 19:57

geoHeil force-pushed the 11-22-tqdm_style_progress_bar_for_dagster_integration branch from 46ef8ea to c6988aa Compare November 23, 2025 19:57

geoHeil requested review from Copilot and danielgafni November 23, 2025 20:13

Copilot started reviewing on behalf of geoHeil November 23, 2025 20:14 View session

geoHeil assigned danielgafni Nov 23, 2025

Copilot finished reviewing on behalf of geoHeil November 23, 2025 20:16

Copilot AI reviewed Nov 23, 2025

View reviewed changes

danielgafni requested changes Nov 23, 2025

View reviewed changes

-### Progress/Logging Example
-Process in chunks with a tqdm-style progress bar (in TTY) and log updates:
-```bash
-dagster dev -m example_integration_dagster.progress_logging
-```
+### Progress/Logging
+Progress bar functionality (tqdm-style) and logging are demonstrated in the existing examples:
+- `non_partitioned.py`
+- `partitioned.py`
+- `branch_subsampled.py`
+Run any of these examples to see chunked processing with progress bars and log updates.

-    if total == 0 or chunk_size <= 0:
+    if chunk_size <= 0:
+        raise ValueError(f"chunk_size must be a positive integer, got {chunk_size}")
+    if total == 0:

tqdm style progress bar for dagster integration #337

Are you sure you want to change the base?

tqdm style progress bar for dagster integration #337

Uh oh!

Conversation

geoHeil commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geoHeil commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 23, 2025

Choose a reason for hiding this comment

Uh oh!

danielgafni left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

geoHeil commented Nov 22, 2025 •

edited

Loading

geoHeil commented Nov 22, 2025 •

edited

Loading

github-actions bot commented Nov 22, 2025 •

edited

Loading

github-actions bot commented Nov 22, 2025 •

edited

Loading

github-actions bot commented Nov 22, 2025 •

edited

Loading

github-actions bot commented Nov 22, 2025 •

edited

Loading

danielgafni left a comment •

edited

Loading