[CH-434] convert partition name to lower case by shuai-xu · Pull Request #435 · Kyligence/ClickHouse

shuai-xu · 2023-04-14T01:47:51Z

Changelog category (leave one):

Bug Fix (user-visible misbehavior in official stable or prestable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

This pr converts partition names get from hdfs path to lower case, as spark paritiions are in lower case.
It fix #434 .

Refactor releasing process

Simplify strip for new packages

…l_flag Resurrect build hash

…ash_v2 One more try to resurrect build hash

New package clickhouse-keeper

Docker release and regular builds

Fix automatic bucket prefix for master

…id IPv6, IPv4 values

…ltiple times

…ease Backport release to 22.3

Backport ClickHouse#35799 to 22.3: Fix extract function parser

Backport ClickHouse#35733 to 22.3: Added settings for insert of invalid IPv6, IPv4 values

Backport ClickHouse#35820 to 22.3: Avoid processing per-column TTL multiple times

- Allow define version as file - Add inline cache - Fix auto_release_type function

…umnar_shuffle

Support functions position/locate

…ence#354) ShuffleSplitter improvement: support multiple subdirs

Support full join with join condition Co-authored-by: shuai.li <shuai.li@kyligence.io>

Support Decimal type in Gluten Co-authored-by: shuai.li <shuai.li@kyligence.io>

…like Column 'deviceid' is not presented in input data (Kyligence#388)

Co-authored-by: shuai.li <shuai.li@kyligence.io>

…voking of JNI_Onload and finalized once at exit (Kyligence#377)

Co-authored-by: shuai.li <shuai.li@kyligence.io>

Support partitioning with expressions

…ion (Kyligence#403)

Co-authored-by: Robert Schulze <robert@clickhouse.com>

…sexplode (Kyligence#421)

kyligence-git · 2023-04-14T01:47:56Z

Can one of the admins verify this patch?

taiyang-li · 2023-04-17T03:01:54Z

utils/local-engine/Common/StringUtils.h

 public:
    static PartitionValues parsePartitionTablePath(const std::string & file);
    static bool isNullPartitionValue(const std::string & value);
+    static std::string toLowerCase(const std::string & value);


it is better to use boost::to_lower or boost::to_lower_copy, like in other places of CH codes

taiyang-li · 2023-04-17T03:06:38Z

utils/local-engine/Common/StringUtils.cpp

        if (position != std::string::npos)
        {
-            result.emplace_back(PartitionValue(item.substr(0,position), item.substr(position+1)));
+            result.emplace_back(PartitionValue(toLowerCase(item.substr(0,position)), item.substr(position+1)));


应该不止这个地方要改。

下面这段代码，从partition_values中搜索name之前，也需要将name统一转化成小写。

parquet文件中的列名匹配是否忽略大小写，是由FormatSetting::ORC::case_insensitive_column_matching和FormatSetting::Parquet::case_insensitive_column_matching控制的。这里最好也加上相应的开关判断。

Felixoid and others added 30 commits April 4, 2022 14:15

Improve black check: show diff in the output

1b298db

Fix version string update, fix ClickHouse#35518

046e742

Make GITHUB_RUN_URL variable and use it

3d43bce

Add build-url label to built docker images

7b01fe2

Merge pull request ClickHouse#33664 from ClickHouse/release-steps

e93a06c

Refactor releasing process

Merge pull request ClickHouse#35533 from ClickHouse/simplify_strip

dec0ec8

Simplify strip for new packages

Push only to the new CI DB

5b06fba

Remove outdated links from CI

5a5a174

Merge pull request ClickHouse#35766 from ClickHouse/resurrect_officia…

8cd46fa

…l_flag Resurrect build hash

Merge pull request ClickHouse#35774 from ClickHouse/ressurect_build_h…

f2e41bc

…ash_v2 One more try to resurrect build hash

Merge pull request ClickHouse#35308 from ClickHouse/clickhouse-keeper

9234a3c

New package clickhouse-keeper

Merge pull request ClickHouse#35211 from ClickHouse/release-docker

49bafdc

Docker release and regular builds

Merge pull request ClickHouse#35854 from ClickHouse/docker-master-head

899d7f2

Fix automatic bucket prefix for master

Backport tests/ci/upload_result_helper.py

b03e8ca

Backport ClickHouse#35733 to 22.3: Added settings for insert of inval…

b4cd668

…id IPv6, IPv4 values

Backport ClickHouse#35820 to 22.3: Avoid processing per-column TTL mu…

c9a1d9c

…ltiple times

Merge pull request ClickHouse#35909 from ClickHouse/backport/22.3-rel…

697dd21

…ease Backport release to 22.3

Merge pull request ClickHouse#35881 from ClickHouse/backport/22.3/35799

ad0a62d

Backport ClickHouse#35799 to 22.3: Fix extract function parser

Merge pull request ClickHouse#35928 from ClickHouse/backport/22.3/35733

82735cb

Backport ClickHouse#35733 to 22.3: Added settings for insert of invalid IPv6, IPv4 values

Merge pull request ClickHouse#35938 from ClickHouse/backport/22.3/35820

abb756d

Backport ClickHouse#35820 to 22.3: Avoid processing per-column TTL multiple times

remove input rel and support java iter from local files

025a573

Update version to 22.3.4.44

55da56d

Tiny improvements to git and version helpers

84020f5

Improve and fix edge cases for docker_server.py

0a43cfe

- Allow define version as file - Add inline cache - Fix auto_release_type function

add coalesce operator

27ee3a4

fix memory double free

574025a

Merge remote-tracking branch 'origin/21.9' into local_engine_with_col…

b228517

…umnar_shuffle

fix compile error in benchmark

507e81a

fix metrics error

131f6a6

Fix action for docker images build

ee1b577

exmy and others added 24 commits March 21, 2023 10:35

[CH-313] Support functions position/locate (Kyligence#314)

405e8e3

Support functions position/locate

[CH-353] ShuffleSplitter improvement: support multiple subdirs (Kylig…

9b896e0

…ence#354) ShuffleSplitter improvement: support multiple subdirs

Support full join with join condition (Kyligence#366)

0c8ea4e

Support full join with join condition Co-authored-by: shuai.li <shuai.li@kyligence.io>

[VL] Support Decimal type in Gluten (Kyligence#378)

ea45be8

Support Decimal type in Gluten Co-authored-by: shuai.li <shuai.li@kyligence.io>

[CH-387] Fix error caused by case-sensitive matching of ORC/Parquet, …

355a070

…like Column 'deviceid' is not presented in input data (Kyligence#388)

[CH-375] Fix create nullable column used ColumnInt8 (Kyligence#376)

6aad021

Co-authored-by: shuai.li <shuai.li@kyligence.io>

[CH-364] Make sure global_context is initialized once at the first in…

ea4a475

…voking of JNI_Onload and finalized once at exit (Kyligence#377)

Fix join key with alpha ASCII (Kyligence#390)

be666e8

Co-authored-by: shuai.li <shuai.li@kyligence.io>

Fix convert empty string to null (Kyligence#389)

eaefd7c

Support function space (Kyligence#321)

dcba13d

Optimize shuffle write (Kyligence#315)

8e5feef

[CH-383] Fix 'Unknown identifier' exception (Kyligence#385)

f57ffab

[CH-326] Support partitioning with expressions (Kyligence#332)

3eca6b8

Support partitioning with expressions

fix size of ColumnBuffer not configurable (Kyligence#396)

5d0e686

[394]Print exception message (Kyligence#395)

6fc029a

[CH-410] Fix coredump issue caused by broadcast join (Kyligence#411)

7d33ae4

Fix UT of benchmark_local_engine.cpp (Kyligence#400)

3028f6b

[CH-402]Enable return null and complex type for get_json_object funct…

876a30d

…ion (Kyligence#403)

[CH-397] support posexplode/sequence functions (Kyligence#399)

0417ceb

Co-authored-by: Robert Schulze <robert@clickhouse.com>

[405] Eliminate data skew in hash shuffle (Kyligence#406)

bffd750

[CH-358]Support json_tuple function (Kyligence#361)

2b37553

[398]Fixed: groupArray's result could be null (Kyligence#404)

ddf2f50

[CH-419] support map type for JSONExtract and fix bugs for explode/po…

ea6d8dd

…sexplode (Kyligence#421)

[CH-434] convert partition name to lower case

2c07773

taiyang-li reviewed Apr 17, 2023

View reviewed changes

liuneng1994 force-pushed the clickhouse_backend branch from 6528ff0 to 52be833 Compare April 25, 2023 06:43

lwz9103 force-pushed the clickhouse_backend branch 2 times, most recently from dc60d55 to 8066113 Compare May 26, 2023 03:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CH-434] convert partition name to lower case#435

[CH-434] convert partition name to lower case#435
shuai-xu wants to merge 765 commits intoKyligence:clickhouse_backendfrom
shuai-xu:CH-434

shuai-xu commented Apr 14, 2023

Uh oh!

kyligence-git commented Apr 14, 2023

Uh oh!

taiyang-li Apr 17, 2023 •

edited

Loading

Uh oh!

taiyang-li Apr 17, 2023 •

edited

Loading

Uh oh!

taiyang-li Apr 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

Conversation

shuai-xu commented Apr 14, 2023

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Uh oh!

kyligence-git commented Apr 14, 2023

Uh oh!

taiyang-li Apr 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taiyang-li Apr 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taiyang-li Apr 17, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

19 participants

taiyang-li Apr 17, 2023 •

edited

Loading

taiyang-li Apr 17, 2023 •

edited

Loading