[Enhancement] introduce a function to obtain the column size #62481

murphyatwork · 2025-08-28T14:23:04Z

Why I'm doing:

This PR resolves issue #60535 by adding the ability to query column size and compressed column size via _META_ scans. This provides users with estimates of data storage for individual columns.

What I'm doing:

Introduced column_size(col) and column_compressed_size(col) built-in functions.
Implemented BE-side meta collection for these new fields in SegmentMetaCollecter and OlapMetaReader.
- column_size uses ColumnMetaPB.total_mem_footprint() as an uncompressed size proxy.
- column_compressed_size calculates the sum of data page sizes by iterating through ordinal page indexes.
Extended FE rules (PushDownAggToMetaScanRule, RewriteSimpleAggToMetaScanRule) to support pushing down SUM(column_size(col)) and SUM(column_compressed_size(col)) to meta scans.

Usage:

Decompressed estimate: SELECT column_size(col) FROM t [_META_];
Compressed estimate: SELECT column_compressed_size(col) FROM t [_META_];
Both can be aggregated and pushed down, e.g., SELECT sum(column_size(col)) FROM t [_META_];

Fixes #60535

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Co-authored-by: huanmingwong <huanmingwong@gmail.com>

cursor · 2025-08-28T14:23:05Z

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

CLAassistant · 2025-08-28T14:23:11Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ murphyatwork
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Co-authored-by: huanmingwong <huanmingwong@gmail.com>

Signed-off-by: Murphy <mofei@starrocks.com>

alvin-celerdata · 2025-08-29T08:46:28Z

@cursor review

be/src/storage/meta_reader.cpp

Signed-off-by: Murphy <mofei@starrocks.com>

cursor

This PR is being reviewed by Cursor Bugbot

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, please upgrade to Bugbot Pro by visiting the Cursor dashboard. Your first 14 days will be free!

be/src/storage/meta_reader.cpp

sonarqubecloud · 2025-09-02T07:33:18Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2025-09-02T16:03:53Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2025-09-02T16:06:36Z

[FE Incremental Coverage Report]

✅ pass : 20 / 20 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/optimizer/rule/transformation/PushDownAggToMetaScanRule.java	9	9	100.00%	[]
🔵	com/starrocks/catalog/FunctionSet.java	2	2	100.00%	[]
🔵	com/starrocks/sql/optimizer/rule/transformation/RewriteSimpleAggToMetaScanRule.java	9	9	100.00%	[]

github-actions · 2025-09-02T16:17:12Z

[BE Incremental Coverage Report]

✅ pass : 51 / 51 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/storage/olap_meta_reader.cpp	2	2	100.00%	[]
🔵	be/src/storage/meta_reader.cpp	41	41	100.00%	[]
🔵	be/src/storage/rowset/column_reader.h	1	1	100.00%	[]
🔵	be/src/storage/rowset/column_reader.cpp	7	7	100.00%	[]

github-actions · 2025-09-03T02:18:42Z

@Mergifyio backport branch-4.0

mergify · 2025-09-03T02:18:59Z

backport branch-4.0

✅ Backports have been created

#62674 [Enhancement] introduce a function to obtain the column size (backport #62481) has been created for branch branch-4.0

Co-authored-by: Cursor Agent <cursoragent@cursor.com> (cherry picked from commit 347b0b1)

…#62481) (#62674) Co-authored-by: Murphy <96611012+murphyatwork@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>

Add column size and compressed size metadata collection support

ca4fc0c

Co-authored-by: huanmingwong <huanmingwong@gmail.com>

github-actions bot added behavior_changed title needs [type] labels Aug 28, 2025

mergify bot assigned murphyatwork Aug 28, 2025

Add meta column size functions tests for null partition tables

4b40f1a

Co-authored-by: huanmingwong <huanmingwong@gmail.com>

murphyatwork changed the title ~~Resolve issue 60535~~ [Feature] introduce a function to obtain the column size Aug 29, 2025

github-actions bot removed title needs [type] behavior_changed labels Aug 29, 2025

murphyatwork marked this pull request as ready for review August 29, 2025 02:50

murphyatwork requested review from a team as code owners August 29, 2025 02:50

cursoragent and others added 3 commits August 29, 2025 03:26

Add test cases for meta column size with nested types

e0749f6

Co-authored-by: huanmingwong <huanmingwong@gmail.com>

fix compile

fd973bd

Signed-off-by: Murphy <mofei@starrocks.com>

support nested column

db4e45b

Signed-off-by: Murphy <mofei@starrocks.com>

murphyatwork changed the title ~~[Feature] introduce a function to obtain the column size~~ [Enhancement] introduce a function to obtain the column size Aug 29, 2025

github-actions bot added the 4.0 label Aug 29, 2025

murphyatwork added 2 commits August 29, 2025 13:02

fix the sub_reader size reading

20495bf

Signed-off-by: Murphy <mofei@starrocks.com>

fix sql test

59e5e9a

Signed-off-by: Murphy <mofei@starrocks.com>

murphyatwork requested a review from a team as a code owner August 29, 2025 05:15

fix format

ad2111f

Signed-off-by: Murphy <mofei@starrocks.com>

This comment was marked as outdated.

Sign in to view

kangkaisen previously approved these changes Aug 29, 2025

View reviewed changes

wyb reviewed Sep 1, 2025

View reviewed changes

be/src/storage/meta_reader.cpp Show resolved Hide resolved

be/src/storage/meta_reader.cpp Outdated Show resolved Hide resolved

recursive visit the sub_readers

40ed69e

Signed-off-by: Murphy <mofei@starrocks.com>

murphyatwork dismissed kangkaisen’s stale review via 40ed69e September 2, 2025 07:26

cursor bot reviewed Sep 2, 2025

View reviewed changes

be/src/storage/meta_reader.cpp Show resolved Hide resolved

wyb approved these changes Sep 2, 2025

View reviewed changes

kangkaisen approved these changes Sep 3, 2025

View reviewed changes

gengjun-git approved these changes Sep 3, 2025

View reviewed changes

murphyatwork merged commit 347b0b1 into main Sep 3, 2025
155 of 161 checks passed

murphyatwork deleted the cursor/resolve-issue-60535-ce6a branch September 3, 2025 02:18

github-actions bot removed the 4.0 label Sep 3, 2025

mergify bot pushed a commit that referenced this pull request Sep 3, 2025

[Enhancement] introduce a function to obtain the column size (#62481)

7192ff6

Co-authored-by: Cursor Agent <cursoragent@cursor.com> (cherry picked from commit 347b0b1)

mergify bot mentioned this pull request Sep 3, 2025

[Enhancement] introduce a function to obtain the column size (backport #62481) #62674

Merged

23 tasks

github-actions bot added the 4.0-merged label Sep 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Enhancement] introduce a function to obtain the column size #62481

[Enhancement] introduce a function to obtain the column size #62481

Uh oh!

murphyatwork commented Aug 28, 2025 •

edited

Loading

Uh oh!

cursor bot commented Aug 28, 2025

Uh oh!

CLAassistant commented Aug 28, 2025 •

edited

Loading

Uh oh!

alvin-celerdata commented Aug 29, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 2, 2025

Uh oh!

github-actions bot commented Sep 2, 2025

Uh oh!

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

mergify bot commented Sep 3, 2025 •

edited

Loading

Uh oh!

Uh oh!

[Enhancement] introduce a function to obtain the column size #62481

[Enhancement] introduce a function to obtain the column size #62481

Uh oh!

Conversation

murphyatwork commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

cursor bot commented Aug 28, 2025

Uh oh!

CLAassistant commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alvin-celerdata commented Aug 29, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

This PR is being reviewed by Cursor Bugbot

Uh oh!

Uh oh!

sonarqubecloud bot commented Sep 2, 2025

Quality Gate passed

Uh oh!

github-actions bot commented Sep 2, 2025

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions bot commented Sep 2, 2025

[FE Incremental Coverage Report]

file detail

Uh oh!

github-actions bot commented Sep 2, 2025

[BE Incremental Coverage Report]

file detail

Uh oh!

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

mergify bot commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

murphyatwork commented Aug 28, 2025 •

edited

Loading

CLAassistant commented Aug 28, 2025 •

edited

Loading

mergify bot commented Sep 3, 2025 •

edited

Loading