Fix frequency inference for monthly data with missing months #199
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Separate merged test functions into distinct test cases - Add test case for frequency inference with missing months - Improve test documentation and organization
- Update frequency detection to correctly identify monthly patterns with missing months - Improve handling of irregular time steps in monthly data - Ensure consistent behavior across different calendar types
- Filter out zero-deltas (duplicates) before frequency inference - Prevents duplicates from dominating median calculation - Now correctly infers 'M' for monthly data with duplicates - Adds new status 'all_duplicates' for time series with only duplicates - Includes comprehensive regression tests for duplicate scenarios Fixes edge case where duplicate timestamps would cause incorrect frequency inference (e.g., returning '10D' instead of 'M' for monthly data with duplicates, or 'None' with too many duplicates).
- Add detection of duplicate timestamps (zero deltas) in time series data
- Filter out zero deltas before calculating median for frequency inference
- Report 'irregular' status when duplicates are present instead of 'valid'
- Handle edge case where all timestamps are duplicates ('all_duplicates' status)
- Add comprehensive regression tests for duplicate handling scenarios
- Fix is_resolution_fine_enough to consistently return 'status' key in all paths
This enhancement improves robustness for real-world climate data that often
contains irregularities like duplicate timestamps, providing accurate frequency
detection and clear diagnostic feedback to users.
- Calculate delta_days even when xarray.infer_freq succeeds - Convert delta_days from numpy.float64 to plain Python float - Convert is_exact from numpy.bool_ to plain Python bool - Improves data readability and consistency across inference paths
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses an edge case in the frequency inference logic where monthly data with a missing month (e.g., Jan, Feb, April) was being incorrectly identified as having a 5W frequency.
Before fix:
After fix: