-
Notifications
You must be signed in to change notification settings - Fork 118
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Description
It's quite annoying that users need to apply filter in majority of their pipelines while the default filtering code is quite heavy: .filter(dc.C("file.name").glob("*.mp3"))
We need to introduce a shortcuts to simplify it:
Pattern based.read_storage("s3://mybkt/dir1/dir2/", patterns="*.mp3")
(note, plural - supports list)- Wildcards
.read_storage("s3://mybkt/dir1/dir2/dir3/*.mp3")
Also, probably we need?
. - Globstar based
.read_storage("s3://mybkt/dir1/dir2/**/*.mp3")
- Brace Expansion
.read_storage("s3://mybkt/dir1/dir2/**/*.{mp3, wav}")
All should just apply .filter()
under the hood.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request