Skip to content

Commit bbccbff

Browse files
davinci26davinci26
authored
Fixup: Sort path matches based on length rather than lexi (#5752)
Since Envoy is greedy matching path routes, order is important. Contour decides to sort the routes in a way that is not really intuitive and can lead to suprises. In particular even tho the comment in the code state that routes are ordered based on legnth the reality is that they are sorted based on string comparison. This PR fixes this. * I think the current behaviour doesnt make much sense and it is a bit brittle. * Updating the behaviour has significant update risk since there might be folks that rely on this routing behaviour without really knowing it. * Should we even merge this PR? I am of two minds and I would like some input: 1. Option (1): Merge it as and make a clear changelog/announcement about the fix 2. Option (2): Create a config flag with a feature-flag e.g. `route_sorting_strategy` and switch the implementation to not do sorting when the flag is present. That way it allows folks to opt-out from the sorting as they need to. Longest path based matching kinda makes sense to me now that I know about it, but it is rough edge than needs users to be familiar with contour and it is harder to socialize in larger teams. Signed-off-by: Sotiris Nanopoulos <sotiris.nanopoulos@reddit.com>
1 parent a9885b0 commit bbccbff

File tree

6 files changed

+110
-71
lines changed

6 files changed

+110
-71
lines changed

‎changelogs/unreleased/5752-davinci26-major.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
## Fix bug with algorithm used to sort Envoy regex/prefix path rules
2+
3+
Envoy greedy matches routes and as a result the order route matches are presented to Envoy is important. Contour attempts to produce consistent routing tables so that the most specific route matches are given preference. This is done to facilitate consistency when using HTTPProxy inclusion and provide a uniform user experience for route matching to be inline with Ingress and Gateway API Conformance.
4+
5+
This changes fixes the sorting algorithm used for `Prefix` and `Regex` based path matching. Previously the algorithm lexicographically sorted based on the path match string instead of sorting them based on the length of the `Prefix`|`Regex`. i.e. Longer prefix/regexes will be sorted first in order to give preference to more specific routes, then lexicographic sorting for things of the same length.
6+
7+
Note that for prefix matching, this change is _not_ expected to change the relative ordering of more specific prefixes vs. less specific ones when the more specific prefix match string has the less specific one as a prefix, e.g. `/foo/bar` will continue to sort before `/foo`. However, relative ordering of other combinations of prefix matches may change per the above description.
8+
### How to update safely
9+
10+
Caution is advised if you update Contour and you are operating large routing tables. We advise you to:
11+
12+
1. Deploy a duplicate Contour installation that parses the same CRDs
13+
2. Port-forward to the Envoy admin interface [docs](https://projectcontour.io/docs/v1.3.0/troubleshooting/)
14+
3. Access `http://127.0.0.1:9001/config_dump` and compare the configuration of Envoy. In particular the routes and their order. The prefix routes might be changing in order, so if they are you need to verify that the route matches as expected.
15+

‎internal/featuretests/v3/replaceprefix_test.go

Lines changed: 14 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -478,46 +478,45 @@ func artifactoryDocker(t *testing.T) {
478478
Resources: resources(t,
479479
envoy_v3.RouteConfiguration("ingress_http",
480480
envoy_v3.VirtualHost("artifactory.projectcontour.io",
481-
482481
&envoy_route_v3.Route{
483-
Match: routePrefix("/v2/container-sandbox/"),
482+
Match: routePrefix("/v2/container-external/"),
484483
Action: withPrefixRewrite(routeCluster("artifactory/service/8080/da39a3ee5e"),
485-
"/artifactory/api/docker/container-sandbox/v2/"),
484+
"/artifactory/api/docker/container-external/v2/"),
486485
},
487486
&envoy_route_v3.Route{
488-
Match: routePrefix("/v2/container-sandbox"),
487+
Match: routePrefix("/v2/container-sandbox/"),
489488
Action: withPrefixRewrite(routeCluster("artifactory/service/8080/da39a3ee5e"),
490-
"/artifactory/api/docker/container-sandbox/v2"),
489+
"/artifactory/api/docker/container-sandbox/v2/"),
491490
},
492491
&envoy_route_v3.Route{
493492
Match: routePrefix("/v2/container-release/"),
494493
Action: withPrefixRewrite(routeCluster("artifactory/service/8080/da39a3ee5e"),
495494
"/artifactory/api/docker/container-release/v2/"),
496495
},
497496
&envoy_route_v3.Route{
498-
Match: routePrefix("/v2/container-release"),
497+
Match: routePrefix("/v2/container-external"),
499498
Action: withPrefixRewrite(routeCluster("artifactory/service/8080/da39a3ee5e"),
500-
"/artifactory/api/docker/container-release/v2"),
499+
"/artifactory/api/docker/container-external/v2"),
501500
},
502501
&envoy_route_v3.Route{
503-
Match: routePrefix("/v2/container-public/"),
502+
Match: routePrefix("/v2/container-sandbox"),
504503
Action: withPrefixRewrite(routeCluster("artifactory/service/8080/da39a3ee5e"),
505-
"/artifactory/api/docker/container-public/v2/"),
504+
"/artifactory/api/docker/container-sandbox/v2"),
506505
},
507506
&envoy_route_v3.Route{
508-
Match: routePrefix("/v2/container-public"),
507+
Match: routePrefix("/v2/container-release"),
509508
Action: withPrefixRewrite(routeCluster("artifactory/service/8080/da39a3ee5e"),
510-
"/artifactory/api/docker/container-public/v2"),
509+
"/artifactory/api/docker/container-release/v2"),
511510
},
512511
&envoy_route_v3.Route{
513-
Match: routePrefix("/v2/container-external/"),
512+
Match: routePrefix("/v2/container-public/"),
514513
Action: withPrefixRewrite(routeCluster("artifactory/service/8080/da39a3ee5e"),
515-
"/artifactory/api/docker/container-external/v2/"),
514+
"/artifactory/api/docker/container-public/v2/"),
516515
},
517516
&envoy_route_v3.Route{
518-
Match: routePrefix("/v2/container-external"),
517+
Match: routePrefix("/v2/container-public"),
519518
Action: withPrefixRewrite(routeCluster("artifactory/service/8080/da39a3ee5e"),
520-
"/artifactory/api/docker/container-external/v2"),
519+
"/artifactory/api/docker/container-public/v2"),
521520
},
522521
),
523522
),

‎internal/featuretests/v3/route_test.go

Lines changed: 28 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1237,26 +1237,26 @@ func TestRouteWithTLS_InsecurePaths(t *testing.T) {
12371237
Resources: routeResources(t,
12381238
envoy_v3.RouteConfiguration("ingress_http",
12391239
envoy_v3.VirtualHost("test2.test.com",
1240-
&envoy_route_v3.Route{
1241-
Match: routePrefix("/secure"),
1242-
Action: envoy_v3.UpgradeHTTPS(),
1243-
},
12441240
&envoy_route_v3.Route{
12451241
Match: routePrefix("/insecure"),
12461242
Action: routecluster("default/kuard/80/da39a3ee5e"),
12471243
},
1244+
&envoy_route_v3.Route{
1245+
Match: routePrefix("/secure"),
1246+
Action: envoy_v3.UpgradeHTTPS(),
1247+
},
12481248
),
12491249
),
12501250
envoy_v3.RouteConfiguration("https/test2.test.com",
12511251
envoy_v3.VirtualHost("test2.test.com",
1252-
&envoy_route_v3.Route{
1253-
Match: routePrefix("/secure"),
1254-
Action: routecluster("default/svc2/80/da39a3ee5e"),
1255-
},
12561252
&envoy_route_v3.Route{
12571253
Match: routePrefix("/insecure"),
12581254
Action: routecluster("default/kuard/80/da39a3ee5e"),
12591255
},
1256+
&envoy_route_v3.Route{
1257+
Match: routePrefix("/secure"),
1258+
Action: routecluster("default/svc2/80/da39a3ee5e"),
1259+
},
12601260
),
12611261
),
12621262
),
@@ -1335,25 +1335,25 @@ func TestRouteWithTLS_InsecurePaths_DisablePermitInsecureTrue(t *testing.T) {
13351335
envoy_v3.RouteConfiguration("ingress_http",
13361336
envoy_v3.VirtualHost("test2.test.com",
13371337
&envoy_route_v3.Route{
1338-
Match: routePrefix("/secure"),
1338+
Match: routePrefix("/insecure"),
13391339
Action: envoy_v3.UpgradeHTTPS(),
13401340
},
13411341
&envoy_route_v3.Route{
1342-
Match: routePrefix("/insecure"),
1342+
Match: routePrefix("/secure"),
13431343
Action: envoy_v3.UpgradeHTTPS(),
13441344
},
13451345
),
13461346
),
13471347
envoy_v3.RouteConfiguration("https/test2.test.com",
13481348
envoy_v3.VirtualHost("test2.test.com",
1349-
&envoy_route_v3.Route{
1350-
Match: routePrefix("/secure"),
1351-
Action: routecluster("default/svc2/80/da39a3ee5e"),
1352-
},
13531349
&envoy_route_v3.Route{
13541350
Match: routePrefix("/insecure"),
13551351
Action: routecluster("default/kuard/80/da39a3ee5e"),
13561352
},
1353+
&envoy_route_v3.Route{
1354+
Match: routePrefix("/secure"),
1355+
Action: routecluster("default/svc2/80/da39a3ee5e"),
1356+
},
13571357
),
13581358
),
13591359
),
@@ -1609,26 +1609,26 @@ func TestHTTPProxyRouteWithTLS_InsecurePaths(t *testing.T) {
16091609
Resources: routeResources(t,
16101610
envoy_v3.RouteConfiguration("ingress_http",
16111611
envoy_v3.VirtualHost("test2.test.com",
1612-
&envoy_route_v3.Route{
1613-
Match: routePrefix("/secure"),
1614-
Action: envoy_v3.UpgradeHTTPS(),
1615-
},
16161612
&envoy_route_v3.Route{
16171613
Match: routePrefix("/insecure"),
16181614
Action: routecluster("default/kuard/80/da39a3ee5e"),
16191615
},
1616+
&envoy_route_v3.Route{
1617+
Match: routePrefix("/secure"),
1618+
Action: envoy_v3.UpgradeHTTPS(),
1619+
},
16201620
),
16211621
),
16221622
envoy_v3.RouteConfiguration("https/test2.test.com",
16231623
envoy_v3.VirtualHost("test2.test.com",
1624-
&envoy_route_v3.Route{
1625-
Match: routePrefix("/secure"),
1626-
Action: routecluster("default/svc2/80/da39a3ee5e"),
1627-
},
16281624
&envoy_route_v3.Route{
16291625
Match: routePrefix("/insecure"),
16301626
Action: routecluster("default/kuard/80/da39a3ee5e"),
16311627
},
1628+
&envoy_route_v3.Route{
1629+
Match: routePrefix("/secure"),
1630+
Action: routecluster("default/svc2/80/da39a3ee5e"),
1631+
},
16321632
),
16331633
),
16341634
),
@@ -1703,25 +1703,25 @@ func TestHTTPProxyRouteWithTLS_InsecurePaths_DisablePermitInsecureTrue(t *testin
17031703
envoy_v3.RouteConfiguration("ingress_http",
17041704
envoy_v3.VirtualHost("test2.test.com",
17051705
&envoy_route_v3.Route{
1706-
Match: routePrefix("/secure"),
1706+
Match: routePrefix("/insecure"),
17071707
Action: envoy_v3.UpgradeHTTPS(),
17081708
},
17091709
&envoy_route_v3.Route{
1710-
Match: routePrefix("/insecure"),
1710+
Match: routePrefix("/secure"),
17111711
Action: envoy_v3.UpgradeHTTPS(),
17121712
},
17131713
),
17141714
),
17151715
envoy_v3.RouteConfiguration("https/test2.test.com",
17161716
envoy_v3.VirtualHost("test2.test.com",
1717-
&envoy_route_v3.Route{
1718-
Match: routePrefix("/secure"),
1719-
Action: routecluster("default/svc2/80/da39a3ee5e"),
1720-
},
17211717
&envoy_route_v3.Route{
17221718
Match: routePrefix("/insecure"),
17231719
Action: routecluster("default/kuard/80/da39a3ee5e"),
17241720
},
1721+
&envoy_route_v3.Route{
1722+
Match: routePrefix("/secure"),
1723+
Action: routecluster("default/svc2/80/da39a3ee5e"),
1724+
},
17251725
),
17261726
),
17271727
),

‎internal/sorter/sorter.go

Lines changed: 30 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -296,33 +296,47 @@ func (s routeSorter) Less(i, j int) bool {
296296
switch a := s[i].PathMatchCondition.(type) {
297297
case *dag.PrefixMatchCondition:
298298
if b, ok := s[j].PathMatchCondition.(*dag.PrefixMatchCondition); ok {
299-
cmp := strings.Compare(a.Prefix, b.Prefix)
300-
switch cmp {
301-
case 1:
299+
switch {
300+
case len(a.Prefix) > len(b.Prefix):
302301
// Sort longest prefix first.
303302
return true
304-
case -1:
303+
case len(a.Prefix) < len(b.Prefix):
305304
return false
306305
default:
307-
if a.PrefixMatchType == b.PrefixMatchType {
308-
return compareRoutesByMethodHeaderQueryParams(s[i], s[j])
306+
cmp := strings.Compare(a.Prefix, b.Prefix)
307+
switch cmp {
308+
case 1:
309+
return true
310+
case -1:
311+
return false
312+
default:
313+
if a.PrefixMatchType == b.PrefixMatchType {
314+
return compareRoutesByMethodHeaderQueryParams(s[i], s[j])
315+
}
316+
// Segment prefixes sort first as they are more specific.
317+
return a.PrefixMatchType == dag.PrefixMatchSegment
309318
}
310-
// Segment prefixes sort first as they are more specific.
311-
return a.PrefixMatchType == dag.PrefixMatchSegment
312319
}
313320
}
314321
case *dag.RegexMatchCondition:
315322
switch b := s[j].PathMatchCondition.(type) {
316323
case *dag.RegexMatchCondition:
317-
cmp := strings.Compare(a.Regex, b.Regex)
318-
switch cmp {
319-
case 1:
324+
switch {
325+
case len(a.Regex) > len(b.Regex):
320326
// Sort longest regex first.
321327
return true
322-
case -1:
328+
case len(a.Regex) < len(b.Regex):
323329
return false
324330
default:
325-
return compareRoutesByMethodHeaderQueryParams(s[i], s[j])
331+
cmp := strings.Compare(a.Regex, b.Regex)
332+
switch cmp {
333+
case 1:
334+
return true
335+
case -1:
336+
return false
337+
default:
338+
return compareRoutesByMethodHeaderQueryParams(s[i], s[j])
339+
}
326340
}
327341
case *dag.PrefixMatchCondition:
328342
return true
@@ -331,9 +345,11 @@ func (s routeSorter) Less(i, j int) bool {
331345
switch b := s[j].PathMatchCondition.(type) {
332346
case *dag.ExactMatchCondition:
333347
cmp := strings.Compare(a.Path, b.Path)
348+
// Sorting function doesn't really matter here
349+
// since we want exact matching. Lexicographic sorting
350+
// is ok
334351
switch cmp {
335352
case 1:
336-
// Sort longest path first.
337353
return true
338354
case -1:
339355
return false

‎internal/sorter/sorter_test.go

Lines changed: 21 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -279,32 +279,35 @@ func TestSortRoutesPathMatch(t *testing.T) {
279279
},
280280
// Note that regex matches sort before prefix matches.
281281
{
282-
PathMatchCondition: matchRegex("/this/is/the/longest"),
282+
PathMatchCondition: matchRegex("/athis/is/the/longest"),
283283
},
284284
{
285285
PathMatchCondition: matchRegex(`/foo((\/).*)*`),
286286
},
287287
{
288-
PathMatchCondition: matchRegex("/"),
288+
PathMatchCondition: matchRegex("/foo.*"),
289+
},
290+
{
291+
PathMatchCondition: matchRegex("/bar.*"),
289292
},
290293
{
291-
PathMatchCondition: matchRegex("."),
294+
PathMatchCondition: matchRegex("/"),
292295
},
293296
// Prefix segment matches sort before string matches.
294297
{
295-
PathMatchCondition: matchPrefixSegment("/path/prefix2"),
298+
PathMatchCondition: matchPrefixSegment("/path/prefix/a"),
296299
},
297300
{
298-
PathMatchCondition: matchPrefixString("/path/prefix2"),
301+
PathMatchCondition: matchPrefixString("/path/prefix/a"),
299302
},
300303
{
301-
PathMatchCondition: matchPrefixSegment("/path/prefix/a"),
304+
PathMatchCondition: matchPrefixString("/path/prf222"),
302305
},
303306
{
304-
PathMatchCondition: matchPrefixString("/path/prefix/a"),
307+
PathMatchCondition: matchPrefixString("/path/prf122"),
305308
},
306309
{
307-
PathMatchCondition: matchPrefixString("/path/prefix"),
310+
PathMatchCondition: matchPrefixString("/path/prfx"),
308311
},
309312
{
310313
PathMatchCondition: matchPrefixSegment("/path/p"),
@@ -389,25 +392,31 @@ func TestSortRoutesLongestHeaders(t *testing.T) {
389392
PathMatchCondition: matchExact("/pathexact"),
390393
},
391394
{
392-
PathMatchCondition: matchRegex("/pathregex"),
395+
PathMatchCondition: matchRegex("/pathregex2"),
396+
HeaderMatchConditions: []dag.HeaderMatchCondition{
397+
presentHeader("header-name"),
398+
},
399+
},
400+
{
401+
PathMatchCondition: matchRegex("/pathregex1"),
393402
HeaderMatchConditions: []dag.HeaderMatchCondition{
394403
exactHeader("header-name", "header-value"),
395404
},
396405
},
397406
{
398-
PathMatchCondition: matchRegex("/pathregex"),
407+
PathMatchCondition: matchRegex("/pathregex1"),
399408
HeaderMatchConditions: []dag.HeaderMatchCondition{
400409
presentHeader("header-name"),
401410
},
402411
},
403412
{
404-
PathMatchCondition: matchRegex("/pathregex"),
413+
PathMatchCondition: matchRegex("/pathregex1"),
405414
HeaderMatchConditions: []dag.HeaderMatchCondition{
406415
exactHeader("long-header-name", "long-header-value"),
407416
},
408417
},
409418
{
410-
PathMatchCondition: matchRegex("/pathregex"),
419+
PathMatchCondition: matchRegex("/pathregex1"),
411420
},
412421
{
413422
PathMatchCondition: matchPrefixSegment("/path"),

‎internal/xdscache/v3/route_test.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3722,9 +3722,9 @@ func TestSortLongestRouteFirst(t *testing.T) {
37223722
PathMatchCondition: &dag.RegexMatchCondition{Regex: "/v1/.+"},
37233723
}},
37243724
want: []*dag.Route{{
3725-
PathMatchCondition: &dag.RegexMatchCondition{Regex: "/v2"},
3726-
}, {
37273725
PathMatchCondition: &dag.RegexMatchCondition{Regex: "/v1/.+"},
3726+
}, {
3727+
PathMatchCondition: &dag.RegexMatchCondition{Regex: "/v2"},
37283728
}},
37293729
},
37303730
"two exact matches": {

0 commit comments

Comments
 (0)