FATAL ERROR: 138 regression(s) detected.

[uerdemiryukatech]

Hi there,

I installed Pelias Planet build on my server with prepare extracts from Geoearth rather than preparing myself but i got some errors when i ran test script. What can be wrong in my configuration?

Describe the bug

Results are under the bellow:

Pass: 503
Improvements: 14
Expected Failures: 101
Placeholders: 0
Regressions: 138
Total tests: 756
Took 19113ms
Test success rate 81.75%

Environment (please complete the following information):

• OS: Ubuntu 20
• Docker versions: Docker version 20.10.21, build baeda1f

[geoops]

I encountered something similar on a planet build. Here is an example regression:

✘ regression [7] "/v1/search?sources=wof&text=Città del Vaticano, Vatican": score 3 out of 6
  diff:
    name
      expected: Vatican City
      actual:   Città del Vaticano
    locality
      expected: Vatican City
      actual:   Città del Vaticano
    country
      expected: Vatican City
      actual:   Vatican

When I compare my local output to the public and dev APIs they seem to have the same "regressions" https://pelias.github.io/compare/#/v1/search?sources=wof&text=Citt%C3%A0+del+Vaticano%2C+Vatican

[michaelkirk]

Sorry for the possible double post... GH UI is giving me grief.

These are integration tests testing the entire stack with real input data. Since the input data can change (planet-latest.osm.pbf, latest WOF, etc.), it follows that the expected output will change.

I just ran the tests against north-america and get a handful of failures:

Aggregate test results
Pass: 233
Improvements: 18
Expected Failures: 28
Placeholders: 0
Regressions: 84
Total tests: 363
Took 10415ms
Test success rate 76.86%

FATAL ERROR: 84 regression(s) detected.

Most of them seem like the kind of thing I'd expect from changing input data. e.g.:

tizen sdk: places
  ✘ [maps_service_search_place] "/v1/nearby?boundary.circle.radius=10&categories=food:pizza&point.lat=40.235771&point.lon=-74.937712": score 1 out of 2
  diff:
    name
      expected: Jule's Thin Crust
      actual:   Jule's Thin Crust Pizza

There's an argument to be made for pinning the tests to known input data, and then updating the inputs & tests in lockstep, but that might be more work for the maintainers.

full output of north-america tests

Tests for: http://api:4000/v1 (docker)

address interpolation
  ✘ regression [1] "/v1/search?text=475f west 26th street, new york": score 0 out of 7
  diff:
    layer
      expected: address
      actual:   locality
    source
      expected: mixed
      actual:   whosonfirst
    housenumber
      expected: 475
      actual:   
    street
      expected: West 26th Street
      actual:   
    match_type
      expected: interpolated
      actual:   fallback
    accuracy
      expected: point
      actual:   centroid
    'New York, NY, USA' is not close enough: distance is 7874m but should be under 50m
  ✘ regression [1a] "/v1/search?size=1&text=475f west 26th street, new york": score 0 out of 7
  diff:
    layer
      expected: address
      actual:   locality
    source
      expected: mixed
      actual:   whosonfirst
    housenumber
      expected: 475
      actual:   
    street
      expected: West 26th Street
      actual:   
    match_type
      expected: interpolated
      actual:   fallback
    accuracy
      expected: point
      actual:   centroid
    'New York, NY, USA' is not close enough: distance is 7874m but should be under 50m
  ✘ regression [2] "/v1/search?text=257 west 26th street, new york": score 0 out of 7
  diff:
    layer
      expected: address
      actual:   locality
    source
      expected: mixed
      actual:   whosonfirst
    housenumber
      expected: 257
      actual:   
    street
      expected: West 26th Street
      actual:   
    match_type
      expected: interpolated
      actual:   fallback
    accuracy
      expected: point
      actual:   centroid
    'New York, NY, USA' is not close enough: distance is 7453m but should be under 50m
  ✘ [2a] "/v1/search?text=257 west 26th street": score 4 out of 7
  diff:
    source
      expected: mixed
      actual:   openaddresses
    match_type
      expected: interpolated
      actual:   exact
    '257 West 26th Street, Northampton, PA, USA' is not close enough: distance is 127200m but should be under 50m

address matching
  ✔ [1] "/v1/search?text=30 w 26 st"
  ✔ [3] "/v1/search?size=1&text=49 Kay Street"
  ✔ [4] "/v1/search?text=461 W Main St, Cheshire, 06410"
  ✘ regression [5] "/v1/search?text=190 dean street brooklyn": score 0 out of 1
  diff:
    label
      expected: 190 Dean Street, Brooklyn, New York, NY, USA
      actual:   190 Dean Street, Brooklyn, NY, USA

address parsing
  ✘ regression [1] "/v1/search?text=101 saint marks pl 10009": score 12 out of 13
  diff:
    label
      expected: 101 Saint Marks Place, Manhattan, New York, NY, USA
      actual:   101 Saint Marks Place, New York, NY, USA
  ✔ [2] "/v1/search?text=1 water st manhattan ny"
  ✔ [2.2] "/v1/search?text=1 water st manhattan ny"
  ✘ regression [3] "/v1/search?size=1&text=450 37th st, new york, ny 11232": score 12 out of 13
  diff:
    label
      expected: 450 37th Street, Brooklyn, New York, NY, USA
      actual:   450 37th Street, Brooklyn, NY, USA
  ✔ [4] "/v1/search?text=starbucks 10010"
  ✘ regression [5] "/v1/search?text=455 43rd st new york ny 11232": score 12 out of 13
  diff:
    label
      expected: 455 43rd Street, Brooklyn, New York, NY, USA
      actual:   455 43rd Street, Brooklyn, NY, USA
  ✘ regression [6] "/v1/search?text=1 main st ny 11201": score 12 out of 13
  diff:
    label
      expected: 1 Main Street, Brooklyn, New York, NY, USA
      actual:   1 Main Street, Brooklyn, NY, USA
  ✘ regression [7] "/v1/search?text=186 Tuskegee St SE Atlanta GA": score 8 out of 11
  diff:
    name
      expected: 186 Tuskegee Street Southeast
      actual:   186 Tuskegee Street SE
    street
      expected: Tuskegee Street Southeast
      actual:   Tuskegee Street SE
    label
      expected: 186 Tuskegee Street Southeast, Atlanta, GA, USA
      actual:   186 Tuskegee Street SE, Atlanta, GA, USA
  ✔ [8] "/v1/search?text=3122 16th St San Francisco, CA 94103"
  ✔ [9] "/v1/search?text=3010 20th St San Francisco CA 94110"
  ✘ regression [10] "/v1/search?text=3577 Jackson St San Francisco, CA 94118": score 7 out of 11
  diff:
    name
      expected: 3577 Jackson St
      actual:   3577 Jackson Street
    postalcode
      expected: 94118
      actual:   94118.0
    street
      expected: Jackson St
      actual:   Jackson Street
    label
      expected: 3577 Jackson St, San Francisco, CA, USA
      actual:   3577 Jackson Street, San Francisco, CA, USA
  ✘ regression [11] "/v1/search?text=whole foods NY": score 4 out of 5
  diff:
    name
      expected: Whole Foods
      actual:   Whole Foods Market
  ✔ [11] "/v1/search?text=301 Commons Park S, Stamford, CT 06902"

address type
  ✘ regression [1] "/v1/search?focus.point.lat=51.53177&focus.point.lon=-0.06672&text=102 Fleet Street": score 1 out of 4
  diff:
    gid
      expected: openstreetmap:address:node:1401849738
      actual:   openaddresses:address:us/nj/statewide:595383c77dad8aa8
    source
      expected: openstreetmap
      actual:   openaddresses
    id
      expected: node:1401849738
      actual:   us/nj/statewide:595383c77dad8aa8
  ✘ regression [2] "/v1/search?focus.point.lat=51.219671&focus.point.lon=-0.779389&text=Shepherd and Flock": score 2 out of 4
  diff:
    gid
      expected: openstreetmap:venue:way:338854694
      actual:   openstreetmap:venue:node/7039565624
    id
      expected: way:338854694
      actual:   node/7039565624
  ✘ regression [3] "/v1/search?focus.point.lat=51.219671&focus.point.lon=-0.779389&text=22 Moor Park Lane": score 1 out of 4
  diff:
    gid
      expected: openstreetmap:address:way:338854694
      actual:   openaddresses:address:us/ny/statewide:4ab9b02cd6aa86e0
    source
      expected: openstreetmap
      actual:   openaddresses
    id
      expected: way:338854694
      actual:   us/ny/statewide:4ab9b02cd6aa86e0

admin lookup
  ✔ [2] "/v1/search?layers=county&text=bronx"
  ✔ [3] "/v1/search?layers=county&text=Lake, MT"
  ✘ regression [4] "/v1/reverse?point.lat=-40.806533324215565&point.lon=144.8101043701172": no results returned
  ✔ [7] "/v1/search?layers=coarse&text=mission, san francisco"
  ✘ regression [8] "/v1/search?layers=coarse&text=crown heights": score 7 out of 8
  diff:
    label
      expected: Crown Heights, Brooklyn, New York, NY, USA
      actual:   Crown Heights, Brooklyn, NY, USA
  ✔ [9] "/v1/search?layers=coarse&text=palo alto"
  ✔ [9-1] "/v1/search?layers=coarse&text=palo alto, ca"
  ✔ [13] "/v1/search?layers=neighbourhood&text=Vista Center"
  ✔ [14] "/v1/search?layers=neighbourhood&text=Yadkin Trail"
  ✔ [15] "/v1/search?layers=neighbourhood&text=Hillside, Santa Clara"
  ✔ [18] "/v1/search?layers=region&text=california"
  ✘ regression [19] "/v1/search?text=5 West 4th Avenue Canada": score 5 out of 6
  diff:
    street
      expected: W 4th Ave
      actual:   West 4th Avenue

autocomplete acushnet antiques
  ✘ regression [1] "/v1/autocomplete?focus.point.lat=41.652889&focus.point.lon=-70.922898&text=acus": score 0 out of 1
  diff:
    name
      expected: Acushnet River Antiques Showroom
      actual:   Acushnet
  ✔ [2] "/v1/autocomplete?focus.point.lat=41.652889&focus.point.lon=-70.922898&text=antiq"
  ✔ [3] "/v1/autocomplete?focus.point.lat=41.652889&focus.point.lon=-70.922898&text=antique"
  ✔ [4] "/v1/autocomplete?text=acushnet an"

autocomplete admin areas
  ✔ [1] "/v1/autocomplete?text=brooklyn"
  ✔ [1-1] "/v1/autocomplete?text=broo"
  ✔ [1-2] "/v1/autocomplete?text=brook"
  ✔ [1-3] "/v1/autocomplete?text=brookl"
  ✔ [1-4] "/v1/autocomplete?text=brookly"
  ✔ [1-5] "/v1/autocomplete?text=brooklyn"
  ✘ [2] "/v1/autocomplete?text=new york (2 expectations)": score 2 out of 3
  diff:
    label
      expected: New York City, Manhattan, NY, USA
      actual:   New York, NY, USA
  ✔ improvement [3] "/v1/autocomplete?text=london"
  ✔ [4] "/v1/autocomplete?text=san francisco"
  ✔ improvement [5] "/v1/autocomplete?text=victoria"
  ✔ [9] "/v1/autocomplete?text=california"

autocomplete boundary.country
  ✔ [1] "/v1/autocomplete?boundary.country=USA&text=London"

autocomplete boundary.rect
  ✘ regression [1] "/v1/autocomplete?boundary.rect.min_lat=19&boundary.rect.max_lat=20&boundary.rect.min_lon=-99&boundary.rect.max_lon=-98&text=san francisco": score 2 out of 3
  diff:
    name
      expected: San Francisco
      actual:   San Francisco Acuautla

autocomplete daly city
  ✔ improvement [1] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=dal"
  ✔ [2] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly"
  ✔ [3] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly "
  ✔ [4] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly c"
  ✔ [5] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly ci"
  ✔ [6] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly cit"
  ✔ [7] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly city"

autocomplete focus
  ✘ regression [1] "/v1/autocomplete?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=DiDi dumpling": score 0 out of 1
  diff:
    label
      expected: DiDi Dumpling, Manhattan, New York, NY, USA
      actual:   DiDi Dumpling, New York, NY, USA
  ✘ [2] "/v1/autocomplete?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=union square": score 0 out of 1
  diff:
    label
      expected: Union Square, New York County, NY, USA
      actual:   Union Square, Elizabeth, NJ, USA
  ✘ [3] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=union square": score 0 out of 1
  diff:
    label
      expected: Union Square, San Francisco County, CA, USA
      actual:   Union Square, San Francisco, CA, USA
  ✘ regression [5] "/v1/autocomplete?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=hard rock cafe": score 0 out of 1
  diff:
    label
      expected: Hard Rock Cafe, Manhattan, New York, NY, USA
      actual:   Hard Rock Cafe, New York, NY, USA
  ✔ improvement [24] "/v1/autocomplete?focus.point.lat=37.743618&focus.point.lon=-122.426117&text=zoo"
  ✔ [25] "/v1/autocomplete?focus.point.lat=40.744131&focus.point.lon=-73.990424&text=1 madison ave"
  ✘ regression [26] "/v1/autocomplete?focus.point.lat=40.744131&focus.point.lon=-73.990424&text=w 26th st": score 3 out of 4
  diff:
    '5th Avenue & West 26th Street, New York, NY, USA' is not close enough: distance is 191m but should be under 100m
  ✘ [27] "/v1/autocomplete?focus.point.lat=41.05343&focus.point.lon=-73.53873&text=New York": score 4 out of 5
  diff:
    'New York, NY, USA' is not close enough: distance is 3463m but should be under 100m
  ✘ [28] "/v1/autocomplete?focus.point.lat=41.05343&focus.point.lon=-73.53873&text=New York, NY": score 4 out of 5
  diff:
    'New York, NY, USA' is not close enough: distance is 3463m but should be under 100m
  ✔ [29] "/v1/autocomplete?focus.point.lat=40.744131&focus.point.lon=-73.990424&text=San Francisco"

autocomplete jitter
  ✘ regression [1.1] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dea": score 3 out of 4
  diff:
    priorityThresh is 2 but found at position 5
  ✘ regression [1.2] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean": score 3 out of 4
  diff:
    priorityThresh is 2 but found at position 3
  ✘ regression [1.3] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean ": score 3 out of 4
  diff:
    priorityThresh is 2 but found at position 3
  ✔ [1.4] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean s"
  ✔ [1.5] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean st"
  ✔ [1.6] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean str"
  ✔ [1.7] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean stre"
  ✔ [1.8] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean stree"
  ✔ [1.9] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street"
  ✔ [1.10] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street "
  ✔ [1.11] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street b"
  ✔ [1.12] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street br"
  ✔ [1.13] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street bro"
  ✔ [1.14] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street broo"
  ✔ [1.15] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street brook"
  ✔ [1.16] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street brookl"
  ✔ [1.17] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street brookly"
  ✔ [1.18] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street brooklyn"

Autocomplete POI
  ✔ improvement [1-1] "/v1/autocomplete?text=Target Eureka CA"
  ✔ improvement [1-2] "/v1/autocomplete?text=Target Eureka California"
  ✔ [1-3] "/v1/autocomplete?focus.point.lat=40.801944&focus.point.lon=-124.163611&text=Target"
  ✔ improvement [1-4] "/v1/autocomplete?focus.point.lat=40.801944&focus.point.lon=-124.163611&text=Target Eureka"
  ✔ [2] "/v1/autocomplete?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=didi"

autocomplete stop words
  ✔ [1] "/v1/autocomplete?text=world trade center"
  ✔ [3] "/v1/autocomplete?text=moscone west"

autocomplete street centroids
  ✘ regression [3] "/v1/autocomplete?sources=osm&layers=street&text=w 26 st, new york": no results returned

autocomplete street fallback
  ✘ regression [2] "/v1/autocomplete?text=w 26th st, new york (2 expectations)": score 4 out of 8
  diff:
    street
      expected: West 26th Street
      actual:   
    layer
      expected: street
      actual:   venue
    street
      expected: West 26th Street
      actual:   
    layer
      expected: address
      actual:   venue

Autocomplete Streets
  ✔ [1-1] "/v1/autocomplete?text=90 Vermilyea (2 expectations)"
  ✔ [1-2] "/v1/autocomplete?text=90 Vermilyea A (2 expectations)"
  ✔ [1-2] "/v1/autocomplete?text=90 Vermilyea Av (2 expectations)"
  ✔ [1-3] "/v1/autocomplete?text=90 Vermilyea Ave (2 expectations)"
  ✔ [1-4] "/v1/autocomplete?text=90 Vermilyea Aven (2 expectations)"
  ✔ [1-5] "/v1/autocomplete?text=90 Vermilyea Avenue (2 expectations)"
  ✘ [2-1] "/v1/autocomplete?text=424 Clay Av": score 2 out of 3
  diff:
    street
      expected: Clay Ave
      actual:   Clay Avenue
  ✘ regression [2-2] "/v1/autocomplete?text=424 Clay Ave (2 expectations)": score 6 out of 7
  diff:
    street
      expected: Clay Ave
      actual:   Clay Avenue
  ✔ [2-3] "/v1/autocomplete?text=424 Clay Avenu"
  ✔ [2-4] "/v1/autocomplete?text=424 Clay Avenue"
  ✔ [3-1] "/v1/autocomplete?focus.point.lat=42.102222&focus.point.lon=-75.911667&text=198 Baker St"
  ✔ [3-2] "/v1/autocomplete?focus.point.lat=42.102222&focus.point.lon=-75.911667&text=198 Baker Str"
  ✔ [3-3] "/v1/autocomplete?focus.point.lat=42.102222&focus.point.lon=-75.911667&text=198 Baker Stre"
  ✔ [3-4] "/v1/autocomplete?focus.point.lat=42.102222&focus.point.lon=-75.911667&text=198 Baker Street"
  ✔ [4-1] "/v1/autocomplete?text=451 Fog Hill"
  ✔ [4-2] "/v1/autocomplete?text=451 Fog Hill rd"
  ✔ [4-3] "/v1/autocomplete?text=451 Fog Hill Road"
  ✘ regression [9] "/v1/autocomplete?text=190 dean street brooklyn": score 0 out of 1
  diff:
    label
      expected: 190 Dean Street, Brooklyn, New York, NY, USA
      actual:   190 Dean Street, Brooklyn, NY, USA

autocomplete
  ✘ regression [1] "/v1/autocomplete?text=DiDi Dumpling": score 0 out of 1
  diff:
    label
      expected: DiDi Dumpling, Manhattan, New York, NY, USA
      actual:   DiDi Dumpling, New York, NY, USA

poi vs. admin areas balance
  ✔ [1] "/v1/autocomplete?focus.point.lat=40.743987&focus.point.lon=-73.990417&text=starbucks"
  ✔ [2] "/v1/search?focus.point.lat=40.743987&focus.point.lon=-73.990417&text=starbucks"
  ✔ [3] "/v1/autocomplete?focus.point.lat=40.743987&focus.point.lon=-73.990417&text=mcdonalds"
  ✔ [4] "/v1/search?focus.point.lat=40.743987&focus.point.lon=-73.990417&text=mcdonalds"
  ✔ [5] "/v1/autocomplete?focus.point.lat=40.74686681162143&focus.point.lon=-73.98983001708986&text=katzs deli"

Brooklyn
  ✔ [1] "/v1/search?boundary.country=USA&text=brooklyn"
  ✔ [2] "/v1/search?layers=borough&boundary.country=USA&text=brooklyn"
  ✔ [3-autocomplete] "/v1/autocomplete?boundary.country=USA&text=brooklyn"
  ✔ [4] "/v1/search?text=brooklyn"
  ✔ [5] "/v1/search?layers=borough&text=brooklyn"
  ✔ [6-autocomplete] "/v1/autocomplete?text=brooklyn"

confidence score
  ✔ [1] "/v1/search?text=1 West 72nd St, New York, NY, 10023"

encoding, or fun with diacritics
  ✔ [1] "/v1/search?text=mexico city"

exact matches
  ✔ [1] "/v1/search?text=100 20th street"
  ✔ [2] "/v1/search?text=40 20th street"
  ✔ [3] "/v1/search?text=120 42nd Street"
  ✔ [4] "/v1/search?text=358 41st street, ny"
  ✔ [6] "/v1/search?text=1359 54 street, ny"
  ✔ [7] "/v1/search?text=310 7 street, ny"
  ✔ [8] "/v1/search?text=921 83 street, ny"
  ✔ [9] "/v1/search?text=518 3 street, ny"

fallback queries
  ✘ regression [3] "/v1/search?text=101 california, san francisco": score 4 out of 5
  diff:
    street
      expected: California
      actual:   California Street

International addresses
  ✔ [6] "/v1/search?boundary.country=CAN&text=490 Sussex Drive Ottawa K1N 1G8"
  ✘ regression [8] "/v1/search?boundary.country=MEX&text=Prolongación Ave. Alfonso Reyes #150 Col. Valle del Poniente Santa Catarina 66196": score 2 out of 3
  diff:
    locality
      expected: Ciudad Santa Catarina
      actual:   Santa Catarina
  ✘ [10] "/v1/search?text=Av. Juárez, Guadalajara, México": score 2 out of 3
  diff:
    name
      expected: AVENIDA JUÁREZ
      actual:   GUADALAJARA
  ✔ [12] "/v1/search?boundary.country=CAN&text=615 MacLeod Trail S.E. 10th Floor Calgary T2G 4T8"
  ✘ regression [14] "/v1/search?boundary.country=MEX&text=Paseo de la Reforma Colonia Cuauhtemoc Mexico D.F. 06500": score 2 out of 4
  diff:
    name
      expected: PASEO DE LA REFORMA
      actual:   MEXICO CITY
    layer
      expected: street
      actual:   locality
  ✘ [15] "/v1/search?boundary.country=MEX&text=Progreso 175 Col. Americana Guadalajara 44160": score 2 out of 3
  diff:
    name
      expected: PROGRESO 175
      actual:   AMERICANA

labels
  ✔ [1] "/v1/search?text=San Francisco, San Francisco County, CA"
  ✘ regression [2] "/v1/search?text=30 West 26th Street, Manhattan, NY": score 0 out of 1
  diff:
    label
      expected: 30 West 26th Street, Manhattan, New York, NY, USA
      actual:   30 West 26th Street, New York, NY, USA
  ✘ regression [4] "/v1/search?text=California, CA": score 0 out of 1
  diff:
    label
      expected: California, USA
      actual:   California, US, El Salvador
  ✘ [15] "/v1/search?sources=geonames&layers=localadmin&text=city of new york": Unexpected status code 400
  ✔ [24] "/v1/search?text=national air and space museum, washington dc"

landmarks
  ✘ [14] "/v1/search?text=statue of liberty": score 0 out of 1
  diff:
    label
      expected: Statue of Liberty, Manhattan, NY, USA
      actual:   Statue of Liberty Museum, New York, NY, USA

missing postcodes
  ✔ [1] "/v1/search?text=267 e 10th st, new york, ny 10010"
  ✘ [2] "/v1/search?text=440 S Combee Rd, Lakeland, FL 33801": score 8 out of 9
  diff:
    locality
      expected: Crystal Lake
      actual:   Lakeland

GET /place
  ✘ regression [1] "/v1/place?ids=openstreetmap:venue:node:2562438641": no results returned
  ✘ regression [2] "/v1/place?ids=osm:venue:node:2562438641": no results returned
  ✘ regression [3] "/v1/place?ids=openstreetmap:venue:way:5013364": no results returned
  ✘ regression [4] "/v1/place?ids=whosonfirst:region:85688525,whosonfirst:locality:101928027 (2 expectations)": score 7 out of 11
  diff:
    gid
      expected: whosonfirst:locality:101928027
      actual:   whosonfirst:region:85688525
    layer
      expected: locality
      actual:   region
    id
      expected: 101928027
      actual:   85688525
    name
      expected: Cape Town
      actual:   North Dakota
  ✘ regression [5] "/v1/place?ids=openstreetmap:address:node:2420772655": no results returned
  ✘ [6] "/v1/place?ids=whosonfirst:locality:123": no results returned

Placeholder alt names
  ✔ [1] "/v1/search?lang=es&text=553 s main st red lion pa usa"
  ✔ [7] "/v1/search/structured?address=285 Fulton St&borough=Manhattan&locality=New York&region=NY&postalcode=10007&lang=pl"
  ✘ regression [8] "/v1/reverse?lang=pt&point.lat=40.712748&point.lon=-74.013296": score 3 out of 4
  diff:
    priorityThresh is 2 but found at position 7
  ✔ [9] "/v1/reverse?layers=coarse&lang=pt&point.lat=40.712748&point.lon=-74.013296"
  ✔ improvement [10] "/v1/nearby?lang=pt&point.lat=40.712748&point.lon=-74.013296"
  ✔ [12] "/v1/search?boundary.circle.lat=31.030974&boundary.circle.lon=-98.326329&boundary.circle.radius=125&lang=es&text=beverly hills"
  ✔ [13] "/v1/search?boundary.rect.min_lon=-106.645646&boundary.rect.min_lat=25.837164&boundary.rect.max_lon=-93.508039&boundary.rect.max_lat=36.500704&lang=sr&text=beverly hills"
  ✔ [14] "/v1/search?focus.point.lon=-82.985586&focus.point.lat=39.98072&lang=ru&text=springfield"
  ✔ [18] "/v1/search?boundary.country=US&lang=bg&text=damascus"
  ✔ [23] "/v1/search?text=30 west 26th street, nueva york, ny"
  ✔ [24] "/v1/search?text=1220 calle de lago Сокорро Nouveau-Mexique"
  ✔ [25] "/v1/search?text=280 Bloor Street West টরোন্টো オンタリオ州 کانادا"

General Placeholder requests
  ✔ [1] "/v1/search?text=port saint lucie, fl"
  ✔ [2] "/v1/search?text=port st. lucie, fl"
  ✔ [3] "/v1/search?text=Sault Sainte Marie, Ontario"
  ✔ [4] "/v1/search?text=Sault Ste Marie, Ontario"
  ✘ [5] "/v1/search?text=Fort Hood, TX": score 0 out of 3
  diff:
    gid
      expected: whosonfirst:locality:101726923
      actual:   whosonfirst:county:102087383
    confidence
      expected: 0.6
      actual:   0.4
    name
      expected: Fort Hood
      actual:   Hood County
  ✘ [6] "/v1/search?text=Ft Hood, TX": score 0 out of 3
  diff:
    gid
      expected: whosonfirst:locality:101726923
      actual:   whosonfirst:county:102087383
    confidence
      expected: 0.6
      actual:   0.4
    name
      expected: Fort Hood
      actual:   Hood County
  ✔ [7] "/v1/search?text=Mount Pleasant, SC"
  ✔ [8] "/v1/search?text=Mt Pleasant, SC"
  ✔ [9] "/v1/search?text=Carmel-by-the-Sea, CA"
  ✔ [10] "/v1/search?text=Carmel by the Sea, CA"
  ✔ [13] "/v1/search?text=City of Socorro, NM"
  ✔ [14] "/v1/search?text=L.A. C.A."
  ✔ [23] "/v1/search?layers=county&text=Socorro, NM"
  ✔ [26] "/v1/search?boundary.rect.min_lat=31.346514&boundary.rect.max_lat=36.987116&boundary.rect.min_lon=-109.049381&boundary.rect.max_lon=-103.006901&text=Las Vegas"

Placeholder should obey geometric filters
  ✘ regression [2] "/v1/search?boundary.country=CA&text=2000 main st, berlin": score 1 out of 2
  diff:
    name
      expected: 2000 Main St
      actual:   2000 Main Street

Sorting rules for admin-only placeholder responses
  ✔ [402] "/v1/search?text=San Francisco"
  ✔ [600] "/v1/search?text=New Mexico"
  ✔ [601] "/v1/search?text=British Columbia"
  ✔ [602] "/v1/search?text=California"
  ✔ [603] "/v1/search?text=Ontario, CA"
  ✔ [700] "/v1/search?text=Manhattan"
  ✔ improvement [701] "/v1/search?text=Queens"
  ✔ [800] "/v1/search?text=Chelsea"
  ✔ [900] "/v1/search?text=Socorro, NM"
  ✔ [1100] "/v1/search?text=Maui"
  ✔ [1200] "/v1/search?text=San Fernando Valley"
  ✔ [1201] "/v1/search?text=Mission District"
  ✔ [1300] "/v1/search?text=Flatiron District"
  ✘ regression [1301] "/v1/search?text=Brockley": score 1 out of 2
  diff:
    gid
      expected: whosonfirst:neighbourhood:85860571
      actual:   whosonfirst:locality:1292928207
  ✔ [1302] "/v1/search?text=Haight Ashbury"
  ✔ [1400] "/v1/search?boundary.country=USA&text=Felton (3 expectations)"

Result properties
  ✔ [1] "/v1/search?focus.point.lon=-73.944108&focus.point.lat=40.815263&layers=venue&text=IHOP"
  ✘ regression [2] "/v1/autocomplete?sources=wof&text=London": score 2 out of 5
  diff:
    gid
      expected: whosonfirst:locality:101750367
      actual:   whosonfirst:locality:101735809
    id
      expected: 101750367
      actual:   101735809
    source_id
      expected: 101750367
      actual:   101735809
  ✔ [3] "/v1/reverse?layers=venue&point.lat=40.744022&point.lon=-73.989231"
  ✘ regression [4] "/v1/place?ids=openstreetmap:venue:node:4509430011": no results returned

quattroshapes popularity
  ✔ [1] "/v1/search?layers=neighbourhood&text=chelsea"
  ✔ [1-1] "/v1/search?layers=coarse&text=chelsea, ny"
  ✔ [2] "/v1/search?layers=neighbourhood&text=williamsburg"
  ✔ [2-1] "/v1/search?layers=neighbourhood&text=williamsburg, ny"
  ✔ [3] "/v1/search?layers=coarse&text=ridgewood"
  ✔ [3-1] "/v1/search?layers=coarse&text=ridgewood, ny"

reverse non-coarse
  ✔ [1] "/v1/reverse?point.lat=40.038123&point.lon=-76.304096"
  ✘ regression [2] "/v1/reverse?point.lat=34.35755&point.lon=-119.126008": score 0 out of 2
  diff:
    layer
      expected: street
      actual:   county
    unexpected property found from {"layer":"county"}
  ✔ [3] "/v1/reverse?point.lat=39.898111&point.lon=-76.607958"
  ✔ [4] "/v1/reverse?layers=venue,locality&point.lat=39.898111&point.lon=-76.607958"
  ✔ [5] "/v1/reverse?layers=address,locality&point.lat=39.898111&point.lon=-76.607958"
  ✘ regression [6] "/v1/reverse?layers=street,locality&point.lat=39.898111&point.lon=-76.607958": score 0 out of 2
  diff:
    layer
      expected: street
      actual:   locality
    unexpected property found from {"layer":"locality"}

reverse coarse
  ✔ [1] "/v1/reverse?layers=coarse&point.lat=40.74426&point.lon=-73.9904"
  ✘ regression [2] "/v1/reverse?layers=coarse&point.lat=40.042013&point.lon=-76.301163": score 5 out of 7
  diff:
    layer
      expected: neighbourhood
      actual:   locality
    neighbourhood
      expected: Rossmere
      actual:   
  ✔ [5] "/v1/reverse?layers=coarse&point.lat=33.214722&point.lon=-106.363415"
  ✔ [10] "/v1/reverse?layers=neighbourhood&point.lat=40.74426&point.lon=-73.9904"
  ✔ [11] "/v1/reverse?layers=borough&point.lat=40.74426&point.lon=-73.9904"
  ✘ [12] "/v1/reverse?layers=locality&point.lat=40.74426&point.lon=-73.9904": score 4 out of 5
  diff:
    county
      expected: New York County
      actual:   Kings County
  ✔ [13] "/v1/reverse?layers=localadmin&point.lat=40.042013&point.lon=-76.301163"
  ✔ [14] "/v1/reverse?layers=county&point.lat=40.74426&point.lon=-73.9904"
  ✔ [16] "/v1/reverse?layers=region&point.lat=40.74426&point.lon=-73.9904"
  ✔ [19] "/v1/reverse?layers=country&point.lat=40.74426&point.lon=-73.9904"
  ✔ [20] "/v1/reverse?point.lat=34.119423&point.lon=-106.284351"
  ✘ regression [27] "/v1/reverse?layers=marinearea&point.lat=38.048214&point.lon=-76.294635": Unexpected status code 400

reverse coordinate wrapping
  ✔ [3] "/v1/reverse?point.lat=40.72956780913899&point.lon=-73.99463653564453"
  ✔ [4] "/v1/reverse?point.lat=40.72956780913899&point.lon=286.00536346435547"
  ✔ [5] "/v1/reverse?point.lat=40.72956780913899&point.lon=-433.99463653564453"
  ✔ [5] "/v1/reverse?point.lat=40.72956780913899&point.lon=-793.9946365356445"
  ✔ [6] "/v1/reverse?point.lat=40.744131&point.lon=286.009575"
  ✔ [7] "/v1/reverse?point.lat=40.744131&point.lon=-433.990425"
  ✔ [8] "/v1/reverse?point.lat=139.255869&point.lon=106.009575"

San Francisco
  ✔ [1] "/v1/search?boundary.country=USA&text=san francisco"
  ✔ [2] "/v1/search?layers=locality&boundary.country=USA&text=san francisco"
  ✔ [3] "/v1/search?layers=locality,localadmin&boundary.country=USA&text=san francisco"
  ✔ [4-autocomplete] "/v1/autocomplete?boundary.country=USA&text=san francisco"
  ✔ [5] "/v1/search?text=san francisco"
  ✔ [6] "/v1/search?layers=locality&text=san francisco"
  ✔ [7] "/v1/search?layers=locality,localadmin&text=san francisco"
  ✔ [8-autocomplete] "/v1/autocomplete?text=san francisco"

search
  ✘ regression [1] "/v1/search?text=brooklyn": score 0 out of 1
  diff:
    label
      expected: Brooklyn, New York, NY, USA
      actual:   Brooklyn, NY, USA
  ✘ regression [2] "/v1/search?text=brooklyn, ny": score 0 out of 1
  diff:
    label
      expected: Brooklyn, New York, NY, USA
      actual:   Brooklyn, NY, USA
  ✔ [4] "/v1/search?text=philadelphia"
  ✔ [4-1] "/v1/search?text=philadelphia, pa"
  ✘ regression [5] "/v1/search?text=new york, new york (2 expectations)": score 3 out of 5
  diff:
    gid
      expected: whosonfirst:county:102081863
      actual:   whosonfirst:locality:85977539
    name
      expected: New York County
      actual:   New York
  ✔ [5:2] "/v1/search?text=new york city"
  ✔ [5:3] "/v1/search?text=new york city, usa"
  ✔ [6] "/v1/search?text=130 dean street brooklyn, ny"
  ✔ [7] "/v1/search?text=billerica"
  ✔ [8] "/v1/search?text=billerica, ma"
  ✔ [9] "/v1/search?text=15 call street billerica, ma"
  ✘ [1425586777012:0] "/v1/search?text=union sq": score 4 out of 7
  diff:
    name
      expected: Union Square
      actual:   Union Square Dining
    county
      expected: San Francisco County
      actual:   Santa Clara County
    label
      expected: Union Square, San Francisco County, CA, USA
      actual:   Union Square Dining, San Jose, CA, USA
  ✔ [1425586777012:2] "/v1/search?text=portland"
  ✔ [1425586777012:2-1] "/v1/search?text=portland, oregon"
  ✔ [10] "/v1/search?text=chelsea, new york"
  ✔ [11] "/v1/search?text=soho, new york"
  ✔ [15] "/v1/search?text=101 saint marks pl, new york"
  ✘ [18] "/v1/search?text=newark airport": score 7 out of 10
  diff:
    name
      expected: Newark Liberty International Airport
      actual:   Marriott Newark Airport
    neighbourhood
      expected: Dayton
      actual:   
    label
      expected: Newark Liberty International Airport, Newark, NJ, USA
      actual:   Marriott Newark Airport, Newark, NJ, USA
  ✔ [19] "/v1/search?focus.point.lat=40.744243&focus.point.lon=-73.990342&layers=address&sources=openstreetmap&text=30 West 26th Street"
  ✘ regression [20] "/v1/search?boundary.country=FRA&text=statue of liberty": no results returned
  ✘ regression [1426636804303:51] "/v1/search?text=4th and King san francisco": score 1 out of 2
  diff:
    name
      expected: San Francisco 4th & King Street Station
      actual:   4th & King
  ✘ [24] "/v1/search?focus.point.lat=37.743618&focus.point.lon=-122.426117&text=zoo": score 2 out of 3
  diff:
    priorityThresh is 1 but found at position 9
  ✔ [25] "/v1/search?size=1&text=82 Whitney Ave, Hamilton, Ontario"

searchAbbvs
  ✘ [1] "/v1/search?text=412 Saint Patrick St, donaldsonville, la": score 2 out of 5
  diff:
    layer
      expected: address
      actual:   locality
    housenumber
      expected: 412
      actual:   
    street
      expected: St Patrick St
      actual:   
  ✘ regression [2] "/v1/search?text=412 St Patrick St, donaldsonville, la": score 2 out of 5
  diff:
    layer
      expected: address
      actual:   locality
    housenumber
      expected: 412
      actual:   
    street
      expected: St Patrick St
      actual:   

search city/country
  ✔ [20] "/v1/search?text=truth or consequences, usa"
  ✔ [21] "/v1/search?text=toronto, ca"
  ✔ [22] "/v1/search?text=yakima, us"
  ✘ regression [31] "/v1/search?text=seattle, canada": score 0 out of 3
  diff:
    layer
      expected: country
      actual:   locality
    country
      expected: Canada
      actual:   United States
    country_a
      expected: CAN
      actual:   USA

search coarse
  ✘ regression [1] "/v1/search?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=brooklyn": score 0 out of 1
  diff:
    label
      expected: Brooklyn, New York, NY, USA
      actual:   Brooklyn, NY, USA
  ✔ [2] "/v1/search?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=new york"

search geodisambiguation
  ✔ [1] "/v1/search?sources=wof&text=Aliquippa, PA"

ISO2->ISO3 internal conversion
  ✘ regression [1] "/v1/search?text=1 Main St, Richlandtown, PA, US": score 4 out of 7
  diff:
    layer
      expected: address
      actual:   locality
    street
      expected: Main St
      actual:   
    housenumber
      expected: 1
      actual:   
  ✘ regression [4] "/v1/search?text=22 Lloyd George Ave, Toronto Ontario CA": score 4 out of 5
  diff:
    street
      expected: Lloyd George Ave
      actual:   Lloyd George Avenue
  ✔ [5] "/v1/search?text=327 Rincon de Romos, Aguascalientes, MX"

search layers
  ✔ [1] "/v1/search?layers=venue&text=london"

search_poi
  ✘ [searchpoi-1] "/v1/search?text=Target Eureka CA": score 4 out of 5
  diff:
    priorityThresh is 1 but found at position 2
  ✘ [searchpoi-2] "/v1/search?text=Target Eureka California": score 4 out of 5
  diff:
    priorityThresh is 1 but found at position 3
  ✘ [searchpoi-3] "/v1/search?focus.point.lat=40.801944&focus.point.lon=-124.163611&text=Target": score 3 out of 4
  diff:
    locality
      expected: Eureka
      actual:   Torrance

search_postalcodes
  ✔ improvement [searchpostal-1] "/v1/search?text=90210"
  ✔ improvement [searchpostal-4] "/v1/search?text=not-an-address, 90210"

search street centroids
  ✘ regression [3] "/v1/search?sources=osm&layers=street&text=w 26 st, new york": score 1 out of 6
  diff:
    name
      expected: West 26th Street
      actual:   New York
    street
      expected: West 26th Street
      actual:   
    layer
      expected: street
      actual:   locality
    source
      expected: openstreetmap
      actual:   whosonfirst
    'New York, NY, USA' is not close enough: distance is 7550m but should be under 1500m

structured address parsing
  ✔ [1] "/v1/search/structured?address=1396 Dual 40 Hwy&locality=Hagerstown&region=MD&postalcode=21740&country=United States"
  ✘ [1.1] "/v1/search/structured?address=1396 Dual 40 Hwy&locality=Hagerstown&region=MD&postalcode=21740&country=United States": score 5 out of 8
  diff:
    layer
      expected: address
      actual:   postalcode
    housenumber
      expected: 1396
      actual:   
    street
      expected: Dual Highway
      actual:   

structured geocoding
  ✘ regression [1] "/v1/search/structured?address=1090 N Charlotte Street&locality=Lancaster&region=PA": score 10 out of 11
  diff:
    priorityThresh is 1 but found at position 2
  ✘ regression [2] "/v1/search/structured?address=North Charlotte Street&locality=Lancaster&region=PA": score 6 out of 9
  diff:
    layer
      expected: street
      actual:   locality
    name
      expected: North Charlotte Street
      actual:   Lancaster
    street
      expected: North Charlotte Street
      actual:   
  ✘ regression [3] "/v1/search/structured?address=12200000 Calle de Lago&locality=Socorro&region=NM": score 6 out of 9
  diff:
    layer
      expected: street
      actual:   locality
    name
      expected: Calle de Lago
      actual:   Socorro
    street
      expected: Calle de Lago
      actual:   
  ✔ [4] "/v1/search/structured?address=1090 Trescube Boulevard&locality=Lancaster&region=PA"
  ✘ regression [5] "/v1/search/structured?address=1338 Kobbe Ave&locality=San Francisco&region=CA": score 6 out of 7
  diff:
    name
      expected: 1338 Kobbe Ave
      actual:   1338 Kobbe Avenue
  ✘ regression [7] "/v1/search/structured?address=5 russian hill pl&locality=San Francisco&region=CA": score 6 out of 7
  diff:
    name
      expected: 5 Russian Hill Pl
      actual:   5 Russian Hill Place
  ✘ regression [8] "/v1/search/structured?address=1090 n charlotte st": score 1 out of 2
  diff:
    name
      expected: 1090 N Charlotte St
      actual:   1090 North Charlotte Street
  ✔ [100] "/v1/search/structured?neighbourhood=Dumbo&borough=Brooklyn"
  ✔ [101] "/v1/search/structured?neighbourhood=Chelsea"
  ✔ [102] "/v1/search/structured?neighbourhood=Noho"
  ✘ regression [200] "/v1/search/structured?locality=Manhattan (2 expectations)": score 18 out of 19
  diff:
    priorityThresh is 3 but found at position 5
  ✔ [201] "/v1/search/structured?borough=Manhattan&locality=New York&region=NY"
  ✔ [300] "/v1/search/structured?locality=Socorro&region=NM"
  ✔ [301] "/v1/search/structured?locality=New York&region=NY&country=USA"
  ✘ regression [401] "/v1/search/structured?sources=wof&locality=Zumbehl": score 3 out of 5
  diff:
    layer
      expected: localadmin
      actual:   locality
    name
      expected: Zumbehl Township
      actual:   Zumbehl
  ✘ regression [402] "/v1/search/structured?sources=wof&locality=Aastad&country=United States": score 5 out of 7
  diff:
    layer
      expected: localadmin
      actual:   locality
    name
      expected: Aastad Township
      actual:   Aastad
  ✘ regression [403] "/v1/search/structured?sources=wof&locality=Bloominggrove&region=Ohio": score 5 out of 7
  diff:
    layer
      expected: localadmin
      actual:   locality
    name
      expected: Bloominggrove Township
      actual:   Bloominggrove
  ✔ improvement [500] "/v1/search/structured?county=Lancaster (2 expectations)"
  ✔ [501] "/v1/search/structured?county=Lancaster&region=PA"
  ✔ [700] "/v1/search/structured?region=PA"
  ✔ [701] "/v1/search/structured?region=New Mexico"
  ✔ [1000] "/v1/search/structured?country=USA"
  ✔ [1001] "/v1/search/structured?country=US"
  ✔ [1002] "/v1/search/structured?country=United States"
  ✘ regression [1100] "/v1/search/structured?venue=police&address=1090 N Charlotte Street&locality=Lancaster&region=PA": score 3 out of 4
  diff:
    name
      expected: Lancaster Bureau of Police
      actual:   Lancaster City Bureau of Police
  ✘ regression [1101] "/v1/search/structured?venue=nightlife&address=1090 N Charlotte Street&locality=Lancaster&region=PA": score 4 out of 5
  diff:
    priorityThresh is 1 but found at position 2
  ✘ regression [1102] "/v1/search/structured?venue=police&address=N Charlotte Street&locality=Lancaster&region=PA": score 3 out of 4
  diff:
    name
      expected: Lancaster Bureau of Police
      actual:   Lancaster City Bureau of Police
  ✘ regression [1103] "/v1/search/structured?venue=unsupported venue&address=1090 N Charlotte Street&locality=Lancaster&region=PA": score 5 out of 6
  diff:
    priorityThresh is 1 but found at position 2
  ✘ regression [1104] "/v1/search/structured?venue=unsupported venue&address=N Charlotte Street&locality=Lancaster&region=PA": score 2 out of 4
  diff:
    layer
      expected: street
      actual:   locality
    street
      expected: North Charlotte Street
      actual:   
  ✔ [1105] "/v1/search/structured?venue=unsupported venue&locality=Lancaster&region=PA"
  ✘ [1202] "/v1/search/structured?address=412 saint patrick st&locality=donaldsonville": score 2 out of 5
  diff:
    layer
      expected: address
      actual:   locality
    housenumber
      expected: 412
      actual:   
    street
      expected: St Patrick St
      actual:   

structured_postalcodes
  ✔ improvement [structuredpostal-1] "/v1/search/structured?postalcode=90210"
  ✔ improvement [structuredpostal-3] "/v1/search/structured?address=not-an-address&postalcode=90210"
  ✔ [structuredpostal-4] "/v1/search/structured?postalcode=83278&country=United States"

tizen sdk: geocode
  ✔ [maps_service_geocode_inside_area] "/v1/search?boundary.rect.min_lon=-80.947265625&boundary.rect.min_lat=41.343824581185686&boundary.rect.max_lat=42.407234661551875&boundary.rect.max_lon=-78.90380859375&text=30 w 26th street"
  ✔ [maps_service_geocode_by_structured_address] "/v1/search/structured?address=30 w 26th street&region=new york&postalcode=10010"
  ✔ [maps_service_reverse_geocode] "/v1/reverse?layers=address&point.lat=40.744022&point.lon=-73.989231"

tizen sdk: places
  ✘ [maps_service_search_place] "/v1/nearby?boundary.circle.radius=10&categories=food:pizza&point.lat=40.235771&point.lon=-74.937712": score 1 out of 2
  diff:
    name
      expected: Jule's Thin Crust
      actual:   Jule's Thin Crust Pizza
  ✔ improvement [maps_service_search_place_by_area] "/v1/nearby?boundary.rect.min_lat=40.235974&boundary.rect.min_lon=-74.937158&boundary.rect.max_lat=40.228095&boundary.rect.max_lon=-74.944906&categories=finance&point.lat=40.235771&point.lon=-74.937712"
  ✔ improvement [maps_service_search_place_by_list] "/v1/nearby?boundary.rect.min_lat=40.235974&boundary.rect.min_lon=-74.937158&boundary.rect.max_lat=40.228095&boundary.rect.max_lon=-74.944906&categories=finance&point.lat=40.235771&point.lon=-74.937712"
  ✘ [maps_service_get_place_details] "/v1/place?ids=openstreetmap:venue:node:223078192": no results returned

university
  ✘ regression [0] "/v1/search?boundary.country=USA&text=Teachers College, Columbia University": score 0 out of 2
  diff:
    name
      expected: Teachers College
      actual:   District of Columbia Teachers College
    region
      expected: New York
      actual:   District of Columbia
  ✘ [1] "/v1/search?boundary.country=USA&text=Washington University in St. Louis": score 1 out of 2
  diff:
    name
      expected: Washington University
      actual:   Washington University in St. Louis
  ✔ [2] "/v1/search?boundary.country=USA&text=Miami University"
  ✘ regression [3] "/v1/search?text=University of Hawaii at Hilo": score 1 out of 2
  diff:
    name
      expected: University of Hawai‘i at Hilo
      actual:   University of Hawaii - Hilo Main Campus
  ✘ regression [4] "/v1/search?text=Universidad de Guadalajara,  Guadalajara, Jalisco, México": score 3 out of 4
  diff:
    name
      expected: CUCEA (Universidad de Guadalajara)
      actual:   Preparatoria 10, Universidad de Guadalajara
  ✘ regression [5] "/v1/search?text=Universidad Autonoma de Ciudad Juarez": score 0 out of 2
  diff:
    name
      expected: Universidad Autónoma de Ciudad Juárez
      actual:   Rectoría Universidad Autónoma de Tamaulipas
    region
      expected: Chihuahua
      actual:   Tamaulipas
  ✔ [6] "/v1/search?text=Oxford College, oxford, ga"
  ✘ regression [9] "/v1/search?text=Ohio State University": score 1 out of 2
  diff:
    localadmin
      expected: Bath
      actual:   Bath Township
  ✔ [10] "/v1/search?text=Antioch University Seattle"
  ✔ [11] "/v1/search?text=Union college, kentucky"
  ✔ improvement [12] "/v1/search?text=Union college kentucky"

washington dc
  ✘ regression [1] "/v1/search?text=1705 P St NW, Washington, dc": score 4 out of 5
  diff:
    street
      expected: P Street Northwest
      actual:   P Street NW
  ✘ regression [2] "/v1/search?text=1705 P St NW, Washington, district of columbia": score 4 out of 5
  diff:
    street
      expected: P Street Northwest
      actual:   P Street NW
  ✔ [3] "/v1/search?text=busboys and poets, washington dc"
  ✘ regression [4] "/v1/search?text=641 S Street NW, washington, district of columbia": score 4 out of 5
  diff:
    street
      expected: S Street Northwest
      actual:   S Street NW

Who's on First counties
  ✔ improvement [2] "/v1/search?sources=wof&text=Potter, PA"
  ✔ [3] "/v1/search?sources=wof&text=Maui, Hawaii"

Who's on First countries
  ✔ [1] "/v1/search?sources=wof&text=United States"
  ✔ [2] "/v1/search?sources=wof&text=Canada"
  ✔ [6] "/v1/search?sources=wof&text=USA"

Who's on First localadmins
  ✔ [2] "/v1/search?sources=wof&layers=localadmin&text=Norwood Young America, Minnesota, US"
  ✘ regression [3] "/v1/search?sources=wof&text=Zickrick, SD": score 4 out of 6
  diff:
    layer
      expected: localadmin
      actual:   locality
    name
      expected: Zickrick Township
      actual:   Zickrick

Who's on First localities
  ✔ [1] "/v1/search?sources=wof&text=New York, New York"
  ✔ [2] "/v1/search?sources=wof&text=Socorro, New Mexico"
  ✔ [3] "/v1/search?sources=wof&text=Truth or Consequences, NM"
  ✔ [5] "/v1/search?sources=wof&text=Carmel-by-the-Sea, California"
  ✔ [6] "/v1/search?sources=wof&layers=locality&text=Springfield"

Who's on First neighbourhoods
  ✔ [1] "/v1/search?sources=wof&text=Crown Heights, Brooklyn, New York"
  ✔ [2] "/v1/search?sources=wof&text=Reparto Vista Alegre, Cuba"
  ✔ [3] "/v1/search?sources=wof&text=Nuevo Cristóbal, Panama"

Who's on First regions
  ✔ [1] "/v1/search?sources=wof&text=Pennsylvania"
  ✔ [2] "/v1/search?sources=wof&text=New Mexico"
  ✔ [3] "/v1/search?sources=wof&text=Newfoundland and Labrador"
  ✔ [4] "/v1/search?sources=wof&text=Prince Edward Island"

Aggregate test results
Pass: 233
Improvements: 18
Expected Failures: 28
Placeholders: 0
Regressions: 84
Total tests: 363
Took 10415ms
Test success rate 76.86%

FATAL ERROR: 84 regression(s) detected.