Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
a909a47
upgrade Puma
wkdewey May 19, 2022
43bb2ba
add specific version of puma to avoid security warnings
wkdewey May 19, 2022
cd38c46
update to later version of puma
wkdewey May 25, 2022
866fae1
another round of gem updates
wkdewey May 25, 2022
03b0030
add facet for matching nested facet
wkdewey May 24, 2022
7ce8727
add filter for matching nested facet
wkdewey May 24, 2022
5073460
change split character, add missing comma
wkdewey May 24, 2022
a4a0c71
parse the array for matching nested fields
wkdewey May 26, 2022
eb5dd1b
change how compound facet name is parsed
wkdewey May 26, 2022
896177b
use facet name as agg name
wkdewey May 26, 2022
06be739
change query to filter
wkdewey May 26, 2022
89e5063
fix nested filter aggregation so it doesn't cause 400 error
wkdewey May 27, 2022
5c43688
check for deeper nesting of buckets
wkdewey May 31, 2022
2c8b64a
Change separator
wkdewey Jun 1, 2022
b2026a3
Fix parsing and query for filter matching
wkdewey Jun 1, 2022
f44188c
rewrite filtered aggregation to be either nested or not
wkdewey Jun 2, 2022
d81c070
filtering on a single item can either be nested or not
wkdewey Jun 2, 2022
322504d
update config for server
wkdewey Sep 26, 2022
15c3c90
revise query to match both the facet and the filter
wkdewey Oct 19, 2022
5977f2b
use reverse nested agg for correct item count
wkdewey Oct 20, 2022
b08f408
used doc_count from reverse nested if it exists
wkdewey Oct 20, 2022
f399f9a
change key for new elasticsearch version
wkdewey Oct 21, 2022
72abce3
change order query to avoid deprecated '_term'
wkdewey Oct 24, 2022
db64319
gitignore master key
wkdewey Oct 26, 2022
59835ed
add basic auth to elasticsearch requests
wkdewey Oct 26, 2022
419bc8b
raise number of results per facet
wkdewey Oct 28, 2022
bd92c49
use facet_limit instead of facet_num to match Orchid
wkdewey Oct 31, 2022
f0653a9
revert, will set in Orchid
wkdewey Oct 31, 2022
19f789f
change facet_num to facet_limit
wkdewey Nov 1, 2022
e922ed7
update nested facets documentation
wkdewey Nov 10, 2022
5eb6476
add links to more detailed documentation
wkdewey Nov 10, 2022
4e2638e
clarify
wkdewey Nov 10, 2022
90281ed
update gems
wkdewey Dec 19, 2022
e52ba03
use reverse nested on simple nested aggregations
wkdewey May 25, 2023
6e92258
fix elasticsearch errors
wkdewey May 26, 2023
f8cc064
titleize bucket values because ES automatically lowercases them
wkdewey Jul 21, 2023
013df32
display total search hits (above 10,000)
wkdewey May 15, 2023
f72c307
fix display of nested field facets
wkdewey Aug 6, 2024
f11a5d7
handle un-normalization of values in case of array rather than string…
wkdewey Aug 20, 2024
3319d19
change method for unnormalization of values to fix bugs
wkdewey Aug 22, 2024
2f7c085
update changelog and improve formatting
wkdewey Aug 23, 2024
e02ffee
fix YAML loading for new version of Ruby
wkdewey Aug 23, 2024
5717c9b
update Ruby and gems for new server
Sep 26, 2024
1097fb1
handle nested field facets in case of single hash rather than array
wkdewey Nov 4, 2024
88ab346
make sure facet key is returned if no source
wkdewey Nov 11, 2024
1239fac
fix de-normalization for non-nested fields
wkdewey Nov 26, 2024
39cf19f
sort facets properly by count when nested
wkdewey Dec 4, 2024
ae7f0a8
fix ES query for nested aggregations so that _source finds right item
wkdewey Dec 9, 2024
f5b61b4
don't try to match up non-string values
wkdewey Dec 16, 2024
9318336
add a comment for future bugfixing
wkdewey Dec 16, 2024
f0b8817
handle any response error from RestClient
wkdewey Jul 14, 2025
d6a704e
Bump Ruby version to 3.1.7
techgique Dec 20, 2025
46d3b0d
Remove unused code in ApplicationController
techgique Dec 20, 2025
b004587
Reformat post code; use auth if creds present
techgique Dec 20, 2025
a0ed47a
Update Gemfile and gems
techgique Dec 20, 2025
c42e264
Set defaults to 5.2, enable new 6.1 defaults
techgique Dec 20, 2025
e4e16f8
Fix logger, listen gem broken from last update
techgique Dec 20, 2025
8e6bdca
Add standard Rails dev and prod env
techgique Dec 20, 2025
30145d1
Augment Elasticsearch request error handling
techgique Dec 20, 2025
afaf758
search_item_req.rb reformatting, refactoring
techgique Dec 20, 2025
3d4842b
Refactor search_item_res.rb
techgique Dec 20, 2025
aac16c0
Update version, config example; review Changelog
techgique Dec 20, 2025
18fc4f5
Merge branch 'dev' into local_testing
techgique Dec 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,5 @@ bower.json
.byebug_history

.DS_Store

/config/master.key
2 changes: 1 addition & 1 deletion .ruby-gemset
Original file line number Diff line number Diff line change
@@ -1 +1 @@
api
api-v2
2 changes: 1 addition & 1 deletion .ruby-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
ruby-3.1.2
ruby-3.1.7
91 changes: 63 additions & 28 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,84 +27,120 @@ Markdown Spec](https://github.github.com/gfm/).

### Contributors
-->
## [2.0.0] - new nested bucket aggregation/query functionality for Habeas release
[Unreleased]: https://github.com/CDRH/api/compare/v1.0.4...dev

## [v2.0.0] - Nested bucket aggregation/query functionality
[v2.0.0]: https://github.com/CDRH/api/compare/v1.0.4...v2.0.0

### Added
- "api_version" added to all response "res" objects
- support for elasticsearch 8.5
- user/password basic authentication with ES 8.5, when querying the index or posting from Datura
- better support for nested fields
- support for nested bucket aggregations, matching a nested value on another nested value. `person.name[person.role#judge]` will return all names of persons where role="judge".
- "api_version" added to all response "res" objects

- `api_version` added to all response `res` objects
- Support for Elasticsearch 8.5+
- User/password basic authentication support when credentials present
- Better support for nested fields
- Support for nested bucket aggregations, matching a nested value on another
nested value. For example, `person.name[person.role#judge]` will return
all names of persons where `role="judge"`
- Updated documentation for new features
- `track total hits` option added to ES queries, to return counts of search
results higher than 10000

### Changed

- Gemset changed to `api-v2`
- Changes reflect new api schemas in Datura, which make heavy use of nested fields
- Added support for aggregating buckets by normalized keyword and returning
the "top_hits" first document result for a non-normalized display
- Changes response format of `facets` key

the `top_hits` first document result for a non-normalized display. Internal logic has been changed because of nested fields, this may cause subtle differences in how facet labels are displayed
- Changes response format of `facets` key. Not only is the response format
itself different, but there may be fewer facets returned since matching
normalized values are combined

From:
```

```json
"facets": {
"WILLA CATHER": 10,
"Willa Cather": 50
}
```

To:
```

```json
"facets": {
"willa cather": { "num" : 60, source: "Willa Cather" }
}
```
Not only is the response format itself different, but there may be fewer
facets returned since normalized values which match are combined
### Changed
- upgraded to Rails 6.1.7 and Ruby 3
- changes reflect new api schemas in Datura, which make heavy use of nested fields

### Migration
- in Datura repos config `private.yml` api to `"api_version": "2.0"` to take advantage of new bucket aggregation functionality (or `"api_version": "1.0"` for legacy repos that have not been updated for the new schema). Please note that a running API index can only use one ES index at a time, and each ES index is restricted to one version of the schema.
- Use Elasticsearch 8.5 or later
- If you are using ES with security enabled, you must configure credentials with Rails in the API repo. See https://guides.rubyonrails.org/v6.1/security.html. Configure the VSCode editor. Run `EDITOR="code --wait" rails credentials:edit` and add

- Add nested facets as described above, if desired
- Orchid apps that connect to the API should use `facet_limit` instead of `facet_num` in options
- In the config files of your Datura repos, (`private.yml` or `public.yml`, set
the api to `"api_version": "2.0"` to take advantage of new bucket aggregation
functionality (or `"api_version": "1.0"` for legacy repos that have not been
updated for the new schema). Please note that a running API index can only use
one ES index at a time, and each ES index is restricted to one version of the
schema. See [new schema (2.0)
documentation](https://github.com/CDRH/datura/docs/schema_v2.md).
- Connect to Elasticsearch 8.5 or later
- If you are using ES with security enabled, you must configure credentials
with Rails in the API repo. See
https://guides.rubyonrails.org/v6.1/security.html. To configure with VSCode
editor run `EDITOR="code --wait" rails credentials:edit` and add to the
secrets file and then close the window to save.
Do not commit `config/master.key` (it should be in `.gitignore`)

```
elasticsearch:
user: username
password: *****
```
to the secrets file and then close the window to save. Do not commit `config/master.key` (it should be in `gitignore`)
- Orchid apps that connect to the API should use `facet_limit` instead of `facet_num` in options.
- Add nested facets as described above, if desired
- Orchid apps should update the local version of `config/initializers/config.rb` to match latest version of Orchid (see `lib/generators/templates/config.rb` and in particular the `get_facets` method). This is required if using nested bucket aggregation functionality.

## [v1.0.4](https://github.com/CDRH/api/compare/v1.0....v1.0.4) - Updates & license
## [v1.0.5] - API v1 on Ruby 3.1.6, Rails 6.1.7
[v1.0.5]: https://github.com/CDRH/api/compare/v1.0.4...v1.0.5

### Changed
- Ruby 3.1.6
- Rails 6.1.7

## [v1.0.4](https://github.com/CDRH/api/compare/v1.0.3...v1.0.4) - Updates & license

### Changed

- Updated Ruby version, gems (which addresses mimemagic dependency problem), and
license added
license added

### Added

- Documentation on facets and highlighting

## [v1.0.3](https://github.com/CDRH/api/compare/v1.0.2...v1.0.3) - gem updates

### Changed

- updates to rails and other gems

## [v1.0.2](https://github.com/CDRH/api/compare/v1.0.1...v1.0.2) - escapes and sorting

### Fixed

- question mark and asterisk behavior in queries
- order of expected, actual in tests
- sort behavior for relevancy

### Added

- support for multivalued and nested field sorting
- documentation moved back into apium from henbit location in order to version it with software

### Changed

- ruby, rails, and other gem versions

## [v1.0.1](https://github.com/CDRH/api/compare/v1.00...v1.0.1) - version 1.0.1

### Changed

- ruby, rails, and other gem versions
- version moved to initializer

Expand All @@ -113,4 +149,3 @@ license added
### Contributors

- Jessica Dussault (jduss4)

35 changes: 15 additions & 20 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -1,44 +1,34 @@
source 'https://rubygems.org'

git_source(:github) do |repo_name|
repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
"https://github.com/#{repo_name}.git"
end

git_source(:github) { |repo| "https://github.com/#{repo}.git" }

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '~> 6.1.7'
# Use sqlite3 as the database for Active Record
gem 'sqlite3', '~> 1.4'
# Use Puma as the app server
gem 'puma', '~> 5.0'
# Transpile app-like JavaScript. Read more: https://github.com/rails/webpacker
# gem 'webpacker', '~> 5.0'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
# gem 'turbolinks', '~> 5'
gem 'puma', '>= 5.0'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.7'
# gem 'jbuilder', '~> 2.7'
# Use Redis adapter to run Action Cable in production
# gem 'redis', '~> 4.0'
# Use Active Model has_secure_password
# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'

# Use Active Storage variant
# Use ActiveStorage variant
# gem 'image_processing', '~> 1.2'

# Reduces boot times through caching; required in config/boot.rb
gem 'bootsnap', '>= 1.4.4', require: false

# Use Rack CORS for handling Cross-Origin Resource Sharing (CORS), making cross-origin AJAX possible
# gem 'rack-cors'

group :development, :test do
# Call 'byebug' anywhere in the code to stop execution and get a debugger console
gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
end

group :development do
# Access an interactive console on exception pages or by calling 'console' anywhere in the code.
gem 'web-console', '>= 4.1.0'
# Display performance information such as SQL time and flame graphs for each request in your browser.
# Can be configured to work on production as well see: https://github.com/MiniProfiler/rack-mini-profiler/blob/master/README.md
gem 'rack-mini-profiler', '~> 2.0'
gem 'listen', '~> 3.3'
# Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
gem 'spring'
Expand All @@ -47,6 +37,11 @@ end
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

# Additions to Rails defaults

# Note: Above list different from other Rails apps
# because this app was created with --api option

# using rest-client because I've had far more luck than the
# stlib net/http
gem 'rest-client', '>= 2.1.0.rc1', '< 2.2'
gem 'rest-client', '~> 2.1'
Loading