Skip to content

feat(environment): Added functionality to perist entries from different instance. #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kulak-at wants to merge 2 commits into
base: master
Choose a base branch
from
Open

feat(environment): Added functionality to perist entries from different instance. #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c88dd39
feat(environment): Added functionality to perist entries from different
Feb 18, 2021
03d7d0b
Merge branch 'master' into master
Haroenv Apr 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,9 @@ module.exports = {
matchFields: ['slug', 'modified'], // Array<String> default: ['modified']
concurrentQueries: false, // default: true
skipIndexing: true, // default: false, useful for e.g. preview deploys or local development
continueOnFailure: false // default: false, don't fail the build if algolia indexing fails
continueOnFailure: false, // default: false, don't fail the build if algolia indexing fails
useEnvironment: 'blog', // default: false, used in conjunction with enablePartialUpdates let you persist entries from different environment
environmentKey: 'product', // default: 'environment', key used when useEnvironment is set.
Comment on lines +86 to +87
Copy link
Contributor

@Haroenv Haroenv Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems confusing, what if we have only the option environmentKey, and use it whenever it's set. I think also the option on line 86 possibly has the wrong example value? (is it a boolean or a string?)

Comment on lines +86 to +87
Copy link
Contributor

@Haroenv Haroenv Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
useEnvironment: 'blog', // default: false, used in conjunction with enablePartialUpdates let you persist entries from different environment
environmentKey: 'product', // default: 'environment', key used when useEnvironment is set.
environmentKey: 'blog', // a key that is unique to the Gatsby records. Records without this key are left in place

},
},
],
Expand Down Expand Up @@ -128,6 +130,9 @@ If you pass `replicaUpdateMode: 'replace'` in the index settings, you can choose

If you pass `replicaUpdateMode: 'merge'` in the index settings, the replica settings will combine the replicas set on your dashboard with the additional ones you set via index settings here.

### Persisting entries
If you want to persist entries that are added to the index (for example added by the different gatsby instance if your index is combining several websites) you can set `useEnvironment` to unique string that represents given instance. All entries with different `environment` value will be ignored. You can change the key accordingly using `environmentKey`.
Copy link
Contributor

@Haroenv Haroenv Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also needs updating based on that


## Concurrent Queries

Sometimes, on limited platforms like Netlify, concurrent queries to the same index can lead to unexpected results or hanging builds. Setting `concurrentQueries` to `false` makes it such that queries are run sequentially rather than concurrently, which may solve some concurrent access issues. Be aware that this option may make indexing take longer than it would otherwise.
Expand Down
5 changes: 5 additions & 0 deletions gatsby-node.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,8 @@
chunkSize = 1000,
enablePartialUpdates = false,
matchFields: mainMatchFields = ['modified'],
useEnvironment = false,
environmentKey = 'environment'
Comment on lines +118 to +119
Copy link
Contributor

@Haroenv Haroenv Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
useEnvironment = false,
environmentKey = 'environment'
environmentKey

} = config;

activity.setStatus(
Expand Down Expand Up @@ -173,6 +175,9 @@

// iterate over existing objects and compare to fresh data
for (const [id, existingObj] of Object.entries(indexedObjects)) {
if (useEnvironment && existingObj[environmentKey] !== useEnvironment) {

Check warning on line 178 in gatsby-node.js

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

gatsby-node.js#L178 

Generic Object Injection Sink
Copy link
Contributor

@Haroenv Haroenv Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (useEnvironment && existingObj[environmentKey] !== useEnvironment) {
if (environmentKey && existingObj[environmentKey] !== environmentKey) {

Copy link
Contributor

@Haroenv Haroenv Apr 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also need to check that similar to matchFields, it's present on all new objects

continue;
}
if (queryResultsMap.hasOwnProperty(id)) {
// key matches fresh objects, so compare match fields
const newObj = queryResultsMap[id];
Expand Down