Skip to content

Implement a new API to get schema dependents #623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jviotti merged 1 commit into from
Jan 27, 2026
Merged

Implement a new API to get schema dependents #623

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions collections/self/v1/schemas/api/schemas/dependents/response.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Sourcemeta One Schema Dependents API Response",
"description": "The response format for the schema dependents API endpoint",
"examples": [
[
{
"from": "https://example.com/schemas/user.json",
"to": "https://example.com/schemas/address.json"
}
]
],
"type": "array",
"items": {
"type": "object",
"required": [ "from", "to" ],
"properties": {
"from": {
"$ref": "https://schemas.sourcemeta.com/sourcemeta/std/v0/ietf/uri/url"
},
"to": {
"$ref": "https://schemas.sourcemeta.com/sourcemeta/std/v0/ietf/uri/url"
}
},
"additionalProperties": false
}
}
22 changes: 22 additions & 0 deletions docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,28 @@ GET /self/v1/api/schemas/dependencies/{path}

The schema does not exist.

### Dependents

*This endpoint retrieves all direct and transitive dependents of the JSON
Schema located at the specified `{path}` parameter. A dependent is a schema
that references this schema, either directly or indirectly through a chain of
references.*

```
GET /self/v1/api/schemas/dependents/{path}
```

=== "200"

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `/*/from` | String | Yes | The absolute URL of the schema that originates the dependency |
| `/*/to` | String | Yes | The absolute URL of the schema being referenced |

=== "404"

The schema does not exist.

### Health

*This endpoint retrieves the health analysis and score for the JSON Schema located at the specified `{path}` parameter.*
Expand Down
117 changes: 114 additions & 3 deletions src/index/generators.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,10 @@

#include <cassert> // assert
#include <filesystem> // std::filesystem
#include <queue> // std::queue
#include <set> // std::set
#include <sstream> // std::ostringstream
#include <utility> // std::move
#include <utility> // std::move, std::pair
#include <variant> // std::visit

namespace sourcemeta::one {
Expand Down Expand Up @@ -192,15 +194,124 @@ struct GENERATE_DEPENDENCIES {
[&result](const auto &origin, const auto &pointer, const auto &target,
const auto &) {
auto trace{sourcemeta::core::JSON::make_object()};
trace.assign("from", sourcemeta::core::JSON{std::string{origin}});
trace.assign("to", sourcemeta::core::JSON{std::string{target}});
trace.assign("from", without_json_extension(origin));
trace.assign("to", without_json_extension(target));
trace.assign("at", sourcemeta::core::to_json(pointer));
result.push_back(std::move(trace));
});
// Otherwise we are returning non-sense
assert(result.unique());
const auto timestamp_end{std::chrono::steady_clock::now()};

std::filesystem::create_directories(destination.parent_path());
sourcemeta::one::write_pretty_json(
destination, result, "application/json",
sourcemeta::one::Encoding::GZIP, sourcemeta::core::JSON{nullptr},
std::chrono::duration_cast<std::chrono::milliseconds>(timestamp_end -
timestamp_start));
}

private:
static auto without_json_extension(const std::string_view uri)
-> sourcemeta::core::JSON {
if (uri.ends_with(".json")) {
return sourcemeta::core::JSON{std::string{uri.substr(0, uri.size() - 5)}};
}

return sourcemeta::core::JSON{std::string{uri}};
}
};

struct GENERATE_DEPENDENCY_TREE {
using Context = sourcemeta::one::Resolver;
static auto
handler(const std::filesystem::path &destination,
const sourcemeta::core::BuildDependencies<std::filesystem::path>
&dependencies,
const sourcemeta::core::BuildDynamicCallback<std::filesystem::path> &,
const Context &) -> void {
const auto timestamp_start{std::chrono::steady_clock::now()};

// Direct dependencies
using DirectMap =
std::unordered_map<sourcemeta::core::JSON::String,
std::unordered_set<sourcemeta::core::JSON::String>>;
DirectMap direct;
for (const auto &path : dependencies) {
const auto contents{sourcemeta::one::read_json(path)};
assert(contents.is_array());
for (const auto &entry : contents.as_array()) {
direct[entry.at("to").to_string()].emplace(
entry.at("from").to_string());
}
}

// Indirect dependencies
using TransitiveMap =
std::unordered_map<sourcemeta::core::JSON::String,
std::set<std::pair<sourcemeta::core::JSON::String,
sourcemeta::core::JSON::String>>>;
TransitiveMap transitive;
for (const auto &[target, _] : direct) {
auto &edges{transitive[target]};
std::unordered_set<sourcemeta::core::JSON::StringView> visited;
visited.emplace(target);
std::queue<sourcemeta::core::JSON::String> queue;
queue.emplace(target);
while (!queue.empty()) {
const auto current{std::move(queue.front())};
queue.pop();
const auto match{direct.find(current)};
if (match == direct.cend()) {
continue;
}

for (const auto &dependent : match->second) {
edges.emplace(dependent, current);
if (visited.emplace(dependent).second) {
queue.emplace(dependent);
}
}
}
}

auto result{sourcemeta::core::to_json(transitive)};
const auto timestamp_end{std::chrono::steady_clock::now()};

std::filesystem::create_directories(destination.parent_path());
sourcemeta::one::write_pretty_json(
destination, result, "application/json",
sourcemeta::one::Encoding::GZIP, sourcemeta::core::JSON{nullptr},
std::chrono::duration_cast<std::chrono::milliseconds>(timestamp_end -
timestamp_start));
}
};

struct GENERATE_DEPENDENTS {
using Context = sourcemeta::core::JSON::String;
static auto
handler(const std::filesystem::path &destination,
const sourcemeta::core::BuildDependencies<std::filesystem::path>
&dependencies,
const sourcemeta::core::BuildDynamicCallback<std::filesystem::path> &,
const Context &context) -> void {
const auto timestamp_start{std::chrono::steady_clock::now()};
const auto contents{sourcemeta::one::read_json(dependencies.front())};
assert(contents.is_object());
auto result{sourcemeta::core::JSON::make_array()};
const auto *match{contents.try_at(context)};
if (match) {
assert(match->is_array());
for (const auto &entry : match->as_array()) {
auto object{sourcemeta::core::JSON::make_object()};
object.assign("from", entry.at(0));
object.assign("to", entry.at(1));
result.push_back(std::move(object));
}
}

const auto timestamp_end{std::chrono::steady_clock::now()};

std::filesystem::create_directories(destination.parent_path());
sourcemeta::one::write_pretty_json(
destination, result, "application/json",
Expand Down
37 changes: 33 additions & 4 deletions src/index/index.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,11 +319,12 @@ static auto index_main(const std::string_view &program,
// This is a pretty fast step that will be useful for us to properly declare
// dependencies for HTML and navigational targets

print_progress(mutex, concurrency, "Reviewing", schemas_path.string(), 1, 1);
print_progress(mutex, concurrency, "Reviewing", schemas_path.string(), 1, 2);
std::vector<std::filesystem::path> directories;
// The top-level is itself a directory
directories.emplace_back(schemas_path);
std::vector<std::filesystem::path> summaries;
std::vector<std::filesystem::path> dependencies;
if (std::filesystem::exists(schemas_path)) {
for (const auto &entry :
std::filesystem::recursive_directory_iterator{schemas_path}) {
Expand All @@ -348,6 +349,8 @@ static auto index_main(const std::string_view &program,
entry.path().parent_path().filename() == SENTINEL) {
summaries.emplace_back(explorer_path / std::filesystem::relative(
entry.path(), schemas_path));
dependencies.emplace_back(entry.path().parent_path() /
"dependencies.metapack");
}
}

Expand All @@ -365,8 +368,32 @@ static auto index_main(const std::string_view &program,
});
}

print_progress(mutex, concurrency, "Reviewing", schemas_path.string(), 2, 2);
DISPATCH<sourcemeta::one::GENERATE_DEPENDENCY_TREE>(
output.path() / "dependency-tree.metapack", dependencies, resolver, mutex,
"Reviewing", schemas_path.string(), "dependencies", adapter, output);

/////////////////////////////////////////////////////////////////////////////
// (10) A further pass on the schemas after review
/////////////////////////////////////////////////////////////////////////////

sourcemeta::core::parallel_for_each(
resolver.begin(), resolver.end(),
[&output, &schemas_path, &resolver, &mutex,
&adapter](const auto &schema, const auto threads, const auto cursor) {
print_progress(mutex, threads, "Reworking", schema.first, cursor,
resolver.size());
const auto base_path{schemas_path / schema.second.relative_path /
SENTINEL};
DISPATCH<sourcemeta::one::GENERATE_DEPENDENTS>(
base_path / "dependents.metapack",
{output.path() / "dependency-tree.metapack"}, schema.first, mutex,
"Reworking", schema.first, "dependents", adapter, output);
},
concurrency, THREAD_STACK_SIZE);

/////////////////////////////////////////////////////////////////////////////
// (10) Generate the JSON-based explorer
// (11) Generate the JSON-based explorer
/////////////////////////////////////////////////////////////////////////////

print_progress(mutex, concurrency, "Producing", explorer_path.string(), 0,
Expand Down Expand Up @@ -406,7 +433,7 @@ static auto index_main(const std::string_view &program,
summaries.pop_back();

/////////////////////////////////////////////////////////////////////////////
// (11) Generate the HTML web interface
// (12) Generate the HTML web interface
/////////////////////////////////////////////////////////////////////////////

if (configuration.html.has_value()) {
Expand Down Expand Up @@ -476,7 +503,7 @@ static auto index_main(const std::string_view &program,
}

/////////////////////////////////////////////////////////////////////////////
// (12) Generate the pre computed routes
// (13) Generate the pre computed routes
/////////////////////////////////////////////////////////////////////////////

sourcemeta::core::URITemplateRouter router;
Expand All @@ -485,6 +512,8 @@ static auto index_main(const std::string_view &program,
sourcemeta::one::HANDLER_SELF_V1_API_LIST_PATH);
router.add("/self/v1/api/schemas/dependencies/{+schema}",
sourcemeta::one::HANDLER_SELF_V1_API_SCHEMAS_DEPENDENCIES);
router.add("/self/v1/api/schemas/dependents/{+schema}",
sourcemeta::one::HANDLER_SELF_V1_API_SCHEMAS_DEPENDENTS);
router.add("/self/v1/api/schemas/health/{+schema}",
sourcemeta::one::HANDLER_SELF_V1_API_SCHEMAS_HEALTH);
router.add("/self/v1/api/schemas/locations/{+schema}",
Expand Down
14 changes: 14 additions & 0 deletions src/server/server.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,19 @@ static auto handle_self_v1_api_schemas_dependencies(
std::nullopt, "/self/v1/schemas/api/schemas/dependencies/response");
}

static auto
handle_self_v1_api_schemas_dependents(const std::filesystem::path &base,
const std::span<std::string_view> matches,
sourcemeta::one::HTTPRequest &request,
sourcemeta::one::HTTPResponse &response)
-> void {
const auto absolute_path{base / "schemas" / matches.front() / SENTINEL /
"dependents.metapack"};
action_serve_metapack_file(
request, response, absolute_path, sourcemeta::one::STATUS_OK, true,
std::nullopt, "/self/v1/schemas/api/schemas/dependents/response");
}

static auto
handle_self_v1_api_schemas_health(const std::filesystem::path &base,
const std::span<std::string_view> matches,
Expand Down Expand Up @@ -238,6 +251,7 @@ static const Handler HANDLERS[] = {handle_default,
handle_self_v1_api_list,
handle_self_v1_api_list_path,
handle_self_v1_api_schemas_dependencies,
handle_self_v1_api_schemas_dependents,
handle_self_v1_api_schemas_health,
handle_self_v1_api_schemas_locations,
handle_self_v1_api_schemas_positions,
Expand Down
21 changes: 11 additions & 10 deletions src/shared/include/sourcemeta/one/shared.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,17 @@ namespace sourcemeta::one {
constexpr auto HANDLER_SELF_V1_API_LIST = 1;
constexpr auto HANDLER_SELF_V1_API_LIST_PATH = 2;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_DEPENDENCIES = 3;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_HEALTH = 4;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_LOCATIONS = 5;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_POSITIONS = 6;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_STATS = 7;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_METADATA = 8;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_EVALUATE = 9;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_TRACE = 10;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_SEARCH = 11;
constexpr auto HANDLER_SELF_V1_API_DEFAULT = 12;
constexpr auto HANDLER_SELF_STATIC = 13;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_DEPENDENTS = 4;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_HEALTH = 5;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_LOCATIONS = 6;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_POSITIONS = 7;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_STATS = 8;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_METADATA = 9;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_EVALUATE = 10;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_TRACE = 11;
constexpr auto HANDLER_SELF_V1_API_SCHEMAS_SEARCH = 12;
constexpr auto HANDLER_SELF_V1_API_DEFAULT = 13;
constexpr auto HANDLER_SELF_STATIC = 14;

} // namespace sourcemeta::one

Expand Down
4 changes: 4 additions & 0 deletions test/cli/index/no-base-uri.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ cd - > /dev/null

cat << 'EOF' > "$TMP/expected.txt"
./configuration.json
./dependency-tree.metapack
./dependency-tree.metapack.deps
./explorer
./explorer/%
./explorer/%/404.metapack
Expand Down Expand Up @@ -84,6 +86,8 @@ cat << 'EOF' > "$TMP/expected.txt"
./schemas/test/schemas/test-1/%/bundle.metapack.deps
./schemas/test/schemas/test-1/%/dependencies.metapack
./schemas/test/schemas/test-1/%/dependencies.metapack.deps
./schemas/test/schemas/test-1/%/dependents.metapack
./schemas/test/schemas/test-1/%/dependents.metapack.deps
./schemas/test/schemas/test-1/%/editor.metapack
./schemas/test/schemas/test-1/%/editor.metapack.deps
./schemas/test/schemas/test-1/%/health.metapack
Expand Down
10 changes: 10 additions & 0 deletions test/cli/index/rebuild-cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ Using configuration: $(realpath "$TMP")/one.json
Detecting: $(realpath "$TMP")/schemas/foo.json (#1)
(100%) Ingesting: https://sourcemeta.com/example/schemas/foo
(100%) Analysing: https://sourcemeta.com/example/schemas/foo
( 50%) Reviewing: $(realpath "$TMP")/output/schemas
(100%) Reviewing: $(realpath "$TMP")/output/schemas
(100%) Reworking: https://sourcemeta.com/example/schemas/foo
( 0%) Producing: $(realpath "$TMP")/output/explorer
( 33%) Producing: example/schemas
( 66%) Producing: example
Expand Down Expand Up @@ -82,7 +84,11 @@ Detecting: $(realpath "$TMP")/schemas/foo.json (#1)
(skip) Analysing: https://sourcemeta.com/example/schemas/foo [blaze-exhaustive]
(skip) Analysing: https://sourcemeta.com/example/schemas/foo [blaze-fast]
(skip) Analysing: https://sourcemeta.com/example/schemas/foo [metadata]
( 50%) Reviewing: $(realpath "$TMP")/output/schemas
(100%) Reviewing: $(realpath "$TMP")/output/schemas
(skip) Reviewing: $(realpath "$TMP")/output/schemas [dependencies]
(100%) Reworking: https://sourcemeta.com/example/schemas/foo
(skip) Reworking: https://sourcemeta.com/example/schemas/foo [dependents]
( 0%) Producing: $(realpath "$TMP")/output/explorer
(skip) Producing: $(realpath "$TMP")/output/explorer [search]
( 33%) Producing: example/schemas
Expand Down Expand Up @@ -120,7 +126,9 @@ Using configuration: $(realpath "$TMP")/one.json
Detecting: $(realpath "$TMP")/schemas/foo.json (#1)
(100%) Ingesting: https://sourcemeta.com/example/schemas/foo
(100%) Analysing: https://sourcemeta.com/example/schemas/foo
( 50%) Reviewing: $(realpath "$TMP")/output/schemas
(100%) Reviewing: $(realpath "$TMP")/output/schemas
(100%) Reworking: https://sourcemeta.com/example/schemas/foo
( 0%) Producing: $(realpath "$TMP")/output/explorer
( 33%) Producing: example/schemas
( 66%) Producing: example
Expand All @@ -144,7 +152,9 @@ Using configuration: $(realpath "$TMP")/one.json
Detecting: $(realpath "$TMP")/schemas/foo.json (#1)
(100%) Ingesting: https://sourcemeta.com/example/schemas/foo
(100%) Analysing: https://sourcemeta.com/example/schemas/foo
( 50%) Reviewing: $(realpath "$TMP")/output/schemas
(100%) Reviewing: $(realpath "$TMP")/output/schemas
(100%) Reworking: https://sourcemeta.com/example/schemas/foo
( 0%) Producing: $(realpath "$TMP")/output/explorer
( 33%) Producing: example/schemas
( 66%) Producing: example
Expand Down
Loading