Skip to content

Update datasources api for chunkingConfig of connectors #7

@tomlynchRNA

Description

@tomlynchRNA

The chunkingConfig property on datasources (of type UnstructuredChunkingConfig) has a new file_type property that can be either 'txt' or 'markdown'. This new property should only be present for connectors, file datasources don't need the file_type property (its optional). Also for connectors (not file uploads), the entire chunkingConfig is optional. If present, that means the embeddingField will be chunked and produce multiple rows, if it's not there it will just embed the embeddingField as-is.

see struct/datasource on branch 512-chunk-synced-rows of agentcloud to see the updated types

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions