Skip to content

Commit e01ad4c

Browse files
rezk2llrezk2ll
committed
feat(nextcloud): trigger endpoint for bulk migrations
Twake users migrating from Nextcloud need a way to import their entire file tree into their Cozy. Konnectors and stack workers are bounded by job timeouts, so the heavy lifting has to live outside the stack. This endpoint is the user-facing entry point: it validates the credentials synchronously, persists the account, creates a tracking document, and hands the work off to a separate migration service via RabbitMQ. The service drives the existing /remote/nextcloud/:account/* routes and updates the tracking document, which the Settings UI watches over realtime.
1 parent 84a60d7 commit e01ad4c

File tree

11 files changed

+1163
-11
lines changed

11 files changed

+1163
-11
lines changed

‎docs/nextcloud.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -579,3 +579,91 @@ Authorization: Bearer eyJhbG...
579579
```http
580580
HTTP/1.1 204 No Content
581581
```
582+
583+
## POST /remote/nextcloud/migration
584+
585+
This route triggers a one-shot bulk migration of a user's Nextcloud files into
586+
their Cozy. The Stack validates the credentials, persists an
587+
`io.cozy.accounts` document, creates an `io.cozy.nextcloud.migrations`
588+
tracking document in `pending` state, and publishes a
589+
`nextcloud.migration.requested` command to the `migration` RabbitMQ exchange.
590+
The actual transfer is performed by an external migration service that
591+
consumes the command and drives the existing `/remote/nextcloud/:account/*`
592+
routes, updating the tracking document as it progresses.
593+
594+
Before persisting anything, the Stack probes the supplied credentials against
595+
the Nextcloud instance via the OCS `user_status` endpoint, so wrong passwords
596+
and unreachable hosts surface synchronously instead of being deferred to the
597+
migration service. The probe also resolves the WebDAV user ID, which is
598+
cached on the account document so the migration service does not need to
599+
re-fetch it.
600+
601+
When an existing `io.cozy.accounts` document for the same `account_type:
602+
"nextcloud"` + `auth.url` + `auth.login` triplet is found, it is reused with
603+
its stored password and `webdav_user_id` refreshed from the request. Only one
604+
migration can be in flight per instance at a time: if a `pending` or `running`
605+
tracking document already exists, the Stack returns `409 Conflict`. Failed
606+
migrations do not block new attempts.
607+
608+
**Note:** a permission on `POST io.cozy.nextcloud.migrations` is required to
609+
use this route.
610+
611+
### Request
612+
613+
```http
614+
POST /remote/nextcloud/migration HTTP/1.1
615+
Host: cozy.example.net
616+
Authorization: Bearer eyJhbG...
617+
Content-Type: application/json
618+
```
619+
620+
```json
621+
{
622+
"nextcloud_url": "https://nextcloud.example.com",
623+
"nextcloud_login": "alice",
624+
"nextcloud_app_password": "xxxxx-xxxxx-xxxxx-xxxxx-xxxxx",
625+
"source_path": "/"
626+
}
627+
```
628+
629+
`source_path` is optional and defaults to `/`. The `nextcloud_app_password`
630+
should be a Nextcloud app password, not the user's main account password.
631+
632+
### Response
633+
634+
```http
635+
HTTP/1.1 201 Created
636+
Content-Type: application/vnd.api+json
637+
```
638+
639+
```json
640+
{
641+
"data": {
642+
"id": "d4e5f6a7b8c94d0ea1b2c3d4e5f6a7b8",
643+
"type": "io.cozy.nextcloud.migrations",
644+
"attributes": {
645+
"status": "pending",
646+
"target_dir": "/Nextcloud",
647+
"progress": {
648+
"files_imported": 0,
649+
"files_total": 0,
650+
"bytes_imported": 0,
651+
"bytes_total": 0
652+
},
653+
"errors": [],
654+
"skipped": [],
655+
"started_at": null,
656+
"finished_at": null
657+
}
658+
}
659+
}
660+
```
661+
662+
#### Status codes
663+
664+
- 201 Created, when the migration has been queued and the tracking document is returned
665+
- 401 Unauthorized, when the Nextcloud credentials are rejected by the remote host
666+
- 409 Conflict, when a `pending` or `running` migration already exists
667+
- 500 Internal Server Error, when the conflict check, account upsert, or tracking document creation fails
668+
- 502 Bad Gateway, when the Nextcloud instance is unreachable
669+
- 503 Service Unavailable, when the migration command cannot be published to RabbitMQ. The tracking document is marked `failed` before returning

‎docs/rabbitmq.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,46 @@ rabbitmq:
134134
- If queue-level `dlx_name`/`dlq_name` are not specified, exchange-level defaults are used.
135135
- Messages that exceed the `delivery_limit` or are rejected will be sent to the DLX and routed to the DLQ.
136136

137+
### Publishers
138+
139+
The Stack also publishes messages to RabbitMQ. Publishers do not declare any
140+
queue or exchange on the Stack side: the exchange must already exist on the
141+
broker, and a queue must be bound by the consuming service. Publishes use the
142+
AMQP `mandatory` flag, so a publish to an exchange with no matching binding
143+
fails with `PublishReturnedError` and the caller is expected to surface the
144+
failure to the user.
145+
146+
#### `auth` exchange
147+
148+
Routing key: `user.deletion.requested`. Published from
149+
`POST /settings/instance/deletion/force` when a user requests permanent
150+
deletion of their account. The payload is the `UserDeletionRequestedMessage`
151+
struct in `pkg/rabbitmq/contracts.go`.
152+
153+
#### `migration` exchange
154+
155+
Routing key: `nextcloud.migration.requested`. Published from
156+
`POST /remote/nextcloud/migration` when a user starts a Nextcloud-to-Cozy
157+
bulk migration. The payload is the `NextcloudMigrationRequestedMessage`
158+
struct in `pkg/rabbitmq/contracts.go`:
159+
160+
```json
161+
{
162+
"migrationId": "d4e5f6a7b8c94d0ea1b2c3d4e5f6a7b8",
163+
"workplaceFqdn": "alice.cozy.example.com",
164+
"accountId": "a1b2c3d4e5f6",
165+
"sourcePath": "/",
166+
"timestamp": 1712563200
167+
}
168+
```
169+
170+
Credentials are never in the payload: they live in the `io.cozy.accounts`
171+
document referenced by `accountId`. The Stack populates `MessageID` with the
172+
migration ID for cross-system tracing. The consuming service is responsible
173+
for declaring its queue, binding it to this exchange, and processing the
174+
messages; if no queue is bound when the Stack publishes, the user receives
175+
`503` and the tracking document is marked `failed`.
176+
137177
### Handlers
138178

139179
Handlers implement a simple interface:

‎model/nextcloud/migration.go‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
package nextcloud
2+
3+
import (
4+
"errors"
5+
"time"
6+
7+
"github.com/cozy/cozy-stack/model/instance"
8+
"github.com/cozy/cozy-stack/pkg/consts"
9+
"github.com/cozy/cozy-stack/pkg/couchdb"
10+
"github.com/cozy/cozy-stack/pkg/couchdb/mango"
11+
"github.com/cozy/cozy-stack/pkg/jsonapi"
12+
)
13+
14+
const (
15+
MigrationStatusPending = "pending"
16+
MigrationStatusRunning = "running"
17+
MigrationStatusCompleted = "completed"
18+
MigrationStatusFailed = "failed"
19+
)
20+
21+
const DefaultMigrationTargetDir = "/Nextcloud"
22+
23+
var ErrMigrationConflict = errors.New("a nextcloud migration is already in progress")
24+
25+
// Migration is the io.cozy.nextcloud.migrations tracking document.
26+
//
27+
// The schema (especially the nested Progress object) is the contract with
28+
// twake-migration-nextcloud. Flat counters would crash the service's progress
29+
// reducer because it spreads doc.progress and adds to its fields.
30+
type Migration struct {
31+
DocID string `json:"_id,omitempty"`
32+
DocRev string `json:"_rev,omitempty"`
33+
Status string `json:"status"`
34+
TargetDir string `json:"target_dir"`
35+
Progress MigrationProgress `json:"progress"`
36+
Errors []MigrationError `json:"errors"`
37+
Skipped []SkippedFile `json:"skipped"`
38+
StartedAt *time.Time `json:"started_at"`
39+
FinishedAt *time.Time `json:"finished_at"`
40+
}
41+
42+
type MigrationProgress struct {
43+
FilesImported int64 `json:"files_imported"`
44+
FilesTotal int64 `json:"files_total"`
45+
BytesImported int64 `json:"bytes_imported"`
46+
BytesTotal int64 `json:"bytes_total"`
47+
}
48+
49+
type MigrationError struct {
50+
Path string `json:"path"`
51+
Message string `json:"message"`
52+
At time.Time `json:"at"`
53+
}
54+
55+
type SkippedFile struct {
56+
Path string `json:"path"`
57+
Reason string `json:"reason"`
58+
Size int64 `json:"size"`
59+
}
60+
61+
func (m *Migration) ID() string { return m.DocID }
62+
func (m *Migration) Rev() string { return m.DocRev }
63+
func (m *Migration) DocType() string { return consts.NextcloudMigrations }
64+
func (m *Migration) SetID(id string) { m.DocID = id }
65+
func (m *Migration) SetRev(rev string) { m.DocRev = rev }
66+
67+
func (m *Migration) Clone() couchdb.Doc {
68+
cloned := *m
69+
70+
if m.Errors != nil {
71+
cloned.Errors = make([]MigrationError, len(m.Errors))
72+
copy(cloned.Errors, m.Errors)
73+
}
74+
if m.Skipped != nil {
75+
cloned.Skipped = make([]SkippedFile, len(m.Skipped))
76+
copy(cloned.Skipped, m.Skipped)
77+
}
78+
if m.StartedAt != nil {
79+
t := *m.StartedAt
80+
cloned.StartedAt = &t
81+
}
82+
if m.FinishedAt != nil {
83+
t := *m.FinishedAt
84+
cloned.FinishedAt = &t
85+
}
86+
return &cloned
87+
}
88+
89+
func (m *Migration) Links() *jsonapi.LinksList { return nil }
90+
func (m *Migration) Relationships() jsonapi.RelationshipMap { return nil }
91+
func (m *Migration) Included() []jsonapi.Object { return nil }
92+
93+
var (
94+
_ couchdb.Doc = (*Migration)(nil)
95+
_ jsonapi.Object = (*Migration)(nil)
96+
)
97+
98+
// NewPendingMigration returns a fresh Migration document in the pending state.
99+
// Errors and Skipped are explicit empty slices so the JSON serialization
100+
// produces "[]" rather than "null" — the migration service consumes them as
101+
// arrays and would crash on null.
102+
func NewPendingMigration(targetDir string) *Migration {
103+
if targetDir == "" {
104+
targetDir = DefaultMigrationTargetDir
105+
}
106+
return &Migration{
107+
Status: MigrationStatusPending,
108+
TargetDir: targetDir,
109+
Errors: []MigrationError{},
110+
Skipped: []SkippedFile{},
111+
}
112+
}
113+
114+
func (m *Migration) MarkFailed(inst *instance.Instance, cause error) error {
115+
now := time.Now().UTC()
116+
m.Status = MigrationStatusFailed
117+
if m.FinishedAt == nil {
118+
m.FinishedAt = &now
119+
}
120+
m.Errors = append(m.Errors, MigrationError{
121+
Message: cause.Error(),
122+
At: now,
123+
})
124+
return couchdb.UpdateDoc(inst, m)
125+
}
126+
127+
// FindActiveMigration returns the first pending or running migration, or
128+
// (nil, nil) if none. A missing doctype database or index is treated as "no
129+
// active migration" so the first call on a fresh instance succeeds.
130+
func FindActiveMigration(inst *instance.Instance) (*Migration, error) {
131+
var docs []*Migration
132+
req := &couchdb.FindRequest{
133+
UseIndex: "by-status",
134+
Selector: mango.In("status", []interface{}{
135+
MigrationStatusPending,
136+
MigrationStatusRunning,
137+
}),
138+
Limit: 1,
139+
}
140+
err := couchdb.FindDocs(inst, consts.NextcloudMigrations, req, &docs)
141+
if err != nil {
142+
if couchdb.IsNoDatabaseError(err) || couchdb.IsNoUsableIndexError(err) {
143+
return nil, nil
144+
}
145+
return nil, err
146+
}
147+
if len(docs) == 0 {
148+
return nil, nil
149+
}
150+
return docs[0], nil
151+
}

‎model/nextcloud/nextcloud.go‎

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
package nextcloud
44

55
import (
6+
"context"
67
"encoding/json"
78
"io"
89
"net/http"
@@ -20,6 +21,7 @@ import (
2021
"github.com/cozy/cozy-stack/pkg/consts"
2122
"github.com/cozy/cozy-stack/pkg/couchdb"
2223
"github.com/cozy/cozy-stack/pkg/jsonapi"
24+
"github.com/cozy/cozy-stack/pkg/logger"
2325
"github.com/cozy/cozy-stack/pkg/safehttp"
2426
"github.com/cozy/cozy-stack/pkg/webdav"
2527
"github.com/labstack/echo/v4"
@@ -360,16 +362,36 @@ func (nc *NextCloud) buildTrashedURL(item webdav.Item, path string) string {
360362
return u.String()
361363
}
362364

365+
// FetchUserIDWithCredentials probes the OCS user_status endpoint and returns
366+
// the user ID, or webdav.ErrInvalidAuth if the credentials are rejected.
367+
//
363368
// https://docs.nextcloud.com/server/latest/developer_manual/client_apis/OCS/ocs-status-api.html#fetch-your-own-status
364-
func (nc *NextCloud) fetchUserID() (string, error) {
365-
logger := nc.webdav.Logger
369+
func FetchUserIDWithCredentials(nextcloudURL, username, password string, logger *logger.Entry) (string, error) {
370+
u, err := url.Parse(nextcloudURL)
371+
if err != nil {
372+
return "", err
373+
}
374+
if u.Scheme == "" || u.Host == "" {
375+
return "", ErrInvalidAccount
376+
}
377+
return fetchUserIDFromHost(u.Scheme, u.Host, username, password, logger)
378+
}
379+
380+
const userStatusProbeTimeout = 30 * time.Second
381+
382+
func fetchUserIDFromHost(scheme, host, username, password string, logger *logger.Entry) (string, error) {
366383
u := url.URL{
367-
Scheme: nc.webdav.Scheme,
368-
Host: nc.webdav.Host,
369-
User: url.UserPassword(nc.webdav.Username, nc.webdav.Password),
384+
Scheme: scheme,
385+
Host: host,
386+
User: url.UserPassword(username, password),
370387
Path: "/ocs/v2.php/apps/user_status/api/v1/user_status",
371388
}
372-
req, err := http.NewRequest(http.MethodGet, u.String(), nil)
389+
// Cap the probe so a hung Nextcloud server can't pin a request goroutine —
390+
// safehttp.ClientWithKeepAlive has handshake timeouts but no overall
391+
// request deadline.
392+
ctx, cancel := context.WithTimeout(context.Background(), userStatusProbeTimeout)
393+
defer cancel()
394+
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
373395
if err != nil {
374396
return "", err
375397
}
@@ -396,6 +418,10 @@ func (nc *NextCloud) fetchUserID() (string, error) {
396418
return payload.OCS.Data.UserID, nil
397419
}
398420

421+
func (nc *NextCloud) fetchUserID() (string, error) {
422+
return fetchUserIDFromHost(nc.webdav.Scheme, nc.webdav.Host, nc.webdav.Username, nc.webdav.Password, nc.webdav.Logger)
423+
}
424+
399425
type OCSPayload struct {
400426
OCS struct {
401427
Data struct {

‎pkg/consts/doctype.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,9 @@ const (
142142
// NextCloudFiles doc type is used when listing files from a NextCloud via
143143
// WebDAV.
144144
NextCloudFiles = "io.cozy.remote.nextcloud.files"
145+
// NextcloudMigrations doc type is used to track bulk Nextcloud to Cozy
146+
// migrations orchestrated by the external migration service.
147+
NextcloudMigrations = "io.cozy.nextcloud.migrations"
145148
// ChatAssistants doc type for AI chat assistants.
146149
ChatAssistants = "io.cozy.ai.chat.assistants"
147150
// ChatConversations doc type is used for a chat between the user and a chatbot.

‎pkg/couchdb/index.go‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ import (
1414

1515
// IndexViewsVersion is the version of current definition of views & indexes.
1616
// This number should be incremented when this file changes.
17-
const IndexViewsVersion int = 37
17+
const IndexViewsVersion int = 38
1818

1919
// Indexes is the index list required by an instance to run properly.
2020
var Indexes = []*mango.Index{
@@ -72,6 +72,10 @@ var Indexes = []*mango.Index{
7272

7373
// Used to find the active sharings
7474
mango.MakeIndex(consts.Sharings, "active", mango.IndexDef{Fields: []string{"active"}}),
75+
76+
// Used to detect an already in-flight Nextcloud migration when a user
77+
// tries to start a new one.
78+
mango.MakeIndex(consts.NextcloudMigrations, "by-status", mango.IndexDef{Fields: []string{"status"}}),
7579
}
7680

7781
// DiskUsageView is the view used for computing the disk usage for files

0 commit comments

Comments
 (0)