Skip to content

make uploaders rotate the unique part of the key on failures (?) #559

@amwolff

Description

@amwolff

https://github.com/storj/common/blob/b9952c61d22febe124585cb2529231854d493a61/accesslogs/uploader.go#L197-L213:

if err := up.store.Put(context.TODO(), up.bucket, up.key, up.body); err != nil {
	if up.retries == u.retryLimit {
		mon.Event("upload_dropped")
		u.log.Error("retry limit reached",
			zap.String("bucket", up.bucket),
			zap.String("prefix", up.key),
			zap.Error(err),
		)
		if done := u.decrementQueueLen(closing); done {
			return nil
		}
		continue // NOTE(artur): here we could spill to disk or something
	}
	up.retries++
	u.queue <- up // failure; don't decrement u.queueLen
	mon.Event("upload_failed")
	continue

I saw failures when upload failing leads to the Storjuploader not being able to upload again because the access used might not have Delete capabilities (needed to overwrite the object). It's surprising this can happen, though, because if the upload fails, the key shouldn't exist with the current guarantees for atomicity.

The first step is to figure out if this is an expected behaviour.

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions