Allow end user to control badger's memory usage #3
Description
Is your feature request related to a problem? Please describe.
Currently the gowarcserver's can consume more memory than they should in their host container
Describe the solution you'd like
Badger has an extensive API that should enable a solution where the end user can configure the memory usage on startup using arguments and/or the config file. @maeb has a previous attempt at this which can be found on over at the gowarc repo (link is not permanent, so it might die at some point).
Additional context
A good place to start might be badgers documentation entry on memory usage https://dgraph.io/docs/badger/get-started/#memory-usage
All option fields in badger v2.2007.2:
https://github.com/dgraph-io/badger/blob/d5a25b83fbf4f3f61ff03a9202e36f5b75544426/options.go#L35
// Required options.
Dir string
ValueDir string
// Usually modified options.
SyncWrites bool
TableLoadingMode options.FileLoadingMode
ValueLogLoadingMode options.FileLoadingMode
NumVersionsToKeep int
ReadOnly bool
Truncate bool
Logger Logger
Compression options.CompressionType
InMemory bool
// Fine tuning options.
MaxTableSize int64
LevelSizeMultiplier int
MaxLevels int
ValueThreshold int
NumMemtables int
// Changing BlockSize across DB runs will not break badger. The block size is
// read from the block index stored at the end of the table.
BlockSize int
BloomFalsePositive float64
KeepL0InMemory bool
BlockCacheSize int64
IndexCacheSize int64
LoadBloomsOnOpen bool
NumLevelZeroTables int
NumLevelZeroTablesStall int
LevelOneSize int64
ValueLogFileSize int64
ValueLogMaxEntries uint32
NumCompactors int
CompactL0OnClose bool
LogRotatesToFlush int32
ZSTDCompressionLevel int
// When set, checksum will be validated for each entry read from the value log file.
VerifyValueChecksum bool
// Encryption related options.
EncryptionKey []byte // encryption key
EncryptionKeyRotationDuration time.Duration // key rotation duration
// BypassLockGaurd will bypass the lock guard on badger. Bypassing lock
// guard can cause data corruption if multiple badger instances are using
// the same directory. Use this options with caution.
BypassLockGuard bool
// ChecksumVerificationMode decides when db should verify checksums for SSTable blocks.
ChecksumVerificationMode options.ChecksumVerificationMode
// DetectConflicts determines whether the transactions would be checked for
// conflicts. The transactions can be processed at a higher rate when
// conflict detection is disabled.
DetectConflicts bool
// Transaction start and commit timestamps are managed by end-user.
// This is only useful for databases built on top of Badger (like Dgraph).
// Not recommended for most users.
managedTxns bool
// 4. Flags for testing purposes
// ------------------------------
maxBatchCount int64 // max entries in batch
maxBatchSize int64 // max batch size in bytes