when accessing the node data we used to use cast to
*[maxAllocSize]byte, which breaks if we try to go across maxAllocSize boundary.
This leads to occasional panics.
Sample stacktrace:
```
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
github.com/boltdb/bolt.(*node).write(0xc208010f50, 0xc27452a000)
$GOPATH/src/github.com/boltdb/bolt/node.go:228 +0x5a5
github.com/boltdb/bolt.(*node).spill(0xc208010f50, 0x0, 0x0)
$GOPATH/src/github.com/boltdb/bolt/node.go:364 +0x506
github.com/boltdb/bolt.(*node).spill(0xc208010700, 0x0, 0x0)
$GOPATH/src/github.com/boltdb/bolt/node.go:336 +0x12d
github.com/boltdb/bolt.(*node).spill(0xc208010620, 0x0, 0x0)
$GOPATH/src/github.com/boltdb/bolt/node.go:336 +0x12d
github.com/boltdb/bolt.(*Bucket).spill(0xc22b6ae880, 0x0, 0x0)
$GOPATH/src/github.com/boltdb/bolt/bucket.go:535 +0x1c4
github.com/boltdb/bolt.(*Bucket).spill(0xc22b6ae840, 0x0, 0x0)
$GOPATH/src/github.com/boltdb/bolt/bucket.go:502 +0xac2
github.com/boltdb/bolt.(*Bucket).spill(0xc22f4e2018, 0x0, 0x0)
$GOPATH/src/github.com/boltdb/bolt/bucket.go:502 +0xac2
github.com/boltdb/bolt.(*Tx).Commit(0xc22f4e2000, 0x0, 0x0)
$GOPATH/src/github.com/boltdb/bolt/tx.go:150 +0x1ee
github.com/boltdb/bolt.(*DB).Update(0xc2080e4000, 0xc24d077508, 0x0, 0x0)
$GOPATH/src/github.com/boltdb/bolt/db.go:483 +0x169
```
It usually happens when working with large (50M/100M) values.
One way to reproduce it is to change maxAllocSize in bolt_amd64.go to 70000 and run the tests.
TestBucket_Put_Large crashes.
This commit adds safety documentation to the data accessor functions
in Bolt as well as the README. This was documented once in the package
level godoc but it's important enough that it should be more clear.
This commit moves the functionality in Tx.Copy() to Tx.WriteTo(). This
allows Tx to be used as an io.WriterTo which makes it easier to mock.
The Tx.Copy() function still exists but it's simply a wrapper around
Tx.WriteTo().
This commit changes the maxAllocSize from 256GB to 2GB to handle large
values. It was previously 0xFFFFFFF and I tried adding one more "F" but
it caused an "array too large" error. I played around with the value
some more and found that 0x7FFFFFFF (2GB) is the highest allowed value.
This does not affect how the data is stored. It is simply used for type
converting pointers to array pointers in order to utilize zero copy
from the mmap.
DB.Batch makes it easy to make lots of small transactions with
significantly better performance. Batch combines multiple concurrent
Update calls into a single disk transaction, managing errors smartly.
This commit fixes a bug where only calling NextSequence() on a Bucket does not cause the Bucket to be
peristed. The simple fix is to simply materialize the root node so that the bucket is flushed out
during commit.
Thanks to Matthew Dawson (@MJDSys) for reporting.
https://github.com/boltdb/bolt/issues/296
This commit expands calls to _assert() that use variadic arguments. These calls require conversion to interface{} so there
was a large number of calls to Go's internal convT2E() function. In some profiling this was taking over 20% of total runtime.
I don't remember seeing this before Go 1.4 so perhaps something has changed.
This commit adds fixes suggested by @tv42 for the mmap step fix in 834b38e:
* Check max size before calculating the new the mmap size.
* Fix mmap step loop to go to 1GB instead of 512MB.
This commit fixes an issue where the database would grow whenever it was opened. This was caused by
a recent change that performed a truncation when the database grew. Now there are fixed growth sizes
for the database (1MB, 2MB, 4MB, 8MB, etc) up to 1GB and then the database will grow by 1GB when it
resizes.
See also: 6bb25854a1
sasha-s
Ben Johnson
Mike Marcacci
chrislusf
Daniel Speichert
funkygao
Baiju Muthukadan
Adam Polanski
Vitor De Mario
Tommi Virtanen
everdev