haskell - How to do memory mapped IO on custom data types? -

the setup

i implemented mmap based file reading , directly ran strange behavior. relevant code is:

-- | map whole aedat file memory , return vector of events -- todo finalizing semantics of this? mmapaerdata :: s.storable => filepath -> io (s.vector (aer.event a)) mmapaerdata name =     -- mmap file memory , find offset behind header     bs <- dropheader <$> mmapfilebytestring name nothing     -- conversion necessary 'foreignptr'     -- 'bytestring'     b.unsafeuseascstring bs $ \ptr ->       fptr <- newforeignptr_ ptr       let count = b.length bs `div` 8 -- sizeof 1 event       return $ s.unsafefromforeignptr0 (castforeignptr fptr) count 

→ code in context

some explanation: aedat format long list of 2 word32s. 1 encodes address other timestamp. before there lines of header text drop in dropheader function. directly on foreignptr if absolutely necessary, prefer use common function works on bytestrings instead.

the storable instances can found here , here. not sure alignment here, suspect alignment of 8 should correct.

the problem

reading data works quite well, after time memory seems corrupted somehow:

>>> es <- dvs.mmapdvsdata "dataset.aedat"  >>> es s.! 1000 event {address = address {polarity = d, posx = 6, posy = 50}, timestamp = 74.771407s} >>> :type es es :: s.vector (dvs.event dvs.address) >>> _ <- evaluate (v.convert es :: v.vector (dvs.event dvs.address)) >>> es s.! 1000 event {address = address {polarity = d, posx = 0, posy = 44}, timestamp = 0s} 

apparently accessing elements of es somehow corrupts memory. or garbage collector recycles it? either way, strange. can that?

mmapfilebytestring performs mmap, creates foreignptr, , sticks foreignptr bytestring. unsafeuseascstring coerces foreignptr ptr, create new foreignptr. take second foreignptr , use s.unsafefromforeignptr0 create vector.

having 2 foreignptrs pointing @ same memory no no. ghc runtime treats them 2 separate objects. after references bytestring gone, finalizer itsforeignptr called, deallocating mmap , reclaiming underlying memory. leaves second foreignptr pointing @ invalid region.

the solution here use data.bytestring.internal.toforeignptr extract , re-use foreignptr bytestring. replace unsafeuseascstring block this:

let (fptr,offset,len) = data.bytestring.internal.toforeignptr bs -- might worthwhile assert offset == 0 let count = len `div` 8 return $ s.unsafefromforeignptr0 (castforeignptr fptr) count 

imho, real solution here not fiddle stuff @ all. conventionally read file bytestring, pull out 8-byte substrings , manually conver them events. mmap , foreignptr stuff dangerous, , not whole lot faster doing things safely , correctly. if want absolute fastest performance without regard safety, program in c.