io - Erlang consumer queue -

i have problem want pull discrete chunks of data disk queue, , dequeue them process. data randomly located on disk, not benefit substantially sequential reads. it's alot of data can't load @ once, nor efficient pull in block @ time.

i'd consumer able operate @ own speed, keep healthy queue of data ready i'm not waiting on disk reads process chunks.

is there established way this? i.e jobs framework or safetyvalve? implementing feels reinventing wheel slow consumer operating on disk data common problem.

any suggestions how best tackle erlang way?

you can use {read_ahead, bytes} option on file:open/2:

{read_ahead, size}

this option activates read data buffering. if read/2 calls less size bytes, read operations towards operating system still performed blocks of size bytes. data buffered , returned in subsequent read/2 calls, giving performance gain since number of operating system calls reduced.

the read_ahead buffer highly utilized read_line/1 function in raw mode, why option recommended (for performance reasons) when accessing raw files using function.

if read/2 calls sizes not less than, or greater size bytes, no performance gain can expected.

you've been vague on sizes mentioned using, seems toying buffer size should decent start implementing need.