Add a Ganeti-specific implementation of Curl Multi
As we want to be able to run queries against multiple nodes in parallel, and furthermore in parallel with other work, we need to implement the Curl Multi interface (see libcurl-multi(3)). This patch adds a Ganeti-specific such implementation, to be used until upstream Curl bindings provide it. The implemented interface (there are two 'multi' interfaces) is the older curl_multi_perform(3). It has one downside (which is also, somewhat, and advantage): we use polling instead of more properly select() or poll. This is due to how waiting for FDs is implemented in Haskell: currently, it's not possible to wait for multiple FDs at once nicely, so we'd have to fork many threads for each FD to be watched, or alternatively one could use FFI select, but that would block the entire runtime. With the current poll method, the implementation achieves consistent ~100 RPC/s per second (with 10 multi interfaces running in parallel, each with 10 easy handles), and over ~1 hour of runtime the memory usage is stable, so memory allocation/deallocation (manual when dealing with FFI) _seems_ to be well handled. Future optimisations could be to move to curl_multi_socket_action(3), which might allow better integration with the Haskell runtime. Signed-off-by:Iustin Pop <iustin@google.com> Reviewed-by:
Michele Tartara <mtartara@google.com>
Loading
Please register or sign in to comment