Skip to content
  • Iustin Pop's avatar
    Add a Ganeti-specific implementation of Curl Multi · cc40185c
    Iustin Pop authored
    
    
    As we want to be able to run queries against multiple nodes in
    parallel, and furthermore in parallel with other work, we need to
    implement the Curl Multi interface (see libcurl-multi(3)).
    
    This patch adds a Ganeti-specific such implementation, to be used
    until upstream Curl bindings provide it. The implemented interface
    (there are two 'multi' interfaces) is the older curl_multi_perform(3).
    
    It has one downside (which is also, somewhat, and advantage): we use
    polling instead of more properly select() or poll. This is due to how
    waiting for FDs is implemented in Haskell: currently, it's not
    possible to wait for multiple FDs at once nicely, so we'd have to fork
    many threads for each FD to be watched, or alternatively one could use
    FFI select, but that would block the entire runtime.
    
    With the current poll method, the implementation achieves consistent
    ~100 RPC/s per second (with 10 multi interfaces running in parallel,
    each with 10 easy handles), and over ~1 hour of runtime the memory
    usage is stable, so memory allocation/deallocation (manual when
    dealing with FFI) _seems_ to be well handled.
    
    Future optimisations could be to move to curl_multi_socket_action(3),
    which might allow better integration with the Haskell runtime.
    
    Signed-off-by: default avatarIustin Pop <iustin@google.com>
    Reviewed-by: default avatarMichele Tartara <mtartara@google.com>
    cc40185c