Commit 62d5242b authored by Iustin Pop's avatar Iustin Pop
Browse files

Switch Luxi sendMsg from strict to lazy ByteStrings

Commit e821050d

 (“Switch the Luxi interface from Strings to
ByteStrings”) was designed to optimise the receive interface, but has
an unfortunate side-effect: when sending non-trivial messages, it
means that both the entire String and the ByteString versions must be
in memory at the same time, leading to much increased memory usage.

By changing the "hPut" from strict to lazy ByteStrings, it means that
both the String and the ByteString values can be evaluated lazily,
with significant effects: for a test query answer, instead of having
a peak from ~600MB to 1.4G during the entire Luxi send operation,
memory consumption actually decreased during the send operation, as
the ByteString chunks are released individually and even the backing
storage of the items that create the JSON string serialisation is
released lazily as well. So instead of slow growth 10→550MB then quick
peak to 1.4GB during Luxi send, we now have an even slower growth to
~580MB and then decrease of memory as the Luxi send progresses.

The only downside is of a small increase in CPU time of a few percents
for the above case; for our use cases, I think this is much desirable.
Signed-off-by: default avatarIustin Pop <iustin@google.com>
Reviewed-by: default avatarHelga Velroyen <helgav@google.com>
parent 13b17073
......@@ -55,7 +55,9 @@ module Ganeti.Luxi
import Control.Exception (catch)
import Data.IORef
import qualified Data.ByteString as B
import qualified Data.ByteString.Lazy as BL
import qualified Data.ByteString.UTF8 as UTF8
import qualified Data.ByteString.Lazy.UTF8 as UTF8L
import Data.Word (Word8)
import Control.Monad
import Text.JSON (encodeStrict, decodeStrict)
......@@ -245,9 +247,9 @@ closeClient = hClose . socket
-- | Sends a message over a luxi transport.
sendMsg :: Client -> String -> IO ()
sendMsg s buf = withTimeout luxiDefRwto "sending luxi message" $ do
let encoded = UTF8.fromString buf
let encoded = UTF8L.fromString buf
handle = socket s
B.hPut handle encoded
BL.hPut handle encoded
B.hPut handle bEOM
hFlush handle
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment