From 62d5242b9568aa4510c71042de22da897ea19b55 Mon Sep 17 00:00:00 2001
From: Iustin Pop <iustin@google.com>
Date: Wed, 12 Dec 2012 02:50:33 +0100
Subject: [PATCH] Switch Luxi sendMsg from strict to lazy ByteStrings
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Commit e821050d (β€œSwitch the Luxi interface from Strings to
ByteStrings”) was designed to optimise the receive interface, but has
an unfortunate side-effect: when sending non-trivial messages, it
means that both the entire String and the ByteString versions must be
in memory at the same time, leading to much increased memory usage.

By changing the "hPut" from strict to lazy ByteStrings, it means that
both the String and the ByteString values can be evaluated lazily,
with significant effects: for a test query answer, instead of having
a peak from ~600MB to 1.4G during the entire Luxi send operation,
memory consumption actually decreased during the send operation, as
the ByteString chunks are released individually and even the backing
storage of the items that create the JSON string serialisation is
released lazily as well. So instead of slow growth 10β†’550MB then quick
peak to 1.4GB during Luxi send, we now have an even slower growth to
~580MB and then decrease of memory as the Luxi send progresses.

The only downside is of a small increase in CPU time of a few percents
for the above case; for our use cases, I think this is much desirable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Helga Velroyen <helgav@google.com>
---
 htools/Ganeti/Luxi.hs | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/htools/Ganeti/Luxi.hs b/htools/Ganeti/Luxi.hs
index ff3989239..9e5b3370f 100644
--- a/htools/Ganeti/Luxi.hs
+++ b/htools/Ganeti/Luxi.hs
@@ -55,7 +55,9 @@ module Ganeti.Luxi
 import Control.Exception (catch)
 import Data.IORef
 import qualified Data.ByteString as B
+import qualified Data.ByteString.Lazy as BL
 import qualified Data.ByteString.UTF8 as UTF8
+import qualified Data.ByteString.Lazy.UTF8 as UTF8L
 import Data.Word (Word8)
 import Control.Monad
 import Text.JSON (encodeStrict, decodeStrict)
@@ -245,9 +247,9 @@ closeClient = hClose . socket
 -- | Sends a message over a luxi transport.
 sendMsg :: Client -> String -> IO ()
 sendMsg s buf = withTimeout luxiDefRwto "sending luxi message" $ do
-  let encoded = UTF8.fromString buf
+  let encoded = UTF8L.fromString buf
       handle = socket s
-  B.hPut handle encoded
+  BL.hPut handle encoded
   B.hPut handle bEOM
   hFlush handle
 
-- 
GitLab