System.OutOfMemoryException

[Anonymous]

Hi, I would like to know how to best handle this situation. I made a socket server using the NetworkComms 2.3.0 library. The socket server expects at least 500 users to connect with continuous send and receive operations. Actually, the application is an ERP application. The problem is that the program can’t handle a lot of methods running asynchronously. Am I doing something wrong? To simulate the problem, I created a program for stress testing. The code can be found below:

Here’s the Socket Tester:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using NetworkCommsDotNet;
using System.Timers;

namespace SocketTest
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();
            this.Closing += (s, e) =>
            {
                NetworkComms.Shutdown();
            };
            var t = new Timer()
            {
                AutoReset = true,
                Enabled = true,
                Interval = 100
            };
            t.Elapsed += (s, e) =>
            {
                NetworkComms.SendObject("poll", "127.0.0.1", 30000, DateTime.Now);
            };

            NetworkComms.AppendGlobalIncomingPacketHandler<DateTime>("server", (packetType, connection, message) =>
            {
                App.Current.Dispatcher.Invoke(() =>
                {
                    haha.Items.Add(message);
                });
            });
        }

        private void Window_PreviewKeyDown(object sender, KeyEventArgs e)
        {
            NetworkComms.SendObject("key", "127.0.0.1", 30000, e.Key.ToString());
        }

    }
}

Here’s the Socket Server:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using NetworkCommsDotNet;
using System.Timers;

namespace SocketServer
{
    class Program
    {
        static void Main(string[] args)
        {
            NetworkComms.DefaultListenPort = 30000;
            NetworkComms.AppendGlobalIncomingPacketHandler<string>("key", (packetType, connection, message) =>
            {
                Console.WriteLine("key input from {0} : {1}", connection.ConnectionInfo.RemoteEndPoint, message);
            });

            NetworkComms.AppendGlobalIncomingPacketHandler<DateTime>("poll", (packetType, connection, message) =>
            {
                Console.WriteLine("poll received from {0} containing {1}", connection.ConnectionInfo.RemoteEndPoint, message);
            });
            TCPConnection.StartListening(true);

            var t = new Timer()
            {
                AutoReset = true,
                Enabled = true,
                Interval = 500
            };
            t.Elapsed += (s, e) =>
            {
                var list = NetworkComms.GetExistingConnection().ToList();
                foreach (var data in list)                {
                    data.SendObject("server", DateTime.Now);
                }
            };
            t.Start();
        }
    }
}

I made like 30 – 50 instances of the Socket Tester until the Socket Server produce a .txt file containing a System.OutOfMemoryException.

Please take a look at my codes if improvements can still be made.

Note: About the NetworkComms.GetExistingConnection().ToList() <– I am aware that this takes memory.

[Anonymous]

Follow Up:

I tried removing the .ToList() from the Note above and the same thing happens after 30 instances of the Socket Tester.

here’s the content of one of the error log:

Base Exception Type: System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown.
at LZMA.LZ.OutWindow.Create(UInt32 windowSize) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\LZ\LzOutWindow.cs:line 36
at LZMA.Decoder.SetDictionarySize(UInt32 dictionarySize) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\LzmaDecoder.cs:line 196
at LZMA.Decoder.SetDecoderProperties(Byte[] properties) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\LzmaDecoder.cs:line 383
at LZMA.SevenZipHelper.DecompressStreamToStream(Stream inStream, Stream outStream) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\SevenZipHelper.cs:line 294
at SevenZipLZMACompressor.LZMACompressor.ReverseProcessDataStream(Stream inStream, Stream outStream, Dictionary`2 options, Int64& writtenBytes) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\LZMACompressor.cs:line 75
at DPSBase.DataSerializer.DeserialiseGeneralObject[T](MemoryStream receivedObjectStream, List1 dataProcessors, Dictionary2 options) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\DPSBase\DataSerializer.cs:line 243
at DPSBase.DataSerializer.DeserialiseDataObject[T](MemoryStream receivedObjectStream, List1 dataProcessors, Dictionary2 options) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\DPSBase\DataSerializer.cs:line 237
at NetworkCommsDotNet.PacketTypeHandlerDelegateWrapper`1.DeSerialize(MemoryStream incomingBytes, SendReceiveOptions options) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\NetworkCommsDotNet\Tools\PacketUnwrappers.cs:line 72
at NetworkCommsDotNet.NetworkComms.TriggerGlobalPacketHandlers(PacketHeader packetHeader, Connection connection, MemoryStream incomingDataStream, SendReceiveOptions options, Boolean ignoreUnknownPacketTypeOverride) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\NetworkCommsDotNet\NetworkComms.cs:line 1252

Stack Trace: at LZMA.LZ.OutWindow.Create(UInt32 windowSize) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\LZ\LzOutWindow.cs:line 36
at LZMA.Decoder.SetDictionarySize(UInt32 dictionarySize) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\LzmaDecoder.cs:line 196
at LZMA.Decoder.SetDecoderProperties(Byte[] properties) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\LzmaDecoder.cs:line 383
at LZMA.SevenZipHelper.DecompressStreamToStream(Stream inStream, Stream outStream) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\SevenZipHelper.cs:line 294
at SevenZipLZMACompressor.LZMACompressor.ReverseProcessDataStream(Stream inStream, Stream outStream, Dictionary`2 options, Int64& writtenBytes) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\SevenZipLZMACompressor\LZMACompressor.cs:line 75
at DPSBase.DataSerializer.DeserialiseGeneralObject[T](MemoryStream receivedObjectStream, List1 dataProcessors, Dictionary2 options) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\DPSBase\DataSerializer.cs:line 243
at DPSBase.DataSerializer.DeserialiseDataObject[T](MemoryStream receivedObjectStream, List1 dataProcessors, Dictionary2 options) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\DPSBase\DataSerializer.cs:line 237
at NetworkCommsDotNet.PacketTypeHandlerDelegateWrapper`1.DeSerialize(MemoryStream incomingBytes, SendReceiveOptions options) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\NetworkCommsDotNet\Tools\PacketUnwrappers.cs:line 72
at NetworkCommsDotNet.NetworkComms.TriggerGlobalPacketHandlers(PacketHeader packetHeader, Connection connection, MemoryStream incomingDataStream, SendReceiveOptions options, Boolean ignoreUnknownPacketTypeOverride) in c:\Users\Karnifexx\Documents\Visual Studio 2010\Projects\networkcomms.net\NetworkCommsDotNet\NetworkComms.cs:line 1252

[Anonymous]

Follow up:

Code modified, further testing confirms that the ‘SendObject’ function is the culprit. It takes a lot of memory especially in the foreach loop.

[Anonymous]

Thanks for posting your issue. The OutOfMemoryException occurs because the compression stage, which is enabled by default, is a fairly memory intensive operation.

You should be able to avoid this problem by disabling the compression on both client and server. Simply add this line to the start of your applications:

NetworkComms.DefaultSendReceiveOptions = new SendReceiveOptions<ProtobufSerializer>();

I would also recommend caching the connection list on the server:

var list = NetworkComms.GetExistingConnection().ToList();

and only updating the cache every 5 seconds or so. This will greatly increasing the performance of the polling.

If you have any further questions please feel free to post back.

Kind regards,
Marc

[Anonymous]

Thanks, that sorted everything. 🙂

[Author]

Posts