ConnectionKeepAlivePollError exceptions killing the server application.

[Anonymous]

I am tinkering with a chat program, and have a problem where the server occasionally just chokes up and dies. What i get in the log file is first a 23-Sep-2013 01:44:16 Fatal [ NetworkComms]: ConnectionKeepAlivePollError

If the server tries to use the connection after that, i get:

23-Sep-2013 01:44:59 Error [        NetworkComms]: Error sending group chat message :  networkcomms.netmunicationException: System.IO.IOException: Unable to write data to the transport connection: A blocking operation was interrupted by a call to WSACancelBlockingCall. ---> System.Net.Sockets.SocketException: A blocking operation was interrupted by a call to WSACancelBlockingCall
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at NetworkCommsDotNet.TCPConnection.SendPacketSpecific(Packet packet)
   at NetworkCommsDotNet.Connection.SendPacket(Packet packet, Int64& packetSequenceNumber)
   at NetworkCommsDotNet.Connection.SendPacket(Packet packet, Int64& packetSequenceNumber)

a text file is written, named like this: ConnectionKeepAlivePollError 1.44.16.581 23-09-2013 [5936-0].

Inside the exception is:

Base Exception Type: System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.WaitHandle.WaitMultiple(WaitHandle[] waitHandles, Int32 millisecondsTimeout, Boolean exitContext, Boolean WaitAll)
   at System.Threading.WaitHandle.WaitAll(WaitHandle[] waitHandles, Int32 millisecondsTimeout, Boolean exitContext)
   at System.Threading.Tasks.Task.WaitAllSTAAnd64Aware(WaitHandle[] waitHandles, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at NetworkCommsDotNet.Connection.AllConnectionsSendNullPacketKeepAlive(Boolean returnImmediately)
   at NetworkCommsDotNet.Connection.ConnectionKeepAliveWorker()

Stack Trace:    at System.Threading.WaitHandle.WaitMultiple(WaitHandle[] waitHandles, Int32 millisecondsTimeout, Boolean exitContext, Boolean WaitAll)
   at System.Threading.WaitHandle.WaitAll(WaitHandle[] waitHandles, Int32 millisecondsTimeout, Boolean exitContext)
   at System.Threading.Tasks.Task.WaitAllSTAAnd64Aware(WaitHandle[] waitHandles, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at NetworkCommsDotNet.Connection.AllConnectionsSendNullPacketKeepAlive(Boolean returnImmediately)
   at NetworkCommsDotNet.Connection.ConnectionKeepAliveWorker()

When this happens, the server application just stops receiving and sending, but it does not crash the application or disconnect the clients. If i try disconnect the connection or call Networkcomms.Shutdown(), the server application hangs and need to be forced to close.

The relevant parts of my server code follows.
setting up the server:

var dataProcessors = new List<DataProcessor>
                {
                    DPSManager.GetDataProcessor<RijndaelPSKEncrypter>()
                };
            var dataProcessorOptions = new Dictionary<string, string>();
            RijndaelPSKEncrypter.AddPasswordToOptions(dataProcessorOptions, "gffsasfrt4345dvb32432fgdfg563cdgvd4323442f");
            DataSerializer dataSerializer = DPSManager.GetDataSerializer<ProtobufSerializer>();
            NetworkComms.DefaultSendReceiveOptions = new SendReceiveOptions(dataSerializer, dataProcessors, dataProcessorOptions);

            NetworkComms.ConnectionEstablishTimeoutMS = 3000;
            NetworkComms.ConnectionAliveTestTimeoutMS = 5000;

            NetworkComms.AppendGlobalIncomingPacketHandler<ChatMessage>("ChatMessage", ProcessMessage);
            NetworkComms.AppendGlobalConnectionCloseHandler(HandleConnectionClosed);
            NetworkComms.AppendGlobalConnectionEstablishHandler(HandleConnectionEstablished);

            AppDomain.CurrentDomain.UnhandledException += (s, e) =>
                {
                    Exception err = (Exception)e.ExceptionObject;

                    NetworkComms.Logger.ErrorException("Unhandled Exception Caught", err);

};

process message:

private void ProcessMessage(PacketHeader header, Connection connection, ChatMessage msg)
        {
            try
            {
                msg.TimeStampMessageRelayed = DateTime.UtcNow;
                switch (msg.MessageType)
                {
				//in here is all the handling of the messages.
					}
            }
            catch (CommsException err)
            {
                NetworkComms.Logger.ErrorException("Exception in ProcessMessage", err);
            }
}

It would be wonderful if i could get some input on what would be wrong.
The complete code is here.

[Anonymous]

Are you able to replicate the problem using our own chat example, here (download to complete example near the top of the page)?

Marc

[Anonymous]

I tried your chat example, and it worked fine over the local network between 2 computers, it send a few hundred messages in a for loop without problem.

I also tried to connect through the internet and send some messages manually, but then the example failed.

What i did was run the example on my tablet with 3G internet, connecting to my computers external IP with opened port.

This is from the tablet:

Initialising WPF chat example. Accepting connections on:
fe80::351b:ae11:8520:9918%16:10000
fe80::657f:147c:745e:a609%15:10000
192.168.137.230:10000
fe80::1dd3:31bb:6687:7eef%12:10000
::1:10000
127.0.0.1:10000
2001:0:9d38:6ab8:4f7:2d36:3f57:7619:10000
fe80::4f7:2d36:3f57:7619%18:10000
fe80::5efe:192.168.137.230%19:10000

tablet – test
tablet – message2
Connection with ‘[TCP] 192.168.137.230:10000 -> 192.168.1.55:10000’ has been closed.
Connection with ‘[TCP] 192.168.137.230:10000 -> 192.168.1.55:10000’ has been closed.
tablet – message3
tablet – message4
Connection with ‘[TCP] 192.168.137.230:10000 -> 192.168.1.55:10000’ has been closed.
tablet – message5
Connection with ‘[TCP] 192.168.137.230:10000 -> 192.168.1.55:10000’ has been closed.
Connection with ‘[TCP] 192.168.137.230:10000 -> 192.168.1.55:10000’ has been closed.
Connection with ‘[TCP] 192.168.137.230:10000 -> 192.168.1.55:10000’ has been closed.
tablet – message6
Connection with ‘[TCP] 192.168.137.230:10000 -> 192.168.1.55:10000’ has been closed.

after the “message2” was sendt, i received the following error:
A CommsException occured while trying to send message to [TCP] 192.168.137.230:10000 -> 192.168.1.55:10000

After the “message6” was sendt, the server crashed with:

Source=mscorlib
StackTrace:
at System.Threading.Tasks.TaskExceptionHolder.Finalize()
InnerException: NetworkCommsDotNet.ConnectionSetupException
Message=A different connection to the same remoteEndPoint already exists. Duplicate connections should be prevented elsewhere.
Source=NetworkCommsDotNetComplete
StackTrace:
at NetworkCommsDotNet.NetworkComms.AddConnectionReferenceByIdentifier(Connection connection)
at NetworkCommsDotNet.Connection.EstablishConnection()
at NetworkCommsDotNet.TCPConnection.GetConnection(ConnectionInfo connectionInfo, SendReceiveOptions defaultSendReceiveOptions, TcpClient tcpClient, Boolean establishIfRequired)
at NetworkCommsDotNet.TCPConnection.<>c__DisplayClassb.<IncomingConnectionWorker>b__a()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
InnerException:

However, my own client and server connects fine through the internet, apart from the mentioned server crash that happens randomly once every day or every two days.

Here is my client code for the network connection.

[Anonymous]

How are you deploying to the tablet, using xamarin?

Regards,
Marc

[Anonymous]

For the example i just used the exe file, as the tablet runs windows 8.

For my own clients, one in silverlight for windows phone 8, and one in WPF for regular windows. The code connecting to the server is pretty similar between them.

I had a theory that the problem might be happening if the internet on the pc hosting the server program drops out for a few seconds. It could answer the randomness, and my router does sometimes misbehave. I don’t know a way to actually test this, or somehow try prevent it though, as it seems it just completely bypasses any try catch i implement myself.

[Anonymous]

Alright, it happened again tonight. Again, clients are not disconnecting, but sending or receiving to or from the server is not working. Connecting to the server also stops working.
3 clients connected:
Client A: local network, same computer as the server
Client B: local network, different computer
Client c: connected from the internet, different computer.

All 3 are windows machines using NetworkCommsDotNetComplete.dll, version 2.3

Oddly enough, data that was sendt from client B was sendt to client A and Client C about 1 hour delayed, after the errors occured. Messages from Client C was delivered accurately to both A and B until the delayed messages were received, then stopped.

Also as before, when i try to stop the server by calling NetworkComms.Shutdown(); it freezes the program, making a

25-Sep-2013 00:06:17 Fatal [ NetworkComms]: ConnectionKeepAlivePollError

and it has to be forced to close.

The log file produced a bunch of errors like these:

24-Sep-2013 21:43:27 Fatal [ NetworkComms]: CompleteIncomingItemTaskError
24-Sep-2013 21:43:27 Fatal [ NetworkComms]: An unhandled exception occured in CompleteIncomingPacketWorker(), connection with [TCP] 192.168.1.59:36410 -> 37.44.133.84:33922 be closed. See log file for more information.
24-Sep-2013 21:43:36 Fatal [ NetworkComms]: CompleteIncomingItemTaskError
24-Sep-2013 21:43:36 Fatal [ NetworkComms]: An unhandled exception occured in CompleteIncomingPacketWorker(), connection with [TCP] 192.168.1.59:36410 -> 37.44.133.84:57723 be closed. See log file for more information.
24-Sep-2013 21:43:41 Error [ NetworkComms]: Error in task: System.AggregateException: A Task’s exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. —> NetworkCommsDotNet.ConnectionSetupException: ServerSide. Connection was closed during setup from [3].
at NetworkCommsDotNet.TCPConnection.EstablishConnectionSpecific()
at NetworkCommsDotNet.Connection.EstablishConnection()
at NetworkCommsDotNet.TCPConnection.GetConnection(ConnectionInfo connectionInfo, SendReceiveOptions defaultSendReceiveOptions, TcpClient tcpClient, Boolean establishIfRequired)
at NetworkCommsDotNet.TCPConnection.<>c__DisplayClassb.<IncomingConnectionWorker>b__a()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
— End of inner exception stack trace —
—> (Inner Exception #0) NetworkCommsDotNet.ConnectionSetupException: ServerSide. Connection was closed during setup from [3].
at NetworkCommsDotNet.TCPConnection.EstablishConnectionSpecific()
at NetworkCommsDotNet.Connection.EstablishConnection()
at NetworkCommsDotNet.TCPConnection.GetConnection(ConnectionInfo connectionInfo, SendReceiveOptions defaultSendReceiveOptions, TcpClient tcpClient, Boolean establishIfRequired)
at NetworkCommsDotNet.TCPConnection.<>c__DisplayClassb.<IncomingConnectionWorker>b__a()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()<—

I also got the error files output, I put them all in a zip here

[Anonymous]

Hmmm, it’s not overly clear what is going wrong but the key is probably the files that you zipped up:

Base Exception Type: System.ArgumentNullException: Value cannot be null.
Parameter name: ipString
at System.Net.IPAddress.InternalParse(String ipString, Boolean tryParse)
at NetworkCommsDotNet.ConnectionInfo.OnDeserialise()

This suggests you are sending/recieving a ConnectionInfo somewhere and you are creating it will a null reference somewhere. This is subsequently throwing an exception at some time later. Looking at the source we don’t validate data on creation of a ConnectionInfo object, although we probably should be.

Marc

[Anonymous]

Ok, so if i understand this correctly, one of the clients is somehow connecting with a broken connectioninfo, creating errors when the server tries to deserialize it.

My last question then is if there is something i am doing wrong or something i can do to stop it, or is it just one of those “wait for next version things”?

Keep up the good work, i’m really enjoying learning about this stuff, and thanks for the fast answers!

[Anonymous]

Ok, so if i understand this correctly, one of the clients is somehow connecting with a broken connectioninfo, creating errors when the server tries to deserialize it.

… Yes. Although having said that it is unlikely to be in our library source otherwise we would have come across this already. It is entirely possible that this is caused by something else, but unless it’s possible to create a specific use case that demonstrates the failure reliably it is nearly impossible to fix. Some further questions:

1. Are you sending/receiving any ConnectionInfo objects yourself?
2. Try enabling packet check sum validation by placing the following at the start of your applications

NetworkComms.EnablePacketCheckSumValidation = true;

3. Consider logging to a txt file which will help track what occurred immediately before the error. Please see https://networkcomms.net/enable-logging/.

Regards,
Marc

[Anonymous]

I’m not sending any connectioninfo objects myself no. I’ll do the logging and checksum validation and see what happens. Does it make a difference that one of the clients is a mobile phone client?

[Author]

Posts