Serializing Objects

[Anonymous]

Hi,

I’m trying to do an object in an object..

Basically I have a server and I have modules that connect to the server. Modules can talk to each other, but only through the server. So each packet has a “from”, “to” address along with the “packetType” and Object data.

The problem is that I cannot serialize objects in protobuf. Doing some research turned up this:

http://stackoverflow.com/questions/923977/protobuf-and-listobject-how-to-serialize-deserialize

However, that code does not work for me. I changed “DataItem” to ModulePacketHeader3 and “Value” to “Data” and “value” to “data”. It basically tells me that the name “data” does not exist in the current content.

What I’m trying to do is to make it so that the server will get packets from the clients, then it will check the from field and know where to send it to. I do not want to have to update the server every time I add new packetTypes to clients and have to do appendGlobalIncomingPacket for every single packet type that comes through. So the way I see it – theres two options. Either modify the NetworkComm code so that way any packets that are not to a known address will get handled by my own custom handler, or override as shown in the URL above – so that way the server will get a bunch of packets with ModulePacketHeader3 and then it would know which client to send it on to without having to maintain the server’s code every time I update the packet types that the clients will send to each other.

I’m curious if anybody knows of any other way to do what I’m doing – and why the example isn’t working – since I believe the author of the NetworkComms posted that post.

So far I love your work and will be more than happy to help support it if I can get this working for my project.

Thanks,
Noah

p.s. sorry if I’m not 100% clear – just spent a few hours on this issue. Feel free to ask me any questions, and I’ll be more than happy to clarify.

[MarcF]

When you say object in object, would something like this be suitable:

[ProtoContract]
public class ClientInfo
{
    [ProtoMember(1)]
    string ClientMessage { get; set; }

    private ClientInfo() { }

    public ClientInfo(string clientMessage)
    {
        this.ClientMessage = clientMessage;
    }
}

[ProtoContract]
public class ServerInfo
{
    [ProtoMember(1)]
    public ClientInfo ClientInfo {get; set;}

    [ProtoMember(2)]
    public string ServerMessage { get; set; }

    private ServerInfo() { }

    public ServerInfo(ClientInfo clientInfo, string serverMessage)
    {
        this.ClientInfo = clientInfo;
        this.ServerMessage = serverMessage;
    }
}

[Anonymous]

Kind of. Except that I have the following:

// Used as a header for all module packets
[ProtoContract]
public class ModulePacketHeader2
{
[ProtoMember(1)]
public string packetType { get; set; }

[ProtoMember(2)]
public string from { get; set; }

[ProtoMember(3)]
public string to { get; set; }

[ProtoMember(4)]
public Object data { get; set; }

private ModulePacketHeader2() { }

public ModulePacketHeader2(string packetType, string from, string to, Object data)
{
this.packetType = packetType;
this.from = from;
this.to = to;
this.data = data;
}
}

And I want to put the following object in the data field of above:

[ProtoContract]
public class ModulePacketType2_authResponse
{
[ProtoMember(1)]
public string authCode { get; set; }

private ModulePacketType2_authResponse() { }

public ModulePacketType2_authResponse(string authCode)
{
this.authCode = authCode;
}
}

 

The idea is that the “server” in the middle will get the entire packet (header (authResponse)) and look at the to field and then send it on to the client that the packet is for. But I get exceptions when I try to send above, because it does not know what “Object” is – which is why I tried doing what the website I linked to suggested (abstract and sealed class) which also gave me errors. Any ideas?

[Anonymous]

Also I couldn’t find which button on the editor would allow me to format my code the way you formatted yours. =-/

[Anonymous]

Just want to clarify more.. Basically the Server will only look at the packet header. It will very rarely if ever care about what is in the Data field (hence it being an object). Right now I have two clients (they are called modules) and will be adding more clients in the future. There will be a few packets that the server will respond to, but it will not respond to most packets other than looking at the from and to field and then relaying the entire message to the client it’s for.

So there could be 100 different packet types, with each being a class. All packets will use the same header. Out of the 100, the server would only have to recognize maybe 2 or 3 – those are packets for modules to authorize against the server – so that way the server knows the module is legit and vice versa. And I will be adding in more packet types all the time as I write new modules (clients) that will communicate through the system. And the last thing I want to do is to have to update the server for every single packet type that I add.

That is why AppendGlobalIncomingPacketHandler T (String, NetworkComms PacketHandlerCallBackDelegate T ) does not work for me on the server side. I would have to add one for every single packet, plus write an individual function for that specific packet type. I also looked at the source code, there is no “if this packet does not have a handler, send it to a function for generic packets”.

For the clients, I will be using AppendGlobalIncomingPacketHandler – its invaluable – because I can have the clients ignore all packets that they cant handle and I can call specific functions for each kind of packet. Perfect, makes my life a lot easier. Not so much for the server part though.

So my options are one of the following for the server:

1.) Figure out how to do what I explained two posts ago and use abstract/sealed classes or other way to allow Protobuf to serialize it. I already got this working using Binary Serialization but it’s a lot slower and the server will be handling A LOT of traffic, so I cannot have any choke points. And that way I can use AppendGlobalIncomingPacketHeader for ModulePacketHeader.
2.) Modify the source code to add in a new callback that I can then hook to – that way if a packet does not match anything in the GlobalIncomingPacket list – it will call my function with the packet code. I will also modify my protocol so that way I wont use PackeHeader for every packet – I would put the to and from fields into every packetType so it will work with protobuf.

3.) Find another way to do this.

I’ll be doing some more research today on Protobuf as well as tinkering with Sealed/Abstract classes. If anybody can help me with this, I would appreciate it a lot.

Sorry for the long post. I re-read the past few posts I made and I felt I didn’t clarify as much as I should have.

[Anonymous]

Finally figured out how to do code in here.
I did the following and it works for declaring data as an object..

[ProtoContract]
 [ProtoInclude(10, typeof(ModulePacketHeader3))]
 public abstract class ModulePacketHeader3
 {
 public static ModulePacketHeader3 Create (T data, string to, string from, string packetType)
 {
 return new ModulePacketHeader3(data, to, from, packetType);
 }
public object Data
 {
 get { return DataImpl; }
 set { DataImpl = value; }
 }
protected abstract object DataImpl {get;set;}
 protected ModulePacketHeader3() { }
public string packetType { get; set; }
 public string from { get; set; }
 public string to { get; set; }
 }
[ProtoContract]
 public sealed class ModulePacketHeader3 : ModulePacketHeader3
 {
 public ModulePacketHeader3() { }
public ModulePacketHeader3(T data, string PacketType, string To, string From)
 { Data = data; packetType = PacketType; to = To; from = From; }
[ProtoMember(1)]
 public string packetType { get; set; }
 [ProtoMember(2)]
 public string from { get; set; }
 [ProtoMember(3)]
 public string to { get; set; }
 [ProtoMember(4)]
 public new T Data { get; set; }
protected override object DataImpl
 {
 get { return Data; }
 set { Data = (T)value; }
 }
}

Its a bit messy. On the client, I do this:

 

ModulePacketHeader3 myHeader = ModulePacketHeader3.Create(new ModulePacketType2_authRequest(this.moduleName, "ModuleManager", this.moduleAuthCode,
 this.moduleType), "ModuleManager", this.moduleName, "Auth-Request");
 myHeader.from = this.moduleName;
 myHeader.to = "ModuleManager";
 myHeader.packetType = "Auth-Request";




And it doesn’t give me an errors during serialization.

On the server side, I have:

// Incoming data callback
 NetworkComms.AppendGlobalIncomingPacketHandler("CustomObject", incomingData);
 NetworkComms.AppendGlobalIncomingPacketHandler("CustomObject", incomingData2);




and the following functions:

 private void incomingData(NetworkCommsDotNet.PacketHeader header, NetworkCommsDotNet.Connection connection, ModulePacketType2_authRequest packet)
 {
 Console.WriteLine("Testing..");
 }

 

private void incomingData2(NetworkCommsDotNet.PacketHeader header, NetworkCommsDotNet.Connection connection, ModulePacketHeader3 packet)
 {
 Console.WriteLine("Testing..");
 }

I put a breakpoint on each of the “Testing..” console writelines. When the packet comes in, it always calls the function for ModulePacketType2_authRequest and not ModulePacketHeader3. Plus all the variables are set to null. I checked the outgoing packet, and all the variables are set to their correct values.

But if I use the following code:



ModulePacketType2_authRequest authPacket =
 new ModulePacketType2_authRequest(this.moduleName, "ModuleManager", this.moduleAuthCode,
 this.moduleType);

myConnection.SendObject("CustomObject", authPacket);

Then I get exactly what I sent from the client to the server.

So I know that it’s not my processor or compressor that is at fault. It seems that if I use an abstract class so I can send a ModulePacketHeader3, the recieving code still thinks – because I’m using the type ModulePacketType2_authRequest – that it is an authRequest class and thus I end up with null values. I’m not sure where in the NetworkComm code this is occuring.

I went back to TriggerGlobalPacketHandlers and TriggerSpecificPacketHandlers. Right now unless anybody else has an ideas, the best way to not lose time is for me to modify one of those functions (probably the global one) so that any unknown packets will be sent to me if they do not match anything on the list.

Just wanted to post this to help others that may have this issue in the future. And I will post any code modifications I make for the same reasons.

[MarcF]

Heyup,

It’s tough to follow exactly what you are doing but a couple of notes:

1. Setting networkComms.net up like this can be dangerous as you are potentially trying to receive two different objects using the some incoming packetType.

// Incoming data callback
 NetworkComms.AppendGlobalIncomingPacketHandler("CustomObject", incomingData);
 NetworkComms.AppendGlobalIncomingPacketHandler("CustomObject", incomingData2);

2. Check your run time folder for potential error log messages.

3.  If you want to step through the conversion, incoming raw bytes are converted back to objects by the methods:

Connection.TriggerSpecificPacketHandlers();
NetworkComms.TriggerGlobalPacketHandlers();

4. If you want the server to simply be forwarding  data without the need for the server to understand it it will be much more efficient to use the following structure. The main advantage of this approach is that you never have to change anything on the server as you add new classes to the clients.

Declare the following objects:

[ProtoContract]
public class ClientObject
{
    [ProtoMember(1)]
    public string ClientMessage { get; set; }

    private ClientObject() { }

    public ClientObject(string clientMessage)
    {
        this.ClientMessage = clientMessage;
    }
}

[ProtoContract]
public class ServerForward
{
    [ProtoMember(1)]
    public string ClientIPAddress { get; set; }

    [ProtoMember(2)]
    public int ClientPort { get; set; }

    [ProtoMember(3)]
    public string PacketType { get; set; }

    [ProtoMember(4)]
    public byte[] ForwardObjectBytes { get; set; }

    private ServerForward() { }

    public ServerForward(string clientIPAddress, int clientPort, string packetType, byte[] forwardObjectBytes)
    {
        this.ClientIPAddress = clientIPAddress;
        this.ClientPort = clientPort;
        this.PacketType = packetType;
        this.ForwardObjectBytes = forwardObjectBytes;
    }
}

Configure the server with the following packet handler:

NetworkComms.AppendGlobalIncomingPacketHandler<ServerForward>("ServerForward", (connection, header, serverForward) =>
                {
                    Console.WriteLine("Recieved server forward object from " + connection);
                    TCPConnection.GetConnection(new ConnectionInfo(serverForward.ClientIPAddress, serverForward.ClientPort)).SendObject(serverForward.PacketType, serverForward.ForwardObjectBytes);
                });

The sending client uses the following:

ClientObject clientObject = new ClientObject("this is a forwarded client object");
            ServerForward serverForward = new ServerForward("192.168.0.10",10000, "ClientObjectRaw", DPSManager.GetDataSerializer<ProtobufSerializer>().SerialiseDataObject<ClientObject>(clientObject).ThreadSafeStream.ToArray());
            TCPConnection.GetConnection(new ConnectionInfo(serverIP, serverPort)).SendObject("ServerForward", serverForward);

The receiving client uses the following:

NetworkComms.AppendGlobalIncomingPacketHandler<byte[]>("ClientObjectRaw", (connection, header, rawBytes) =>
                {
                    ClientObject incomingClientObject = DPSManager.GetDataSerializer<ProtobufSerializer>().DeserialiseDataObject<ClientObject>(rawBytes);
                    Console.WriteLine("Server forwarded client object which says + " incomingClientObject.ClientMessage);
                });

 

[Anonymous]

Thank you very much. #4 gave me the break through – I did not know that it was possible to serialize / deserialize using your code from byte[] to an object, and that allowed me to define what kind of data “data” is (array of bytes) and thus avoid errors related to not defining what kind of object it is.

The reason why I use the same string for different packets, is because I have a class called PacketHeader. So when I do myConnection.SendObject(“PacketHeader”, packet) – the other side knows it uses the packetHeader class with byte[] data. Then depending on the packet type (as defined in the header), I can further deserialize the data into another packet.

But I’m also using stringSendingPacketType for other packets too. Like authentication against the server – that only the server will get (not get and forward like with PacketHeader packets) I would use “ReportModuleStatistics” or something similar.

Thank you once again for your help. Great work on the code. =-)

[Author]

Posts