Byte array interpretation in Protobuf communication between Golang and Java

• The Problem

  • I was using protobufs to transfer a ecdsa public key over a tcp connection.
  • The server was written in go, the client was an Android app (Google Play services)
  • The public key consists of two large numbers (X and Y).
  • The X and Y values generated are always positive(not sure why yet, need to see if this is as per design)
  • Client also performs a check using signum to see if the values are negative and throws and exception. The following message is from the adb logs

  • ehou: Cannot parse public key: Point encoding must use only non-negative integers
    

  • The numbers that were positive on the server somehow turned negative on the client.

  • This behaviour was intermittent. The transfer was successful sometimes.

• Experiments and Observations

  • Tried checking if endianness is the problem. But the client continued to throw the same exception.
  • The values X and Y are 32 bytes long.
  • Since I cannot really go and debug what’s going on in the client, since its a system app on android and my phone is not rooted at the moment. I have tried to mimic the conditions using the following snippet.

  • package javaapp;
     
    import java.net.*;
    import java.util.Arrays;
    import java.io.*;
    import java.math.BigInteger;
     
    public class App {
        public static void main(String[] args) {
            String serverHost = "localhost"; // Change to the actual server host
            int serverPort = 3000; // Change to the actual server port
     
            try {
                Socket socket = new Socket(serverHost, serverPort);
                InputStream inputStream = socket.getInputStream();
     
                byte[] buffer = new byte[1024];
                int bytesRead = inputStream.read(buffer);
     
                for(int i=0; i< bytesRead ; i++) {
                    System.out.println(buffer[i]);
                }
                Message.EcP256PublicKey message = Message.EcP256PublicKey.parseFrom(Arrays.copyOf(buffer, bytesRead));
                
     
                // Now you can work with the unmarshalled protobuf message
                // System.out.println((new BigInteger("2").pow(Integer.MAX_VALUE)).subtract(new BigInteger("1")));
     
                System.out.println("Received message:");
                System.out.println("X as byte array: " + Arrays.toString(message.getX().toByteArray()));
                System.out.println("Y as byte array: " + Arrays.toString(message.getY().toByteArray()));
                System.out.println("X as big int: "+new BigInteger(message.getX().toByteArray()).toString(10));
                System.out.println("Y as big int: "+new BigInteger(message.getY().toByteArray()).toString(10));
     
                socket.close();
            } catch (Exception e) {
                e.printStackTrace();
            }
     
        }
    }
     

  • The Golang server is as below

  • package main
     
    import (
    	"crypto/ecdsa"
    	"crypto/elliptic"
    	"crypto/rand"
    	"fmt"
    	"math/big"
    	"net"
     
    	"google.golang.org/protobuf/proto"
    )
     
    func main() {
    	port := 3000
    	fmt.Println("Starting the server on port:", port)
    	l, _ := net.Listen("tcp4", "127.0.0.1:3000")
    	for {
    		conn, _ := l.Accept()
    		// buf := make([]byte, 1024)
    		privateKey, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
    		publicKey := privateKey.PublicKey
    		
    		message, _ := proto.Marshal(&EcP256PublicKey{X: publicKey.X.Bytes(), Y: publicKey.Y.Bytes()})
    		fmt.Println("X: ", publicKey.X, "\nY: ", publicKey.Y, "\n")
    		fmt.Println("X: ", publicKey.X.Bytes(), "\nY: ", publicKey.Y.Bytes(), "\n")
    		
    		conn.Write(message)
    	}
    }
     

  • To my surprise I found that byte array implementation was different in golang and java.
  • In golang each byte in the byte array is unsigned.
  • In java each byte is a signed byte.
  • Though both are same in memory, we see the difference in interpretation of the byte value.
  • And the byte array in java considers the msb as a signed bit.
  • So whenever there is a 1 in the msb the biginteger in java interpreted is as a negative number, this explains the intermittent behaviour.
  • To fix this issue I had to prefix a 0 byte to the beginning of the byte array that was sent from golang. This would make java BigInteger to parse the number correctly since the msb is not 1 anymore.
  • The change in golang code is as below

  • // using FillBytes with size 33 will add a leading 0 byte to the byte array
    message, _ := proto.Marshal(&EcP256PublicKey{X: publicKey.X.FillBytes(make([]byte,33)), Y: publicKey.Y.FillBytes(make([]byte,33))})