Unhandled WebException

May 19, 2011 at 12:47 PM

Hi Oliver,

I had to cut out early of RFinance to catch a flight and was sorry to have missed your presentation on RserveCLI, but happy to have found it here.  I downloaded and installed yesterday and after one or two hiccups was able to get things working.  Now I am attempting to use it to do some "real" work, but I have been getting an unhandled webexception "expected x bytes, but received y bytes" in RserveCLI.Qap1.  It's kinda weird since it works fine but arbitrary craps out not always at the same point in the data.  Here's how I'm calling it:

   var df = Sexp.MakeDataFrame();
   df["y"] = Sexp.Make(m_fdom);
   df["change.1d"] = Sexp.Make(m_1Day);
   df["change.7d"] = Sexp.Make(m_7Day);
   df["change.6m"] = Sexp.Make(m_6Month);
   m_r["d"] = df;
   
  var lm = m_r["summary(lm(y ~ change.1d + change.7d +    change.6m, d))"];
  var coefs = lm["coefficients"];

I am building a dataframe of lists of doubles.  The linear regression is called after new data is added.  It works fine a few dozen times and then dies on the call to lm().  I copied the exception details below.  Any ideas on what the problem might be?

Many thanks,
Lawson

System.Net.WebException was unhandled
  Message=Expected 2972 bytes of data, but received 2916.
  Source=RserveCli
  StackTrace:
       at RserveCli.Qap1.Command(Int32 cmd, IList`1 data) in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\RserveCLI\RserveCLI\Qap1.cs:line 540
       at RserveCli.RConnection.Eval(String s) in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\RserveCLI\RserveCLI\RConnection.cs:line 267
       at RserveCli.RConnection.get_Item(String s) in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\RserveCLI\RserveCLI\RConnection.cs:line 229
       at Metolius.SnP.FDOMVariable.OnFill(Order order) in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\Backtester\Metolius\SnP\FDOMVariable.cs:line 111
       at Balsam.Common.Strategy.SubmitOrder(Order order) in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\Backtester\Backtester\Common\Strategy.cs:line 1168
       at Balsam.Common.Strategy.BuyClose(BarSeries series, Double quantity, String signalName) in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\Backtester\Backtester\Common\Strategy.cs:line 1768
       at Balsam.Common.Strategy.BuyClose() in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\Backtester\Backtester\Common\Strategy.cs:line 1712
       at Metolius.SnP.FDOMVariable.OnBar() in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\Backtester\Metolius\SnP\FDOMVariable.cs:line 80
       at Balsam.Common.Strategy.Simulate() in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\Backtester\Backtester\Common\Strategy.cs:line 896
       at Balsam.Common.Strategy.RunSimulation() in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\Backtester\Backtester\Common\Strategy.cs:line 949
       at Metolius.Program.Main(String[] args) in C:\Users\Lawson\Documents\Visual Studio 2010\Projects\Backtester\Metolius\Program.cs:line 47
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException: 

 

Jun 4, 2011 at 7:21 PM

Found a solution here:  http://rdotnet.codeplex.com/

Nov 10, 2011 at 5:34 AM

Hi ,

 

I have encountered the same error when I need to get a 1000000 by 2 matrix from R serve. The error I get is very similar to yours:

System.Net.WebException: Expected 16000028 bytes of data, but received 1943256.
   at RserveCli.Qap1.Command(Int32 cmd, IList`1 data) in C:\Users\yin\Desktop\rservecli-2614\RserveCLI\trunk\RserveCLI\Qap1.cs:line 540
   at RserveCli.RConnection.Eval(String s) in C:\Users\yin\Desktop\rservecli-2614\RserveCLI\trunk\RserveCLI\RConnection.cs:line 259
   at RserveCli.RConnection.get_Item(String s) in C:\Users\yin\Desktop\rservecli-2614\RserveCLI\trunk\RserveCLI\RConnection.cs:line 221
   at <StartupCode$FSI_0014>.$FSI_0014.main@() in C:\Users\yin\Desktop\rservecli-2614\RserveCLI\trunk\FSharpExample\Script1.fsx:line 47

 

But the R Serve project claims that there is no such a limit on the object size:

http://www.rforge.net/Rserve/faq.html

[On default, 16MB for data sent to R Serve, and unlimited for data sent out]

 

R Serve has a Java client too. I will try that to see if the same problem exists there. If not, then probably I can try to solve the bug in the RServeCLI.

 

Best,

Yin

Nov 11, 2011 at 1:26 AM

The Java client works correct. I also solved the bug in the code, actually there are two bugs, both in Qap1.cs:

1. When the data object is large, the this.socket.Recieve(buf) cannot get all the data at once, we need to keep receiving data.

2. The first 4 bytes are headers, which indicate the data type and length. The length formula is only correct for small objects, and wrong for large objects, say > 40MB. 

I corrected both bugs, please see the following modified code:

 

        public List<object> Command(int cmd, IList<object> data)
        {
            int toConsume = this.SubmitCommand(cmd, data);
            var res = new List<object>();
            while (toConsume > 0)
            {
                var dhbuf = new byte[4];
                if (this.socket.Receive(dhbuf) != 4)
                {
                    throw new WebException("Didn't receive a header.");
                }

                byte typ = dhbuf[0];

                // ReSharper disable RedundantCast
                // this formulae is wrong! only correct for small objects
                int dlength = dhbuf[1] + (((int)dhbuf[2]) << 8) + (((int)dhbuf[3]) << 16);

                dlength = toConsume - 4;
                // ReSharper restore RedundantCast
                var dvbuf = new byte[dlength];
                var tempbuf = new byte[dlength];
                var curr = 0;

                //System.Threading.Thread.Sleep(500);
                var received = this.socket.Receive(dvbuf);

                while (curr + received < dlength )
                {
                    Console.WriteLine(curr.ToString() + " + " + received.ToString() + " = " + (curr + received).ToString() + " to " + dlength.ToString());
                    curr += received;
                    received = this.socket.Receive(tempbuf);
                    Array.Copy(tempbuf, 0, dvbuf, curr, received);
                }

                /*
                if (received != dvbuf.Length)
                {
                    var tempR = this.socket.Receive(dvbuf);
                    throw new WebException("Expected " + dvbuf.Length + " bytes of data, but received " + received + ".");
                }
                */

                switch (typ & 15) // only the lower 4 bits

Nov 11, 2011 at 1:31 AM

To see what's been changed, here is the original code:

        public List<object> Command(int cmd, IList<object> data)
        {
            int toConsume = this.SubmitCommand(cmd, data);
            var res = new List<object>();
            while (toConsume > 0)
            {
                var dhbuf = new byte[4];
                if (this.socket.Receive(dhbuf) != 4)
                {
                    throw new WebException("Didn't receive a header.");
                }

                byte typ = dhbuf[0];

                // ReSharper disable RedundantCast
                int dlength = dhbuf[1] + (((int)dhbuf[2]) << 8) + (((int)dhbuf[3]) << 16);

                // ReSharper restore RedundantCast
                var dvbuf = new byte[dlength];
                var received = this.socket.Receive(dvbuf);
                if (received != dvbuf.Length)
                {
                    throw new WebException("Expected " + dvbuf.Length + " bytes of data, but received " + received + ".");
                }

                switch (typ)

Mar 21, 2012 at 10:22 AM

thank you, works as it should!

Coordinator
Jun 17, 2012 at 5:21 AM

I submitted that fix to the Subversion repository. I'll wait a week or so for other feedback and if all is well I'll release a new version then.

May 7, 2013 at 12:59 AM
To get this to work, I also had to change a line in "public Sexp DecodeSexp(byte[] data, ref int start)" from
var length = data[start + 1] + (((int)data[start + 2]) << 8) + (((int)data[start + 3]) << 8);
to
int length = data.Length - 4;