Heap limit

All except GUI problems

Heap limit

Postby dmaksimiuk » Thu Dec 01, 2016 6:31 pm

Dear All,
In the trivial code below I am crating a linked list of buffers. At first I was thinking that the limit on the number of allocated elements depends on the available RAM memory (in 32bit address space). To my surprise, the BBox environment started "yelling" at me when the number of allocated buffers was (it varies a bit):
Num elements in the list= 316623
Num of allocated bytes= 666174792

There some questions:
1. Why we cannot go beyond a list that can store more than let say 1GB worth of data?
2. When the garbage collector should claim all the memory back? After assignments
First := NIL;
Head := NIL;
CurrEl := NIL;
the environment still showed allocated memory around 670_000_000 bytes. Only unloading the module reclaim the memory. In addition, the Win7 Task Manager shows more than 674_000_000 bytes allocated to the BBox even if internally the heap is at level of ~500k.

BTW , does anybody has any experience with using a DLL from BBox that starts a task (let say an asynchronous process) , and still is able to communicate with a program written in CP ?

Cheers,
Darek

Code: Select all
MODULE dmTestsTestList;


   IMPORT

      Log := StdLog;

   CONST
      BuffSize = 2100;

   TYPE

      tData* = ARRAY BuffSize OF BYTE;

      tPtrNode = POINTER TO tNode;
      tNode = RECORD
         Buffer*: tData;
         Next: tPtrNode;
      END; (* record *)


   VAR
      First, Head, CurrEl: tPtrNode;
      NumElemCnt: INTEGER;
      RetriveFirstElem: BOOLEAN;


      (* -------------------------------------------------------------------------------------------------------------------------------------------- *)
   PROCEDURE Init* ();
   BEGIN
      NumElemCnt := 0;
      RetriveFirstElem := FALSE;
      First := NIL; Head := NIL; CurrEl := NIL;
      Log.String("Node  size is: "); Log.Int(SIZE(tNode)); Log.Ln();

   END Init;

   (* -------------------------------------------------------------------------------------------------------------------------------------------- *)

   PROCEDURE StoreData* (v: tData);
   BEGIN
      IF NumElemCnt = 0 THEN
         NEW(CurrEl);
         IF CurrEl = NIL THEN         
            HALT(100);
         END;
         First := CurrEl;
         Head := CurrEl;
         CurrEl.Buffer := v;
         CurrEl.Next := NIL;
         INC(NumElemCnt);
      ELSE
         NEW(CurrEl);
         IF CurrEl = NIL THEN
              Log.Ln(); Log.String("Num elements in the list="); Log.Int(NumElemCnt); Log.Ln();
                     Log.String("Num of allocated bytes="); Log.Int(NumElemCnt * SIZE(tNode)); Log.Ln();
            HALT(100);
         END;
         CurrEl.Buffer := v;
         CurrEl.Next := NIL;
         Head.Next := CurrEl;
         Head := CurrEl;
         INC(NumElemCnt);
      END; (* if *)
   END StoreData;

   (* -------------------------------------------------------------------------------------------------------------------------------------------- *)
   PROCEDURE IsEmpty* (): BOOLEAN;
   BEGIN
      RETURN (Head = NIL);
   END IsEmpty;
   (* -------------------------------------------------------------------------------------------------------------------------------------------- *)

   PROCEDURE RetriveData* (VAR v: tData);
   BEGIN
      IF ~RetriveFirstElem THEN
         Head := First;
         v := Head.Buffer;
         Head := Head.Next;
         RetriveFirstElem := TRUE;
         First := NIL (* added for testing *)
      ELSE
         IF Head # NIL THEN
            v := Head.Buffer;
            Head := Head.Next;
         END;
      END; (* if *)
   END RetriveData;

   (* -------------------------------------------------------------------------------------------------------------------------------------------- *)
   PROCEDURE GetNumElements* (): INTEGER;
   BEGIN
      RETURN(NumElemCnt);
   END GetNumElements;

   (* -------------------------------------------------------------------------------------------------------------------------------------------- *)

   PROCEDURE RunTest*;
      CONST
         NumFiles = 500000;
      VAR
         i: INTEGER;
         data: tData;
   BEGIN
      Init();
      Log.String("Adding data to the list ...");
      FOR i := 0 TO LEN(data) - 1 DO data[i] := 1; END;
      FOR i := 1 TO NumFiles DO
         StoreData(data);
      END; (* for *)
      Log.String("done."); Log.Ln();
      Log.String("Num elements  in the list"); Log.Int(GetNumElements()); Log.Ln();
      WHILE ~IsEmpty() DO
         RetriveData(data);
      END; (* while *)
      First := NIL;
      Head := NIL;
      CurrEl := NIL;
      Log.String("DONE "); Log.Ln();
   END RunTest;
   (* -------------------------------------------------------------------------------------------------------------------------------------------- *)

BEGIN
END dmTestsTestList.




 dmTestsTestList.RunTest

 DevDebug.Unload dmTestsTestList
dmaksimiuk
 
Posts: 28
Joined: Sun Jun 14, 2015 3:56 pm
Location: Delft, The Netherlands

Re: Heap limit

Postby Josef Templ » Mon Dec 05, 2016 12:01 pm

In my tests I was able to allocate 1.08 GB (108 x 10MB) on a Vista machine with 2GB Ram
and 1.39 GB (139 x 10MB) on a Win10 machine with 8GB Ram.
The limit seems to depend on the underlying platform.

- Josef
Josef Templ
 
Posts: 215
Joined: Tue Sep 17, 2013 6:50 am

Re: Heap limit

Postby dmaksimiuk » Mon Dec 05, 2016 1:05 pm

Hi Josef,
thanks for testing it. My machine runs 64bit Win10 with 16GB RAM and yet, I got (so far) the worst results :roll: .

Cheers,
Darek
dmaksimiuk
 
Posts: 28
Joined: Sun Jun 14, 2015 3:56 pm
Location: Delft, The Netherlands

Re: Heap limit

Postby Josef Templ » Mon Dec 05, 2016 7:20 pm

For better comparison of our results, here is my test program.
It allocates a sequence of 10MB blocks until it gets no more data
and then outputs the number of blocks.

Code: Select all
MODULE TestHeapsize;

   IMPORT Log := StdLog;

   TYPE
      Data = POINTER TO RECORD
      data: ARRAY 10000000 OF BYTE;
         next: Data;
      END;

   PROCEDURE Run*;
      VAR
         i: INTEGER;
         data, root: Data;
   BEGIN
      NEW(data); root := NIL; i := 0;
      WHILE data # NIL DO INC(i); data.next := root; root := data; NEW(data) END;
      Log.String("nof allocated 10MB blocks: "); Log.Int(i); Log.Ln()
   END Run;

BEGIN
END TestHeapsize.

TestHeapsize.Run


- Josef
Josef Templ
 
Posts: 215
Joined: Tue Sep 17, 2013 6:50 am

Re: Heap limit

Postby dmaksimiuk » Mon Dec 05, 2016 8:30 pm

Hi Josef,
the results of running your test program is:
nof allocated 10MB blocks: 67
which is roughly similar to the results I got with small blocks. Could be something about my version of the BBox system?
I am running 1.7-a1 Build:480, 2016.03.22.

Cheers,
Darek
dmaksimiuk
 
Posts: 28
Joined: Sun Jun 14, 2015 3:56 pm
Location: Delft, The Netherlands

Re: Heap limit

Postby dmaksimiuk » Mon Dec 05, 2016 8:40 pm

... just checked BBox 1.6 and the result is:
nof allocated 10MB blocks: 139

So, it could something magic about the version I am using.

Cheers,
Darek
dmaksimiuk
 
Posts: 28
Joined: Sun Jun 14, 2015 3:56 pm
Location: Delft, The Netherlands

Re: Heap limit

Postby Ivan Denisov » Mon Dec 05, 2016 9:51 pm

My test with 1.7.1-a1 version in Wine (Ubuntu) gives only 60 blocks also. So we should think about this issue now!...
1.6 version gives 80 blocks for me.
User avatar
Ivan Denisov
 
Posts: 238
Joined: Tue Sep 17, 2013 12:21 am
Location: Krasnoyarsk, Russia

Re: Heap limit

Postby Josef Templ » Tue Dec 06, 2016 7:49 am

Look into module Kernel.
The heap allocation has not been changed between 1.6 and 1.7.1-a1,
as far as I remember. I guess that there is a limit
imposed by Windows or wine. If this is the case, we cannot do
much about it.

- Josef
Josef Templ
 
Posts: 215
Joined: Tue Sep 17, 2013 6:50 am

Re: Heap limit

Postby Robert » Tue Dec 06, 2016 9:43 am

We used to be able to allocate about 1500 MBytes (after I asked Oms to increase the heap size many years ago).

Then we hit problems, the amount reduced to about 800. This problem was specific to work machines, peoples home machines were still ok. The problem did not seem to be related to any particular version of Windows; it was a real mystery, and a major inconvenience for years.

Finally we found it was an incompatibility with the program "Lumensions". It is fixed in some recent versions of Lumensions, but our IT people like to use the incompatible versions.

Maybe you are seeing this, or some other, external incompatibility.

Below is our test program; it just allocates 10 MByte chunks. (I am talking about BlackBox 1.6, but I expect 1.7 is the same.)

Code: Select all
MODULE  TestAlloc;   (*  Date   :  26  August  2013   *)
   (*  Author :  Robert D Campbell  *)

IMPORT  Out;

CONST
  size  =  10 * 1024 * 1024;   (*  10  MByte  *)

TYPE
  Blob  =  POINTER  TO  ARRAY  OF  BYTE;

  Link  =  POINTER  TO  RECORD
             next  :  Link;
             blob  :  Blob
           END;

PROCEDURE  Do*;
  VAR
    base, link  :  Link;
    cnt         :  INTEGER;
  BEGIN
    NEW (base); cnt  :=  0;
    LOOP
      NEW (link); NEW (link.blob, size);
      IF (link.blob = NIL)  OR  (cnt  =  500)  THEN  EXIT  END;
      link.next  :=  base.next; base.next  :=  link; INC (cnt);

      Out.Char ('+');   
      IF     cnt  MOD  50  =  0  THEN  Out.Ln
      ELSIF  cnt  MOD   5  =  0  THEN  Out.Char (' ')  END;
    END;
    Out.Ln;
    Out.String ('Blobs allocated :'); Out.Int (cnt, 5); Out.Ln
   END  Do;

PROCEDURE  DoMany*;
  VAR
    k  :  INTEGER;
  BEGIN
    FOR  k  :=  1  TO  100  DO  Do  END
  END  DoMany;

END  TestAlloc.
 
   TestAlloc.Do
    TestAlloc.DoMany
User avatar
Robert
 
Posts: 112
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: Heap limit

Postby Robert » Tue Dec 06, 2016 9:15 pm

Robert wrote:(I am talking about BlackBox 1.6, but I expect 1.7 is the same.)
Just run this on BlackBox 1.7, Windows 7 Professional, 64-bit, AMD processor with 4 GByte.
It also can allocate 1500 MByte.
User avatar
Robert
 
Posts: 112
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Next

Return to BlackBox Framework

Who is online

Users browsing this forum: No registered users and 2 guests

cron