Page 1 of 2

Heap limit

Posted: Thu Dec 01, 2016 6:31 pm
by dmaksimiuk
Dear All,
In the trivial code below I am crating a linked list of buffers. At first I was thinking that the limit on the number of allocated elements depends on the available RAM memory (in 32bit address space). To my surprise, the BBox environment started "yelling" at me when the number of allocated buffers was (it varies a bit):
Num elements in the list= 316623
Num of allocated bytes= 666174792

There some questions:
1. Why we cannot go beyond a list that can store more than let say 1GB worth of data?
2. When the garbage collector should claim all the memory back? After assignments
First := NIL;
Head := NIL;
CurrEl := NIL;
the environment still showed allocated memory around 670_000_000 bytes. Only unloading the module reclaim the memory. In addition, the Win7 Task Manager shows more than 674_000_000 bytes allocated to the BBox even if internally the heap is at level of ~500k.

BTW , does anybody has any experience with using a DLL from BBox that starts a task (let say an asynchronous process) , and still is able to communicate with a program written in CP ?

Cheers,
Darek

Code: Select all

MODULE dmTestsTestList;


	IMPORT

		Log := StdLog;

	CONST
		BuffSize = 2100;

	TYPE

		tData* = ARRAY BuffSize OF BYTE;

		tPtrNode = POINTER TO tNode;
		tNode = RECORD
			Buffer*: tData;
			Next: tPtrNode;
		END; (* record *)


	VAR
		First, Head, CurrEl: tPtrNode;
		NumElemCnt: INTEGER;
		RetriveFirstElem: BOOLEAN;


		(* -------------------------------------------------------------------------------------------------------------------------------------------- *)
	PROCEDURE Init* ();
	BEGIN
		NumElemCnt := 0;
		RetriveFirstElem := FALSE;
		First := NIL; Head := NIL; CurrEl := NIL;
		Log.String("Node  size is: "); Log.Int(SIZE(tNode)); Log.Ln();

	END Init;

	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)

	PROCEDURE StoreData* (v: tData);
	BEGIN
		IF NumElemCnt = 0 THEN
			NEW(CurrEl);
			IF CurrEl = NIL THEN			
				HALT(100);
			END;
			First := CurrEl;
			Head := CurrEl;
			CurrEl.Buffer := v;
			CurrEl.Next := NIL;
			INC(NumElemCnt);
		ELSE
			NEW(CurrEl);
			IF CurrEl = NIL THEN
    			 Log.Ln(); Log.String("Num elements in the list="); Log.Int(NumElemCnt); Log.Ln();
                     Log.String("Num of allocated bytes="); Log.Int(NumElemCnt * SIZE(tNode)); Log.Ln();
				HALT(100);
			END;
			CurrEl.Buffer := v;
			CurrEl.Next := NIL;
			Head.Next := CurrEl;
			Head := CurrEl;
			INC(NumElemCnt);
		END; (* if *)
	END StoreData;

	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)
	PROCEDURE IsEmpty* (): BOOLEAN;
	BEGIN
		RETURN (Head = NIL);
	END IsEmpty;
	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)

	PROCEDURE RetriveData* (VAR v: tData);
	BEGIN
		IF ~RetriveFirstElem THEN
			Head := First;
			v := Head.Buffer;
			Head := Head.Next;
			RetriveFirstElem := TRUE;
			First := NIL (* added for testing *)
		ELSE
			IF Head # NIL THEN
				v := Head.Buffer;
				Head := Head.Next;
			END;
		END; (* if *)
	END RetriveData;

	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)
	PROCEDURE GetNumElements* (): INTEGER;
	BEGIN
		RETURN(NumElemCnt);
	END GetNumElements;

	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)

	PROCEDURE RunTest*;
		CONST
			NumFiles = 500000;
		VAR
			i: INTEGER;
			data: tData;
	BEGIN
		Init();
		Log.String("Adding data to the list ...");
		FOR i := 0 TO LEN(data) - 1 DO data[i] := 1; END;
		FOR i := 1 TO NumFiles DO
			StoreData(data);
		END; (* for *)
		Log.String("done."); Log.Ln();
		Log.String("Num elements  in the list"); Log.Int(GetNumElements()); Log.Ln();
		WHILE ~IsEmpty() DO
			RetriveData(data);
		END; (* while *)
		First := NIL;
		Head := NIL;
		CurrEl := NIL;
		Log.String("DONE "); Log.Ln();
	END RunTest;
	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)

BEGIN
END dmTestsTestList.




 dmTestsTestList.RunTest

 DevDebug.Unload dmTestsTestList

Re: Heap limit

Posted: Mon Dec 05, 2016 12:01 pm
by Josef Templ
In my tests I was able to allocate 1.08 GB (108 x 10MB) on a Vista machine with 2GB Ram
and 1.39 GB (139 x 10MB) on a Win10 machine with 8GB Ram.
The limit seems to depend on the underlying platform.

- Josef

Re: Heap limit

Posted: Mon Dec 05, 2016 1:05 pm
by dmaksimiuk
Hi Josef,
thanks for testing it. My machine runs 64bit Win10 with 16GB RAM and yet, I got (so far) the worst results :roll: .

Cheers,
Darek

Re: Heap limit

Posted: Mon Dec 05, 2016 7:20 pm
by Josef Templ
For better comparison of our results, here is my test program.
It allocates a sequence of 10MB blocks until it gets no more data
and then outputs the number of blocks.

Code: Select all

MODULE TestHeapsize;

   IMPORT Log := StdLog;

   TYPE
      Data = POINTER TO RECORD
		data: ARRAY 10000000 OF BYTE;
         next: Data;
      END;

   PROCEDURE Run*;
      VAR
         i: INTEGER;
         data, root: Data;
   BEGIN
      NEW(data); root := NIL; i := 0;
      WHILE data # NIL DO INC(i); data.next := root; root := data; NEW(data) END;
      Log.String("nof allocated 10MB blocks: "); Log.Int(i); Log.Ln()
   END Run;

BEGIN
END TestHeapsize.

TestHeapsize.Run
- Josef

Re: Heap limit

Posted: Mon Dec 05, 2016 8:30 pm
by dmaksimiuk
Hi Josef,
the results of running your test program is:
nof allocated 10MB blocks: 67
which is roughly similar to the results I got with small blocks. Could be something about my version of the BBox system?
I am running 1.7-a1 Build:480, 2016.03.22.

Cheers,
Darek

Re: Heap limit

Posted: Mon Dec 05, 2016 8:40 pm
by dmaksimiuk
... just checked BBox 1.6 and the result is:
nof allocated 10MB blocks: 139

So, it could something magic about the version I am using.

Cheers,
Darek

Re: Heap limit

Posted: Mon Dec 05, 2016 9:51 pm
by Ivan Denisov
My test with 1.7.1-a1 version in Wine (Ubuntu) gives only 60 blocks also. So we should think about this issue now!...
1.6 version gives 80 blocks for me.

Re: Heap limit

Posted: Tue Dec 06, 2016 7:49 am
by Josef Templ
Look into module Kernel.
The heap allocation has not been changed between 1.6 and 1.7.1-a1,
as far as I remember. I guess that there is a limit
imposed by Windows or wine. If this is the case, we cannot do
much about it.

- Josef

Re: Heap limit

Posted: Tue Dec 06, 2016 9:43 am
by Robert
We used to be able to allocate about 1500 MBytes (after I asked Oms to increase the heap size many years ago).

Then we hit problems, the amount reduced to about 800. This problem was specific to work machines, peoples home machines were still ok. The problem did not seem to be related to any particular version of Windows; it was a real mystery, and a major inconvenience for years.

Finally we found it was an incompatibility with the program "Lumensions". It is fixed in some recent versions of Lumensions, but our IT people like to use the incompatible versions.

Maybe you are seeing this, or some other, external incompatibility.

Below is our test program; it just allocates 10 MByte chunks. (I am talking about BlackBox 1.6, but I expect 1.7 is the same.)

Code: Select all

MODULE  TestAlloc;	(*  Date   :  26  August  2013   *)
	(*  Author :  Robert D Campbell  *)

IMPORT  Out;

CONST
  size  =  10 * 1024 * 1024;	(*  10  MByte  *)

TYPE
  Blob  =  POINTER  TO  ARRAY  OF  BYTE;

  Link  =  POINTER  TO  RECORD
             next  :  Link;
             blob  :  Blob
           END;

PROCEDURE  Do*;
  VAR
    base, link  :  Link;
    cnt         :  INTEGER;
  BEGIN
    NEW (base); cnt  :=  0;
    LOOP
      NEW (link); NEW (link.blob, size);
      IF (link.blob = NIL)  OR  (cnt  =  500)  THEN  EXIT  END;
      link.next  :=  base.next; base.next  :=  link; INC (cnt);

      Out.Char ('+');    
      IF     cnt  MOD  50  =  0  THEN  Out.Ln
      ELSIF  cnt  MOD   5  =  0  THEN  Out.Char (' ')  END;
    END;
    Out.Ln;
    Out.String ('Blobs allocated :'); Out.Int (cnt, 5); Out.Ln
   END  Do;

PROCEDURE  DoMany*;
  VAR
    k  :  INTEGER;
  BEGIN
    FOR  k  :=  1  TO  100  DO  Do  END
  END  DoMany;

END  TestAlloc.
 
   TestAlloc.Do
    TestAlloc.DoMany

Re: Heap limit

Posted: Tue Dec 06, 2016 9:15 pm
by Robert
Robert wrote:(I am talking about BlackBox 1.6, but I expect 1.7 is the same.)
Just run this on BlackBox 1.7, Windows 7 Professional, 64-bit, AMD processor with 4 GByte.
It also can allocate 1500 MByte.