Heap limit

All except GUI problems
dmaksimiuk
Posts: 28
Joined: Sun Jun 14, 2015 3:56 pm
Location: Delft, The Netherlands

Heap limit

Post by dmaksimiuk »

Dear All,
In the trivial code below I am crating a linked list of buffers. At first I was thinking that the limit on the number of allocated elements depends on the available RAM memory (in 32bit address space). To my surprise, the BBox environment started "yelling" at me when the number of allocated buffers was (it varies a bit):
Num elements in the list= 316623
Num of allocated bytes= 666174792

There some questions:
1. Why we cannot go beyond a list that can store more than let say 1GB worth of data?
2. When the garbage collector should claim all the memory back? After assignments
First := NIL;
Head := NIL;
CurrEl := NIL;
the environment still showed allocated memory around 670_000_000 bytes. Only unloading the module reclaim the memory. In addition, the Win7 Task Manager shows more than 674_000_000 bytes allocated to the BBox even if internally the heap is at level of ~500k.

BTW , does anybody has any experience with using a DLL from BBox that starts a task (let say an asynchronous process) , and still is able to communicate with a program written in CP ?

Cheers,
Darek

Code: Select all

MODULE dmTestsTestList;


	IMPORT

		Log := StdLog;

	CONST
		BuffSize = 2100;

	TYPE

		tData* = ARRAY BuffSize OF BYTE;

		tPtrNode = POINTER TO tNode;
		tNode = RECORD
			Buffer*: tData;
			Next: tPtrNode;
		END; (* record *)


	VAR
		First, Head, CurrEl: tPtrNode;
		NumElemCnt: INTEGER;
		RetriveFirstElem: BOOLEAN;


		(* -------------------------------------------------------------------------------------------------------------------------------------------- *)
	PROCEDURE Init* ();
	BEGIN
		NumElemCnt := 0;
		RetriveFirstElem := FALSE;
		First := NIL; Head := NIL; CurrEl := NIL;
		Log.String("Node  size is: "); Log.Int(SIZE(tNode)); Log.Ln();

	END Init;

	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)

	PROCEDURE StoreData* (v: tData);
	BEGIN
		IF NumElemCnt = 0 THEN
			NEW(CurrEl);
			IF CurrEl = NIL THEN			
				HALT(100);
			END;
			First := CurrEl;
			Head := CurrEl;
			CurrEl.Buffer := v;
			CurrEl.Next := NIL;
			INC(NumElemCnt);
		ELSE
			NEW(CurrEl);
			IF CurrEl = NIL THEN
    			 Log.Ln(); Log.String("Num elements in the list="); Log.Int(NumElemCnt); Log.Ln();
                     Log.String("Num of allocated bytes="); Log.Int(NumElemCnt * SIZE(tNode)); Log.Ln();
				HALT(100);
			END;
			CurrEl.Buffer := v;
			CurrEl.Next := NIL;
			Head.Next := CurrEl;
			Head := CurrEl;
			INC(NumElemCnt);
		END; (* if *)
	END StoreData;

	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)
	PROCEDURE IsEmpty* (): BOOLEAN;
	BEGIN
		RETURN (Head = NIL);
	END IsEmpty;
	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)

	PROCEDURE RetriveData* (VAR v: tData);
	BEGIN
		IF ~RetriveFirstElem THEN
			Head := First;
			v := Head.Buffer;
			Head := Head.Next;
			RetriveFirstElem := TRUE;
			First := NIL (* added for testing *)
		ELSE
			IF Head # NIL THEN
				v := Head.Buffer;
				Head := Head.Next;
			END;
		END; (* if *)
	END RetriveData;

	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)
	PROCEDURE GetNumElements* (): INTEGER;
	BEGIN
		RETURN(NumElemCnt);
	END GetNumElements;

	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)

	PROCEDURE RunTest*;
		CONST
			NumFiles = 500000;
		VAR
			i: INTEGER;
			data: tData;
	BEGIN
		Init();
		Log.String("Adding data to the list ...");
		FOR i := 0 TO LEN(data) - 1 DO data[i] := 1; END;
		FOR i := 1 TO NumFiles DO
			StoreData(data);
		END; (* for *)
		Log.String("done."); Log.Ln();
		Log.String("Num elements  in the list"); Log.Int(GetNumElements()); Log.Ln();
		WHILE ~IsEmpty() DO
			RetriveData(data);
		END; (* while *)
		First := NIL;
		Head := NIL;
		CurrEl := NIL;
		Log.String("DONE "); Log.Ln();
	END RunTest;
	(* -------------------------------------------------------------------------------------------------------------------------------------------- *)

BEGIN
END dmTestsTestList.




 dmTestsTestList.RunTest

 DevDebug.Unload dmTestsTestList
Josef Templ
Posts: 262
Joined: Tue Sep 17, 2013 6:50 am

Re: Heap limit

Post by Josef Templ »

In my tests I was able to allocate 1.08 GB (108 x 10MB) on a Vista machine with 2GB Ram
and 1.39 GB (139 x 10MB) on a Win10 machine with 8GB Ram.
The limit seems to depend on the underlying platform.

- Josef
dmaksimiuk
Posts: 28
Joined: Sun Jun 14, 2015 3:56 pm
Location: Delft, The Netherlands

Re: Heap limit

Post by dmaksimiuk »

Hi Josef,
thanks for testing it. My machine runs 64bit Win10 with 16GB RAM and yet, I got (so far) the worst results :roll: .

Cheers,
Darek
Josef Templ
Posts: 262
Joined: Tue Sep 17, 2013 6:50 am

Re: Heap limit

Post by Josef Templ »

For better comparison of our results, here is my test program.
It allocates a sequence of 10MB blocks until it gets no more data
and then outputs the number of blocks.

Code: Select all

MODULE TestHeapsize;

   IMPORT Log := StdLog;

   TYPE
      Data = POINTER TO RECORD
		data: ARRAY 10000000 OF BYTE;
         next: Data;
      END;

   PROCEDURE Run*;
      VAR
         i: INTEGER;
         data, root: Data;
   BEGIN
      NEW(data); root := NIL; i := 0;
      WHILE data # NIL DO INC(i); data.next := root; root := data; NEW(data) END;
      Log.String("nof allocated 10MB blocks: "); Log.Int(i); Log.Ln()
   END Run;

BEGIN
END TestHeapsize.

TestHeapsize.Run
- Josef
dmaksimiuk
Posts: 28
Joined: Sun Jun 14, 2015 3:56 pm
Location: Delft, The Netherlands

Re: Heap limit

Post by dmaksimiuk »

Hi Josef,
the results of running your test program is:
nof allocated 10MB blocks: 67
which is roughly similar to the results I got with small blocks. Could be something about my version of the BBox system?
I am running 1.7-a1 Build:480, 2016.03.22.

Cheers,
Darek
dmaksimiuk
Posts: 28
Joined: Sun Jun 14, 2015 3:56 pm
Location: Delft, The Netherlands

Re: Heap limit

Post by dmaksimiuk »

... just checked BBox 1.6 and the result is:
nof allocated 10MB blocks: 139

So, it could something magic about the version I am using.

Cheers,
Darek
User avatar
Ivan Denisov
Posts: 362
Joined: Tue Sep 17, 2013 12:21 am
Location: Krasnoyarsk, Russia

Re: Heap limit

Post by Ivan Denisov »

My test with 1.7.1-a1 version in Wine (Ubuntu) gives only 60 blocks also. So we should think about this issue now!...
1.6 version gives 80 blocks for me.
Josef Templ
Posts: 262
Joined: Tue Sep 17, 2013 6:50 am

Re: Heap limit

Post by Josef Templ »

Look into module Kernel.
The heap allocation has not been changed between 1.6 and 1.7.1-a1,
as far as I remember. I guess that there is a limit
imposed by Windows or wine. If this is the case, we cannot do
much about it.

- Josef
User avatar
Robert
Posts: 177
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: Heap limit

Post by Robert »

We used to be able to allocate about 1500 MBytes (after I asked Oms to increase the heap size many years ago).

Then we hit problems, the amount reduced to about 800. This problem was specific to work machines, peoples home machines were still ok. The problem did not seem to be related to any particular version of Windows; it was a real mystery, and a major inconvenience for years.

Finally we found it was an incompatibility with the program "Lumensions". It is fixed in some recent versions of Lumensions, but our IT people like to use the incompatible versions.

Maybe you are seeing this, or some other, external incompatibility.

Below is our test program; it just allocates 10 MByte chunks. (I am talking about BlackBox 1.6, but I expect 1.7 is the same.)

Code: Select all

MODULE  TestAlloc;	(*  Date   :  26  August  2013   *)
	(*  Author :  Robert D Campbell  *)

IMPORT  Out;

CONST
  size  =  10 * 1024 * 1024;	(*  10  MByte  *)

TYPE
  Blob  =  POINTER  TO  ARRAY  OF  BYTE;

  Link  =  POINTER  TO  RECORD
             next  :  Link;
             blob  :  Blob
           END;

PROCEDURE  Do*;
  VAR
    base, link  :  Link;
    cnt         :  INTEGER;
  BEGIN
    NEW (base); cnt  :=  0;
    LOOP
      NEW (link); NEW (link.blob, size);
      IF (link.blob = NIL)  OR  (cnt  =  500)  THEN  EXIT  END;
      link.next  :=  base.next; base.next  :=  link; INC (cnt);

      Out.Char ('+');    
      IF     cnt  MOD  50  =  0  THEN  Out.Ln
      ELSIF  cnt  MOD   5  =  0  THEN  Out.Char (' ')  END;
    END;
    Out.Ln;
    Out.String ('Blobs allocated :'); Out.Int (cnt, 5); Out.Ln
   END  Do;

PROCEDURE  DoMany*;
  VAR
    k  :  INTEGER;
  BEGIN
    FOR  k  :=  1  TO  100  DO  Do  END
  END  DoMany;

END  TestAlloc.
 
   TestAlloc.Do
    TestAlloc.DoMany
User avatar
Robert
Posts: 177
Joined: Sat Sep 28, 2013 11:04 am
Location: Edinburgh, Scotland

Re: Heap limit

Post by Robert »

Robert wrote:(I am talking about BlackBox 1.6, but I expect 1.7 is the same.)
Just run this on BlackBox 1.7, Windows 7 Professional, 64-bit, AMD processor with 4 GByte.
It also can allocate 1500 MByte.
Post Reply