Record Loading in 4th Dimension v6.5, Part II

By Tim Tonooka, ACI Technical Support

Technical Note 99-40

Technical Notes for Technical Notes for 99-09-September 1999

In older versions of 4th Dimension, the mechanics of record loading were generally taken for granted. With the addition of newer features such as the BLOB (Binary Large OBjects) data type, there are now more reasons why your database might include records that are very large in size. For this reason, the details of record loading in 4D v6.5 deserve a closer look, particularly as this concerns large records and memory limitations. Understanding 4D's record loading behavior can also help you optimize performance by minimizing network traffic, which is especially important over a wide area network.

This is the second of a three-part technical note. Topics covered in Part 1 included:

A quick outline of how 4D uses memory.

A detailed explanation of how 4D works with records.

The definition of record loading.

The "On loading a record" trigger.

How 4D handles record loading when memory is limited.

The differences in record loading behavior between 4D standalone and 4D Client/Server.

The memory requirements for loading and handling large records.

Part 2 reveals how to configure 4D's memory, featuring:

An outline of how 4D uses memory.

How to find the size of the data cache and the user heap.

How to adjust the size of the data cache and user heap areas.

Tips on proportioning memory for optimal handling of large records.

Part 3 concludes the technical note with:

Diagnostic techniques for analyzing record loading.

A list of the commands and activities that cause records to be loaded.

Full details on how each of these specific operations impacts record loading.

Strategies for managing record loading to optimize performance and reliability.

Some terms used throughout this technical note are defined here. The memory that 4D uses has three main sections. In this technical note, these are referred to as Engine Memory, the Data Cache, and the User Heap.

Engine Memory -- When a 4D application starts, it loads portions of its engine (program code and certain resources) into memory. The majority of these objects remain locked in memory as long as the application is running. That memory cannot be reused for anything else. (Any references to engine memory in this technical note apply to all 4D applications, not just a structure compiled and merged with 4D Engine.)

The Data Cache -- A single large block of memory is allocated at startup for use as the data cache. The data cache holds copies of records that have been read from the hard disk, or are about to be written to the hard disk. The other objects held by the data cache are the allocation bit map, address tables, record address pages (primary and secondary), index pages (primary and secondary), current selections, and named selections.

The User Heap -- This consists of the rest of the 4D application heap that is not used for engine memory or the data cache. The user heap holds objects that the database is currently using. These objects are loaded into the user heap when they are needed, and can be purged when they are not. A copy of the current record for each table in each process can be stored here. These copies are used for editing, as opposed to the static copies of records held in the data cache. The user heap holds structure objects such as forms, methods, and lists. Also held in the user heap are process stacks, variables, arrays, sets, plug-in code, plug-in memory, and resources. Examples of resources that can be loaded into the user heap include network components, cursors, icons, menus, fonts, patterns, string resources, TRIC resources, and pictures such as the splash screen graphic.

If you have enabled the "Accelerated Screen Updates" setting in Database Properties, a portion of the user heap will be used as an offscreen bitmap area, enabling faster screen redraws. The larger the resolution of your monitor and the higher its color bit depth setting, the more memory is required for this offscreen bitmap. It can be 3 MB or more in size.

Current selections are held in the data cache. If there is not enough space left in the data cache to hold the selection, part of it will be paged to disk in a temporary ".SEL" file. 4D uses this virtual memory technique when necessary.

Named selections are held in the data cache. Before 4D v6.5, a named selection could be temporarily moved to the user heap to perform certain operations on it, then it would be moved back to the data cache. In 4D v6.5, the named selection stays in the data cache the whole time.

If there is not enough space left in the data cache, new named selections will be held in the user heap. If there is also not enough space in the user heap, you will get the error message, "Not enough room in memory. The named selection could not be created."

When you start a database, the amount of memory that 4D will allocate for the data cache depends not only on the memory settings set in Database Properties or with Customizer Plus, but also on the amount of memory the operating system has available for your 4D application to open.

The actual size of the data cache in a running database is displayed in the Watch page of the Runtime Explorer and in the Watch pane in the upper-left corner of the Debugger. The "Cache Statistics" section of the Watch listing displays a value such as "112 bytes / 9 636 Kb (0%), 6 handles." In that example, the size of the data cache is 9,636K, of which 112 bytes are currently in use, to hold 6 objects.

In both the Runtime Explorer and the Debugger, an item in the "Information" section of the Watch listing displays the amount of "Free Memory," measured in kilobytes. This is the amount of unused space in the user heap. (Displaying the Runtime Explorer or the Debugger uses up a small amount of memory in the user heap. This can be as little as 26K).

One of the most useful items in the Runtime Explorer's Information section is "Memory Load." This is based on the number of times per second 4D has to call an operating system routine to purge memory. If 4D has to call this purge routine very frequently, then the memory load is poor.

To programmatically measure the size of the data cache in a running database, use the Get database parameter command with the value 9 for the selector parameter. It returns the size, measured in bytes. This command can be used from 4D Client to find the size of the data cache on the server.

If you want to find the amount of unused space in the user heap programmatically, the ACI Pack AP AVAILABLE MEMORY command returns that value in its third parameter. This is also shown as "Free Memory" in the Runtime Explorer's Information section. But because the amount of free memory can fluctuate frequently, the two values may not be exactly the same unless you update both at exactly the same time. Also, having a plug-in loaded reduces the amount of free memory.

You might want your code to check the amount of free memory when the database first opens, to alert the user if memory is excessively low. For anything else, programmatically measuring the free memory is of limited practical value, because between the moment when your code takes the measurement and a split-second later when your code might try to act on that information, the actual amount of free memory may have changed substantially, especially in a multi-user database, due to the actions of another process or user.

In 4D Server, the server window displays useful information about memory and the data cache. The upper portion of this window shows the amount of "Total Memory" and "Cache Memory," along with thermometers showing "Activity" and the "Cache Hit Ratio."

"Total Memory" shows the amount of the computer's memory that 4D Server has allocated at the moment.

On Mac OS, "Total Memory" will be about 167 to 213K less than the amount entered as the "Preferred size" in the Macintosh "Get Info" dialog (unless that much memory is not available). Since memory on Mac OS is allocated all at once when the 4D application is started, the "Total Memory" size shown in the 4D Server window does not change while the server is running.

On Windows, when you first start 4D Server, the "Total Memory" value can be very low. You should not be alarmed. This value can start low because Windows dynamically allocates memory. "Total Memory" shows the total size of the memory blocks that 4D Server is actually using (excluding the memory used to hold the 4D engine and the data cache). As the server is used, "Total Memory" can increase. Any changes to the "Total Memory" value occur in increments equal to the Block Size set in Database Properties.

If you want to know the total amount of memory that your 4D Server is using on Windows NT, use the Windows NT Task Manager. In the Task Manager window, click the "Processes" tab to see a list of all the processes running on the computer. Look for the 4D Server entry in this list. The information shown in the listing includes the total memory used by each running application.

As an example, suppose the Application Main Memory setting is 40 blocks of 1,024K each. This is a total of 40,960K. The 4D Server 6.5.2 window shows 1,773K for "Total Memory" and 8,192K for "Cache Memory," and the Runtime Explorer shows 40,896K of "Free Memory." This is shown in the following screen. With these settings, the Windows NT Task Manager would show 4D Server memory usage to be about 14,480K at startup. (These memory settings are only cited to illustrate this example. They are not a recommendation.)

In the 4D Server window, "Cache Memory" shows the size of the data cache. The amount shown is the full size of the data cache, not just the portion of that is currently in use.

The "Activity" thermometer shows the current level of server activity on the network. The more requests sent to the server, the higher this reading will be.

The "Cache Hit Ratio" thermometer shows the ratio between the number of times 4D actually finds objects in the data cache, compared to the total number of times it tries to access objects in the data cache. (This also appears as "Global Hit" in the "Cache Statistics" section of the Runtime Explorer's Watch page and the Debugger's Watch pane.) For example, after you open a database in the Custom Menus environment, the cache hit ratio is zero. The first time you load a record, the cache hit ratio becomes 25%. This means that one out of four attempts to find an object in the cache was successful. Why are there four attempts the first time you load a record, instead of just one? Because to get your record into the cache after the database has been freshly opened, other things need to be loaded into the cache as well, for example, record address pages. If you then try to load the same record again, the cache hit ratio increases to 40%, which means that since the database has been opened, five tries have been made to find an object in the cache, of which two were successful. This time, it only took one try to find the record in the cache, because the record was already there from when we'd loaded it before. The next attempt to load the same record sends the cache hit ratio to 50% -- three out of six tries. Next it will go to 57% -- four out of seven tries. In this manner, the ratio could steadily increase to nearly 100%.

If you click the "Cache Hit Ratio" thermometer, a window displays five thermometers showing how the data cache is being used. For each type of data that can be kept in the cache, a thermometer indicates how much of the data is present in the cache. Higher readings are better, because accessing that type of data will then be less likely to require accessing the disk.

In Windows NT, the Task Manager displays statistics about memory usage for each process running on the computer, including your 4D application. The Task Manager can be accessed from the contextual pop-up menu that appears when you right-click on the Windows task bar. The Task Manager "Processes" tab page can display information such as Memory Usage, Virtual Memory size, etc.

More statistics about a process running on Windows NT can be graphed and analyzed in the Performance Monitor. In the Windows Start menu, the Performance Monitor is listed in the Administrative Tools hierarchical menu located in the Programs section.

In the "About This Computer" dialog accessed from the Apple menu of the Macintosh Finder, each running application is listed, along with a thermometer graph of the memory usage within that application. Starting from the left, it shows the portion of the application's memory that is currently in use. In the thermometer for your 4D application, the filled portion of this graph includes the entire data cache, regardless of how empty or full the data cache actually is. The empty portion inside the thermometer is equal to the amount of Free Memory shown in the 4D Runtime Explorer.

To the operating system, the data cache is one big block of memory within the 4D memory area that is marked as being in use. This is why the entire data cache area shows up in the used portion of the thermometer. 4D has its own internal cache manager process that takes care of moving things in and out of this data cache memory block.

Giving your 4D application more memory is usually better, but it is possible to reach a point beyond which adding more memory will actually slow down your database. This is because when objects are loaded into memory, either in the data cache or in the user heap, 4D has to maintain a list of all the objects. 4D stores as many objects as it can in memory, so it does not have to go back to the disk in case they are needed again. 4D will keep these objects in memory until it runs out of space to load new objects. (At that point it has to purge some of the unused objects to make space for new ones.) As a result, if you have a tremendous amount of memory, it can end up holding a huge number of objects, most of which are unlikely to be reused anytime soon. The processing overhead of maintaining a huge list of objects can negate the normal benefits of additional memory.

You should not allocate all of the memory not used by the operating system to your 4D application, because that will not leave enough room for stacks the operating system may need to maintain in memory. Also, on Windows NT, the operating system may need as much as 64 MB of RAM outside the operating system to maintain its own disk cache. (Your data goes from the 4D disk cache to the NT disk cache, before it gets written to the hard disk.)

The way that you go about setting the amounts of memory that will be used for the data cache and the user heap is different on Windows and Mac OS. Also, on Mac OS, there are two ways of handling these settings, depending on whether you are using the "new memory allocation scheme."

Though you can programmatically read the size of the data cache, you cannot set the size of the data cache via code.

After changing any memory settings, you need to restart the 4D application for them to take effect. You can use the Runtime Explorer to verify that the changes are in effect.

On Windows, applications allocate their memory dynamically, on an as-needed basis. An application starts out with a base amount of memory, then grabs more memory as the need arises. In comparison, Mac OS applications allocate the memory they are going to use all at once, at the time they start up.

All the 4D memory settings used on Windows can be accessed in the "System settings" page of the Database Properties dialog (known in 4D v6 as the "Tune Up" page). In 4D Server, you can select Database Properties from the File menu. In 4D standalone, you need to be in the Design environment, where you select Database Properties from the File menu.

The three settings for configuring memory on Windows are:

Maximum Cache, located in the "Database Cache Memory" section of the page. The allowed values range from a minimum of 500K to a maximum of 3,276,700K (3.1 GB).

Used Blocks, located in the "Application Main Memory" section of the page. The allowed values range from a minimum of 2 blocks to a maximum of 256 blocks. The default value is 10.

Block Size, located in the "Application Main Memory" section of the page. The allowed values range from a minimum of 500K to a maximum of 64,000K (62.5 MB). The default value is 1,024K.

The data cache size can also be set by using Customizer Plus to edit the "Properties" resource. The Application Main Memory settings can also be changed with Customizer Plus, by editing the "Preferences" or "Main Memory" resources.

The amount of memory that will be used for the data cache is allocated all at once, when the application is started. "Maximum Cache" sets the amount of memory, measured in kilobytes, that will be used for the data cache.

4D dynamically allocates memory for the user heap. The initial block that is allocated is relatively small (749K on 4D Server 6.5.2). When 4D needs to store more objects in memory, it grabs more blocks of memory for the user heap. The number of "Used Blocks" multiplied by the "Block Size" sets the maximum amount of additional memory that can be allocated to the user heap after the initial block. For example, on 4D Server 6.5.2, if "Used Blocks" is set to 10 and the "Block Size" is 1,024K, the maximum amount of memory that could be allocated to the user heap would be 10,240K plus the initial 749K block, a total of 10,989K.

The "Block Size" setting actually specifies the minimum block size. If 4D needs to load an object that is larger then the specified "Block Size," it will allocate a memory block large enough to accommodate the object. When 4D allocates a larger block, the size will be an increment of the specified block size. Otherwise, any new memory blocks allocated will be the size specified in "Block Size."

Unlike the application "Preferred size" in the Macintosh "Get Info" dialog, the Windows "Application Main Memory" does not set the total amount of memory that will be used by your 4D application. It only sets the size limit of the user heap. In addition to the user heap, 4D uses a separate block of memory for the data cache. For example, on 4D Server 6.5.2, if the number of "Used Blocks" is 10, the "Block Size" is 1,024K, and "Maximum Cache" is 10,000K, then the combined total of the user heap and data cache could reach a maximum of 20,989K. In addition to that, portions of the 4D application's executable file (such as "4D.exe" or "4DServer.exe") are loaded into memory. Another 3 MB of memory holds the 4D application's resource file (such as "4D.rsr" or "4DServer.rsr"). Some other smaller items are also loaded into memory.

A frequently-asked question is, when you want to increase the Application Main Memory, should you increase the number of blocks, the size of the block, or both?

The answer will depend on the specific database. In general, big blocks are preferred, since they are easier for 4D to handle. Also, the process of asking the OS memory manager for another block is a time-consuming operation, so it is better if you do not have to do that as often.

When you divide memory into multiple blocks, the blocks doled out by the operating system will not necessarily be adjacent to each other in memory, so you may not get the benefit of having contiguous memory among those blocks. For example, on Windows NT, the operating system can place small blocks of stack memory between the blocks of memory it allocates to 4D, which can sometimes cause 4D to not have contiguous memory among all of its blocks.

If you have other applications besides 4D running on the same computer, as all these applications run, the memory blocks that the operating system doles out to the other applications could separate the blocks of memory it gives to 4D, which would prevent you from having contiguous memory among the 4D memory blocks.

If you reach the point where all the blocks that could fit within the Application Main Memory size limit have already been allocated and are in use, if 4D needs to fit a large object in memory, the chances of finding enough contiguous space will be less if the blocks are smaller. If practical, try to make your block size bigger than the biggest BLOB or record you will be handling, to create contiguous memory areas in case you need them later.

As a general rule, block sizes of 7 to 15 MB are best. Blocks should not be smaller than 5 to 7 MB. For example, if you need 40 MB for Application Main Memory, 4 blocks of 10 MB are recommended. It is difficult to write generic rules for this. It really depends on the specifics of what you are doing; for example, if you are using sets, and how you are using them. Small details in programming could change the equation tremendously.

To effectively change the Application Main Memory settings, you have to understand the different places where they are stored, and the hierarchy of priority these locations have when 4D starts a database.

The application main memory settings can be stored in the "Preferences" resource, in these locations:

An interpreted structure file (".4DB").

A compiled structure file (".4DC").

A structure file compiled and merged with 4D Engine (".EXE").

The ".4DC" file in the "Distrib" folder of a compiled and merged database.

The 4D Engine itself ("4DEngine.4de").

4th Dimension ("4D.exe").

4D Server ("4DServer.exe").

4D Client ("4DClient.exe").

4D Runtime ("4DRuntime.exe").

4D Runtime Classic ("4DRunClassic.exe").

The "Preferences" resource can be edited in Customizer Plus, as shown:

The Application Main Memory settings can also be stored in the "Main Memory" resource, which can be created in the preference files for 4D applications:

"4DV6Prf.RSR" -- Used by any database on 4D or 4D Server, and by 4D Client and 4D Tools.

"RunV6Prf.RSR" -- Used by any database on 4D Runtime or 4D Runtime Classic.

"EngV6Prf.RSR" -- Used by any database compiled and merged with 4D Engine.

These preferences files are all stored in the ACI folder inside the Windows or Winnt folder on your boot hard disk. If they are moved to another location on the hard disk, they will not be recognized. When a 4D application is run, if a preferences file for it does not exist in the Windows/Winnt ACI folder, it will create one. When these preferences files are created, they do not by default contain the "Main Memory" resource (the resource is disabled).

Note: The "4DV6Prf.RSR" preferences file also stores your 4D v6 and 4D Server v6 licensing information, including expansion pack licenses, so do not delete it. 4D v6.5 and 4D Server v6.5 store their licensing and expansion pack information in the Windows Registry. The "4DV6Prf.RSR" file also stores the list of the paths to the databases most recently used with 4D v6.5. The "RunV6Prf.RSR" and"EngV6Prf.RSR" files do not store licensing information.

The following screen shows the "Main Memory" resource of a preferences file opened for editing in Customizer Plus:

Whenever you edit the Application Main Memory settings in the Database Properties dialog of 4D or 4D Server, the changed settings are stored in the "Main Memory" resource in the "4DV6Prf.RSR" preferences file. If the "Main Memory" resource does not already exist (if it is disabled) in the preferences file, this creates it. When you change the settings through the Database properties dialog, the changes are not stored in the database structure file, or in the 4D application. The settings stored in the "4DV6Prf.RSR" preferences file will be used by any database opened with 4D or 4D Server on the machine.

If you edit the Application Main Memory settings in the Database Properties dialog in the Design environment of 4D Client, the changed settings are stored in the "4DV6Prf.RSR" preferences file on the client machine. These changed settings will not be reproduced into the "4DV6Prf.RSR" preferences file on the server machine. 4D Client reads its memory settings from the "4DV6Prf.RSR" preferences file on the client machine.

1. When a 4D application is run, the first place it looks for the Application Main Memory settings is in the preferences file for that application.

2. If a preferences file does not exist, or if the preferences file does not contain the "Main Memory" resource (if the resource is disabled, as it is by default), the second place it looks is in the "Preferences" resource of the 4D application (such as "4D.exe" or "4D Server.exe"), not in the database structure file.

The Application "Main Memory" settings stored in the Preferences" resource of an uncompiled structure file are only recognized when compiling and merging the structure with 4D Engine.

Note: This is an exception to the usual order of priority in which settings are read. The usual order of priority is:

1. The preferences file

2. The structure file

3. The 4D application

If you compile and merge a database structure with 4D Engine, from where will the resulting database get its default Application Main Memory settings? The following hierarchy applies only to the Application Main Memory settings, not to the data cache size setting:

1. "EngV6Prf.RSR" -- When the executable structure compiled and merged with 4D Engine is run, the first place it looks for the Application Main Memory settings is in this preferences file. By default, the "Main Memory" resource does not exist (it is disabled) in the "EngV6Prf.RSR" file. You have to create it by editing this preferences file with Customizer Plus. (The Application Main Memory settings that are created in the new merged structure during compilation do not come from the "EngV6Prf.RSR" file.)

2. The structure file -- If the "EngV6Prf.RSR" file does not exist, or if the "Main Memory" resource has not been created in it, the Application Main Memory settings stored in the "Preferences" resource of the compiled merged structure (the ".EXE" file) will be used when it runs. This "Preferences" resource is created in the merged structure during compilation. The Application Main Memory settings in it are copied from the "Preferences" resource of the interpreted structure file (".4DB") if they exist there. Customizer Plus must be used to create the "Preferences" resource in the interpreted structure file. That resource does not exist there by default (it is disabled).

3. The engine itself -- If the "Main Memory" resource was not found in the engine preferences file, and the "Preferences" resource was not found in the interpreted structure file used during compilation, the Application Main Memory settings stored in the "Preferences" resource of the compiled merged structure (the ".EXE" file) will be copied from the "Preferences" resource of the engine itself ("4DEngine.4de").

After you have compiled and merged a structure with 4D Engine, where can you change the Application Main Memory settings? The hierarchy of priority for the locations where the settings are stored is:

1. "EngV6Prf.RSR" -- If this preferences file contains a "Main Memory" resource, this is the first place from which the Application Main Memory settings will be taken. This resource does not exist in the "EngV6Prf.RSR" file by default (the resource is disabled). The "Main Memory" resource can be created in the "EngV6Prf.RSR" file with Customizer Plus. The settings in this file will also be used for any other compiled and merged structure opened on the machine.

2. The compiled merged structure -- If there is no "EngV6Prf.RSR" file, or if the "EngV6Prf.RSR" file contains no "Main Memory" resource, the Application Main Memory settings stored in the "Preferences" resource of the compiled merged structure (the ".EXE" file) will be used. This resource contains the Application Main Memory settings it acquired during compilation.

Though you can edit the data cache settings in the ".4DC" file stored in the Distrib folder of the compiled merged structure, you cannot edit the Application Main Memory settings in the "Preferences" resource of this file. Those settings are non-enterable in this file. They contain the same values as in the merged ".EXE" file.

On Windows, while the Application Main Memory settings can be shared among different databases, the data cache size setting is unique to each particular database. 4D uses the Data Cache Maximum Size setting stored in the database's structure file.

The size of the data cache can be set either in the Database Properties dialog, or by using Customizer Plus. When you use the Database Properties dialog, changes to the data cache settings are saved in the "Properties" resource of the structure file. Customizer Plus must be used to change the data cache setting in any database that does not allow access to the Database Properties dialog, such as a compiled database.

You can use Customizer Plus to edit the data cache setting in the "Properties" resource for:

An interpreted structure: open the ".4DB" file

An already compiled structure: open the ".4DC" file.

A structure that has been compiled and merged with 4D Engine: open the ".4DC" file that 4D Compiler creates inside the "Distrib" folder.

On Windows, there is no "Properties" resource in a compiled and merged structure's executable file (the ".EXE" file), in 4D Engine itself, or in a 4D application such as 4th Dimension, 4D Server, etc.

If you use Customizer Plus to change the data cache size setting in an interpreted structure file, then compile and merge it with 4D Engine, the changed data cache setting will carry through to the compiled merged application.

In the following screen, Customizer Plus has been used to open the "Properties" resource of a database structure file for editing:

In the "Database cache memory" section of the dialog, the only setting used on Windows is "Maximum Size." The allowed values range from a minimum of 500K to a maximum of 3,276,700K (3.1 GB).

On Mac OS, when a 4D application is started, it immediately allocates all the memory that it is going to use, all at once. In comparison, on Windows, applications allocate memory dynamically, starting with a base amount of memory, then grabbing more memory later as needed.

In older versions of 4D, the 4D application memory was divided very differently than the way it is now, which has resulted in some misunderstanding.

In comparison to Windows, the storage of memory settings is simpler. Each setting is only stored in one place, so there is no hierarchy of possible locations of which to keep track.

The setting for the amount of memory that will be allocated to the application (4D, 4D Server, 4D Runtime, 4D Runtime Classic, or a structure compiled and merged with 4D Engine), is stored in the application itself, and accessed through the Macintosh "Get Info" dialog.

The setting to enable the "New Memory Allocation Scheme on Macintosh" and the data cache settings used by that scheme are stored in the structure file.

The memory settings used by the old memory allocation scheme are stored in the 4D application itself (4D, 4D Server, 4D Runtime, 4D Runtime Classic, or a structure compiled and merged with 4D Engine).

The "Accelerated Screen Updates" setting is stored in the preferences file ("4DV6Prf," "EngV6Prf," or "RunV6Prf"). On Mac OS, it is the only setting in the preferences file that can have an impact on memory. This is not actually a setting that proportions memory, but it can use up as much as 3 MB or more of the user heap memory, so it's worth mentioning here.

To use the new Mac OS memory allocation scheme, you must select the "Use New Memory Allocation Scheme on Macintosh" check box in the "System settings" page of the Database Properties dialog (formerly known in 4D v6 as the "Tune Up" page), or in the "Database Cache Memory" section of the "Properties" resource in a structure file. In 4D Server, you can select Database Properties from the File menu. In 4D standalone, you need to be in the Design environment to select Database Properties from the File menu.

By default, 4D applications do not use the new memory allocation scheme. The memory allocation scheme specified in the structure will be used by any application that uses the structure: 4D, 4D Server, 4D Runtime, 4D Runtime Classic, or the structure created by compiling and merging with 4D Engine.

ACI recommends that you use the New Memory Allocation Scheme. The benefit of using it is that the developer can control how much memory will be used for the data cache. With the old memory scheme, the size of the data cache is a percentage based on the memory size of the application, so the size of your data cache can change depending on how the user changes the amount of memory given to the application. If the user decreases the memory given to the application, the size of the data cache will shrink proportionally. Another disadvantage of the old memory allocation scheme is that it keeps the data cache size settings in the 4D application, which means that if the user replaces their copy of 4D, your data cache settings are replaced by the default values.

Configuring the new Mac OS memory allocation scheme

The new Mac OS memory allocation scheme can be chosen and configured either by editing the "Database Cache Memory" settings in the Database Properties dialog, or by using Customizer Plus to edit those settings in the "Properties" resource of a structure file. Either way you edit the settings, they are stored in the "Properties" resource of the structure file, which could be:

An interpreted structure

A compiled structure

A structure that has been compiled and merged with 4D Engine

With the new memory allocation scheme, 4D reserves two areas in the computer's memory for its use: the 4D application's memory area, whose size is set in the Macintosh "Get Info" dialog; and the data cache, which is held in a separate block of memory outside of the 4D application's memory.

More recent versions of the Macintosh "About This Computer" dialog, such as in OS 8.5.1, show the combined total of these two areas as the amount used by the 4D application. Earlier versions of the "About This Computer" dialog, such as in OS 7.6.1, do not include the size of the data cache in the size of the 4D application. But the memory taken up by the data cache outside the 4D application area does count toward reducing the size of the "Largest Unused Block" measurement shown in the dialog.

The following screen shows the "System settings" page of the Database Properties dialog, which includes the Database Cache Memory settings used by the new memory allocation scheme:

The following screen shows the "Properties" resource of the structure file, opened for editing in Customizer Plus. It includes the Database Cache Memory settings used by the new memory allocation scheme:

In either place, the Database Cache Memory settings you can edit are:

The "Use New Memory Allocation Scheme on Macintosh" check box. This allows you to select from two very different methods of memory allocation.

Maximum Cache. The allowed values range from a minimum of 500K to a maximum of 3,276,700K (3.1 GB). This sets the upper size limit of the data cache that will be used if there is enough memory available. The value that will be used is rounded down from the entered value to the nearest even number.

Minimum Cache. This entry area is hidden unless the "Use New Memory Allocation Scheme on Macintosh" check box is selected. The allowed values range from a minimum of 500K to a maximum of 3,276,700K (3.1 GB). This sets the minimum amount of memory that must be available outside the 4D application's memory space for use as the data cache, in order for the new Macintosh memory allocation scheme to be used. If enough memory is not available to accommodate the minimum data cache size specified, 4D automatically switches to using the old memory allocation scheme.

The values you set for the minimum and maximum data cache size should be almost the same. Developers sometimes make the mistake of setting the minimum data cache size to the smallest amount they want the data cache to be, then they set the maximum data cache size to a very high amount, figuring that if the database gets deployed on a machine with more memory, then more of that memory will be used for the data cache. The problem with this is the way 4D Server works when there is not enough memory available to accommodate the maximum data cache size specified. What happens then is that 4D repeatedly subtracts 512 bytes (that's 0.5K, not 512K) from the maximum data cache size specified, and tries to allocate that memory, until the size gets low enough that it can succeed. If this has to be repeated many times, 4D Server can take a very long time to start up.

Since the new memory allocation scheme keeps the data cache in a block of memory outside of the 4D application's memory area, the 4D application's memory heap only needs to hold the 4D engine components, then the rest of it is devoted to the user heap.

The size of the 4D application's memory heap is equal to the "Preferred size" set in the Macintosh "Get Info" dialog, minus a small amount of memory used for an application stack and other purposes (assuming that much memory is available). That amount is about 213K under Mac OS 7.6.1, and about 167K under Mac OS 8.5.1 or 8.6. The remaining memory is the 4D application's heap zone, which is shown in the Macintosh 4D Server window as "Total Memory."

If you know the amount of memory used to hold the 4D engine components, you can subtract it from the Total Memory size, to find the size of the user heap.

The amount of memory needed to store the 4D engine components is much larger if virtual memory is disabled. If virtual memory is turned off, the entire data fork of the 4D application has to be loaded into memory. That is not necessary if virtual memory is enabled. The exact amount of memory needed to store 4D engine components also varies between the various 4D applications. Several means to determine the size of engine memory are described in step 6 of the section "Calculating the old memory scheme" later in this technical note.

When the new memory allocation scheme is used, settings in the "Memory" resource such as memory ranges, "Cache memory," and "Kernel memory" have no effect.

If you have not selected the "Use New Memory Allocation Scheme on Macintosh" check box option in the "System settings" page of Database Properties, 4D's memory is configured by editing the "Memory" resource with Customizer Plus:

The "Memory" resource is present in:

4th Dimension

4D Server

4D Runtime

4D Runtime Classic

4D Engine

A structure that has been compiled and merged with 4D Engine

The settings in the "Memory" resource used for setting memory sizes are:

Cache memory. The default value is 50%. The allowed values range from 16% to 100%.

Kernel memory. This setting is widely misunderstood. The default value is 512K. The allowed values range from a minimum of 100K to a maximum of 1,073,741,824K.

The old Mac OS memory allocation scheme can be a bit confusing. What happens is that 4D first makes some calculations about how to divide memory, basing this on a now non-existent memory model. Then the actual partitioning of 4D's memory space is performed in a very different way.

The basic formula first allocates a fixed base amount of memory for the 4D engine components. The "Kernel memory" amount is then added to this base amount. The size of the remaining memory is then multiplied by the "Cache memory" percentage, to get the size of the data cache.

To perform the actual partition of memory, 4D first allocates a part of its memory space to the 4D engine components that need to be loaded. Then the data cache is allocated, using the data cache size calculated from the formula. All the remaining memory that is not taken up by engine memory or the data cache is used for the user heap.

The result of this is that the settings for "Cache memory" and "Kernel memory" both end up adjusting the division of the non-engine memory between the data cache and the user heap, each in a different way. The combination of both these settings determines how 4D's non-engine memory will be split between the data cache and the user heap:

In the "Cache memory" setting, you enter a percentage of the formula's unreserved portion of memory that you want devoted to the data cache.

The "Kernel memory" setting allows you to slide the dividing line between the data cache and the user heap up or down by a specified amount.

In some earlier versions of 4D that ran on Macintoshes with 68000-series CPUs, you could adjust the size of the "kernel memory" area where the program segments of 4D's engine loaded.

On the Macintosh platform, 4D v6.5 only runs on PowerPC processors. It does not run on 68000-series processors. On PowerMacs, the only way you can affect how much memory will be used for loading program code is by choosing to use or not use virtual memory. The "Kernel memory" setting of 4D v6.5 has no effect whatsoever on the amount of memory used to store the code of the 4D engine. However, it most definitely does affect how 4D allocates memory, but in a very different way than its name would imply.

If you increase "Kernel memory" by a given amount, the size of the data cache decreases by half that amount. The "Information" section of the Runtime Explorer's Watch page shows that "Free Memory" is increased by half the given amount.

If you decrease "Kernel memory" by a given amount, the size of the data cache increases by half that amount. The "Information" section of the Runtime Explorer's Watch page shows that "Free Memory" is decreased by half the given amount.

By adjusting the "Kernel memory" amount together with the "Cache memory" percentage, you can cause the size of the data cache to be reduced to as little as 64K. If you have a 4D database that does not use the data file at all, this would be an ideal allocation for your memory. Certain web applications would fit this description.

The old memory allocation scheme also includes a memory ranges feature. The thermometer in the "Memory" resource covers a range of the possible sizes for the 4D application's memory (the "Get Info" size). You can create markers in this scale to divide the possible sizes for the application's memory into several ranges. When the application runs, its actual memory size may fall into one of these ranges, and for each range you can assign a different set of memory parameters to be used.

This feature might have been useful in the past, but nowadays it is nearly useless. The minimum recommended memory on 4D v6.5 applications is 8,000K, but the largest memory size you can create a memory range marker for is about 7,928K.

To make some of the measurements used in this formula, you need a memory analysis tool. Metrowerks ZoneRanger is a free utility that enables you to easily examine the details of how Macintosh applications use their memory. You can download it from the Metrowerks web site at <http://www.metrowerks.com/tools/software/zoneranger.stm>. In the examples described here, ZoneRanger version 1.7 was used.

You can learn some interesting things using ZoneRanger, but it cannot tell you everything that 4D is doing with memory. 4D can perform a lot of its own manipulation of the contents inside memory blocks. For example, the entire data cache is one big block of memory. 4D moves objects in and out of the data cache, but all that activity is invisible to ZoneRanger.

1. First you need to know the amount of memory your 4D application is actually using. 4D uses as much of the "Preferred size" amount in the Macintosh "Get Info" dialog as it can find in one contiguous block not already in use by the system software or other applications on the computer.

Even when the full "Preferred size" amount is available, 4D applications use slightly less than that. A small amount of memory is used for an application stack and other purposes. That amount is about 213K under Mac OS 7.6.1, and about 167K under Mac OS 8.5.1 or 8.6. The remaining amount is the size of the 4D application's heap zone.

You can measure the size of the 4D application's heap zone precisely with ZoneRanger. Start ZoneRanger, then open your database with the 4D application it will be deployed with. In ZoneRanger's "Heap Zones" window, there is a list of the applications currently running on the computer. Double-click on the entry for your 4D application. This opens up several windows in ZoneRanger that display information about the memory of your 4D application. In the "Summary" window, the bottom figure in the "Blocks" column is the number of bytes of memory that your 4D application is using. (You may prefer to convert the figures in ZoneRanger's Summary window from bytes to kilobytes.) We will refer to this amount as the "Total Memory." If you are using 4D Server, the "Total Memory" amount is displayed in the upper part of the 4D Server window.

2. Now that you know the Total Memory size, you need to know the Base Memory size. Here are some examples, for v6.5 and v6.5.2:

0K -- Virtual memory on: 4D, 4D Server, 4D Runtime, 4D Runtime Classic, a structure compiled and merged with 4D Engine.

2,444K -- Virtual memory off: 4D Server.

3,908K -- Virtual memory off: 4D, 4D Runtime, 4D Runtime Classic, a structure compiled and merged with 4D Engine.

You may want to know how to calculate this value, in case it changes in another release. Use the Runtime Explorer to find the size of your 4D application's data cache. Divide the size of the data cache by the "Cache memory" percentage. For example, if the size of the data cache is 13,266K and the "Cache memory" setting is 50%,

13,266K / 0.5 = 26,532K

Subtract this number from the Total Memory. Then subtract the Kernel Memory size. This gives you the Base Memory size. For example, if the Total Memory is 30,000K, and the Kernel Memory size is 1,024K,

30,000K - 26,532K - 1,024K = 2,444K

The Base Memory size in this example (4D Server 6.5.2 with virtual memory off) is 2,444K.

3. Now that you know the Base Memory size, you can use the following formula to find the size of the data cache you get with different memory settings:

Data Cache Size = (Total Memory - Base Memory - Kernel Memory) * Cache Memory

4. When the old memory allocation scheme is used, the Maximum Cache setting (in the Database Properties dialog, and in the "Properties" resource of a structure file) is effective on new databases created from 4D or 4D Server. Once you change the value of this setting, it is ignored. Bear this in mind when you create a new database. If you do not change this setting, your database will not be able to use more than the default amount of 4,096K for the data cache.

5. To calculate the size of the user heap, use this formula:

User Heap Size = Total Memory - Engine Memory - Data Cache

6. Engine Memory is the size of the 4D engine components that get loaded. This could vary depending on various factors, including:

The application used: 4D, 4D Server, 4D Runtime, 4D Runtime Classic, or a structure compiled and merged with 4D Engine.

Whether virtual memory is turned on or off. If virtual memory is turned off, then the entire data fork of the 4D application needs to be loaded into memory. If virtual memory is turned on, the data fork is not loaded into memory. For example, in the case of 4D Server 6.5.2, the Macintosh "Get Info" dialog states "Memory requirements will increase by 3,564K if virtual memory is turned off in the Memory control panel."

The size of the engine components can vary from one minor release version of 4D to another. If the authors of 4D modify the application's code, the length of code segments will probably be different.

The number of engine components that load into memory can vary depending on various factors, such as whether the database is set to publish on the web on startup, or whether an attempt by the database to make a network connection at startup is completed or not. It might also vary depending on the version of the operating system in use.

Some approximate examples of the size of Engine Memory are:

4D 6.5.2, with virtual memory off: 5,604-5,920K

4D 6.5.2, with virtual memory on: 704-860K

4D Server 6.5.2, with virtual memory off: 4,404-4,464K

4D Server 6.5.2, with virtual memory on: 844-904K

Unless you change the virtual memory status, the variation in the Engine Memory size will not be more than about 316K.

To measure the Engine Memory with ZoneRanger, start with the bottom figure in the "Pointers" column of ZoneRanger's Summary window. This is the total amount of non-relocatable memory in your 4D application. That also includes the entire data cache. In ZoneRanger's memory map window, the "Pointer" memory is colored red by default.

Use 4D's Runtime Explorer to find the size of the data cache. For example, in the "Cache Statistics" section of the Runtime Explorer, if the information shown was "56 bytes / 3 636 Kb (0%), 3 handles," the total size of the data cache is 3,636K.

Subtract the size of the data cache from the total size of the pointer blocks, to get the size of the Engine Memory.

7. Using ZoneRanger, you can measure the exact size of the user heap area by adding two figures in ZoneRanger's Summary Window:

The bottom figure in the Free column is the total size of 4D's unused memory. This is the same as the value shown in the Runtime Explorer's Information section as "Free Memory." Because the memory usage may be fluctuating, you may not always see exactly the same two values in ZoneRanger and the Runtime Explorer, unless you update both at exactly the same time. In ZoneRanger's memory map window, the "Free" memory is colored green by default.

The bottom figure in the Handles column of ZoneRanger's Summary window is the total amount of the user heap area that is currently in use. In ZoneRanger's memory map window, the "Handle" memory is colored yellow by default.

Add the totals from the Free column and the Handles column to get the total size of the user heap area.

Some of the values factored into this formula could change in future versions of 4D. By using the Runtime Explorer, you can verify the actual results achieved by any changes made to the memory settings.

1. For example, with virtual memory turned on, 4D Server 6.5.2 is assigned 24,000K as the "Preferred size" in the Macintosh "Get Info" dialog. When 4D Server is started, the "Total Memory" shown in the 4D Server window is 23,833K.

2. With virtual memory off, the Base Memory size for 4D Server 6.5.2 is 0K.

3. In Customizer Plus, the "Cache Memory" is set to 50% and the "Kernel Memory" is set to 512K. The Runtime Explorer shows the size of the data cache to be 11,660K.

This formula can be used to calculate the size of the data cache with various memory settings:

(Total Memory - Base Memory - Kernel Memory) * Cache Memory = Data Cache size

(23,833K - 0K - 512K) * 0.5 = 11,660K

If the Cache Memory setting was changed to 25%, the calculation for the data cache size would be:

(23,833K - 0K - 512K) * 0.25 = 5,830K

4. In this example, the Maximum Cache setting has been changed from the default value of 4,096K. Therefore, our data cache size will not be limited to that amount.

5. Next we'll calculate the size of Engine Memory. The total size of the pointers listed in ZoneRanger is 12,504K. This formula calculates the size of Engine Memory:

Pointers - Data Cache = Engine Memory

12,504K - 11,660K = 844K

6. Now that we know the size of Engine Memory is 844K, we can calculate the size of the User Heap, using this formula:

Total Memory - Engine Memory - Data Cache = User Heap

23,833K - 844K - 11,660K = 11,329K

Using the measurements you have taken, you can now accurately predict the effect of any changes you might make to the "Cache memory" or "Kernel memory" settings. For example, if you change the Kernel memory setting to 844K, leaving the "Cache memory" set to 50%, the resulting data cache size will be 11,494K. With the "Kernel memory" set to 844K and the "Cache memory" set to 25%, the data cache size will be about 5,747K.

This setting does not proportion memory, but enabling it can cause as much as 3 MB or more of the user heap memory to be used, so it is worth mentioning here.

The "Accelerated Screen Updates" setting can be changed either in the "General" tab of the Database Properties dialog, or by using Customizer Plus to edit the "Screen Update" resource of the preference file.

If you enable the "Accelerated Screen Updates" option, a portion of the user heap will be used as an offscreen bitmap area, enabling faster screen redraws. The larger the resolution of your monitor and the higher its color bit depth setting, the more memory will be used for this offscreen bitmap. The formula to calculate the amount of memory (in bytes) this will use is:

(Screen width x screen height x (color bit depth/8)) + ((Screen height x screen width)/64)

For example, if your monitor resolution is 640 x 480 pixels, and your color bit depth is "256 colors," then 305K of memory will be needed for the offscreen bitmap.

If your monitor resolution is 1,024 x 768 pixels, and your color bit depth is "millions of colors," then 3,084K (over 3 MB) of memory will be needed for the offscreen bitmap.

The only place where this setting is stored is in the "Screen Update" resource of the preference files. On Windows, these files are named "4DV6Prf.RSR," "EngV6Prf.RSR," and "RunV6Prf.RSR." On Macintosh, these files are named "4DV6Prf," "EngV6Prf," and "RunV6Prf."

The preferences files do not by default contain the "Screen Update" resource (the resource is disabled). This resource is created if you change the Accelerated Screen Updates setting, whether you change the setting directly by using Customizer Plus, or if you change the setting in the Database Properties dialog. If there is no preferences file, or if the "Screen Update" resource does not exist in the preferences file, then the setting defaults to being turned off.

Note: The "4DV6Prf.RSR" file also stores your 4D v6 and 4D Server v6 licensing information, including expansion pack licenses, so do not casually delete it. (4D v6.5 and 4D Server v6.5 store their licensing and expansion pack information in the Windows Registry. The "4DV6Prf.RSR" file also stores the list of the paths to the databases most recently used with 4D v6.5.) The "RunV6Prf.RSR" and"EngV6Prf.RSR" files do not store licensing information.

Given the amount of memory that you have available to devote to your 4D application, how much of that should you allocate for the data cache vs. the user heap?

The value of having a large data cache can be overestimated. A common belief is that if some data cache is good, more is better, and even more is better yet. That is only true up to a certain point. Having a large data cache is nice, but having sufficient space for the user heap can be critical. If your user heap is not big enough, you are likely to crash. If your data cache is a little smaller, retrieving frequently-used records might not be quite as fast, but that is quite different from crashing. Make sure you have set aside sufficient space for the user heap before you increase the size of the data cache.

If you are handling large records, there are specific ways to proportion your application's memory to help accommodate them. These are detailed below, but they should not be your only considerations.

If the data cache is not big enough to handle a large record, the 4D application can instead use space in the user heap, but there is a penalty: the amount of additional space used in the user heap is twice as much as the amount of space that would have been used in the data cache. You'll be better off if the size of your data cache is big enough to hold your largest record.

On Mac OS, this is a critical concern because you cannot save an existing record if there is not enough room in the data cache to hold a complete copy of the record. So, the size of your data cache should be at least large enough to hold your largest record.

The need to keep the size of the user heap area big enough so that you do not run out of memory during peak operating conditions imposes a practical limit on the size you can increase your data cache to, without reducing the reliability of your database. This limit on the upper size of your data cache in turn limits the maximum size of a record that you can handle. If you deal with large data types such as BLOBs and pictures, it would be prudent for your code to limit the maximum size of BLOBs and pictures that can be saved in a record.

If you have a critical database, you should not create excessively large records. If there is not enough memory to handle them, 4D can crash. On 4D Server, the memory needs of handling large records are amplified enormously, because the server needs to maintain copies of the current records for the tables of each process of each connected user (and any stored procedures as well). You may want to use a semaphore system to limit the number of large records that can be simultaneously accessed, to avoid the memory strain that could otherwise result.

On 4D Server, among the basic record-handling operations (loading an existing record, saving an existing record, saving a new record, and deleting a record), saving an existing record requires the most memory. Assuming that you have enough space in the data cache to hold the record, you also need enough space in the user heap to hold three copies of the record. Considering this, you could split the memory usage so that the size of the user heap is three times greater than the size of the data cache. This will allow you to handle the largest possible record in your available memory (assuming that only one process or user at a time is using a large record).

On 4D standalone, the basic record-handling operations need only enough space for one copy of the record in the data cache, and another copy in the user heap. To permit handling of the largest possible record, you could split the memory usage so that the data cache is the same size as the user heap (assuming that only one process at a time is using a large record).

By understanding the record loading behavior of 4D, you can optimize the performance and the reliability of your database. When you are using large records or running your database over a wide area network, this can be a critical issue.

This is the second of a three-part technical note. Part 3 will include diagnostic techniques for analyzing record loading, list the commands and activities that cause records to be loaded, provide full details on how each of these specific operations impacts record loading, and discuss strategies for more efficiently managing record loading.

Customizer Plus Reference manual (Customizer_Plus65.pdf).

4D Server Reference manual (4D_Server6_Ref.pdf).

ACI_PACK Reference manual (Aci_Pack_6_Reference.pdf)

For background about memory on Macintosh:

Apple Computer, Inc. Inside Macintosh: Memory. Reading MA: Addison-Wesley Publishing Co., 1992. Available online at:

<http://developer.apple.com/techpubs/mac/Memory/Memory-2.html>.