About Caching |
Shake uses caching to speed up reprocessing. Caching saves a snapshot of certain nodes with all of the parameters that make up that node in a directory called the cache directory. When Shake processes any tree, it very quickly scans the directory to see if this exact tree has been calculated before. If it has, it calls up the snapshot rather than recalculate it.
The cached images are labeled with input files, date of file creation, the node tree, and all of the parameters in the nodes, including resolution/proxy settings, to guarantee identical results. It is not accessible by the user.
The Shake cache is persistent. If you quit out of the program, and then relaunch the program the next day, it still calls up the files cached the previous day.
There are four settings for the cache:
Type | Setting | Function |
None | 0 | No images are read from or written to the cache. |
Read Only | 1 | Images are only read from the cache. No images are written to it. |
Regular | 2 | Images are read from and written to the cache in a conservative algorithm. Only non-animated branches are cached. |
Aggressive | 3 | As Regular, but animated branches are also cached. This results in a lot of cache activity, so make sure you have a large cache disk if you to use this mode. |
Shake also has transient caching, which is a temporary cache of viewed images. This means you can usually toggle between two images quickly. As you work on nodes that aren't connected to the cached image, the cache is disposed of.
The cache has a default maximum size of 512 MB. When files are written to the cache that push its size above the maximum limit, the oldest cached files removed, allowing for new space. If you are using the Aggressive caching, you should have a lot of disk space and RAM, since otherwise you are constantly deleting old files and writing new ones, defeating the intended purpose of the cache.
By default, the cache is written to:
To clear disk space, you can remove this directory, but the caching information is lost. This is not vital, however, to the script. It simply forces Shake to completely rerender the compositing tree. Also note that Shake automatically creates the cache directories if they do not exist.
To manually declare your cache size, create a startup .h file. There are two main settings for the cache:
Function | What it does |
cache.cacheMemory = 64;
|
Controls how much memory
is available to nodes when processing.
|
diskCache.cacheMemory = 64; | For the caching of images into memory. |
The cache.cacheMemory setting works on a per-proc basis, and 64 MB is usually enough. For particularly complex scripts, you may want to bump this up to 96 MB. It is not necessary to increase this setting to a very large number to get better performance, since it controls the amount of memory Shake can use for processing, but not for caching of frames (which is controlled by diskCache.cacheMemory). Processing memory is needed only by layer nodes and some operators that can't work 'in-place' (that is, require a temp buffer), but even then, most of these nodes grab only a small amount of memory. You can override this setting on the command line with the -mem setting.
On the other end, scaling diskCache.cacheMemory proportionally to the amount of memory available on your machine is going to improve performance significantly, as more images are kept around. Set to approximately 25 to 30 percent of your total memory, or enough to hold about four or five images in memory. A 2K, 4-channel float image takes about 51 MB of memory.
There is a limit on what size a process can grow to (2G on most OS). Past that point, the OS either prevents the process from allocating memory, or just kills it. Keeping diskCache.cacheMemory to around 1000 (1GB) should give ample caching space while staying clear of the OS hard limit.
Here are some additional functions for the startup .h files:
Function | What it does |
diskCache.cacheLocation="/var/tmp/Shake/cache"; diskCache.cacheLocation="C:/TEMP/Shake/cache"; |
Set this to a location with gobs of unused space (and expect to use it all). |
diskCache.cacheMaxFile=2048; | The maximum number of files in the cache before they are replaced. Least used, first out. You should leave this setting unchanged. |
diskCache.cacheMaxFileSize = 2048*2048*4*2; |
These values represent width*height*channels*bytes This is probably the most difficult to calculate. You want to make sure that you can fit at the very least four to five images in memory cache (the more images you can fit in memory, the better your interactivity). If you want to cache 2048x1556 8-bit images, then you are looking at about 12.7 MB per image (derived from the formula above). For this to work well, you need to designate about 64 MB of cache memory (hence the default setting for cache.cacheMemory). This means you need about 256 MB total memory. Of course some kind of balance needs to be found here with your working resolution, work habits, proxy settings, etc. |
diskCache.cacheSize | The maximum size in MB of your cache directory. |
Example
Let's say you have a 1 GB, 2-proc machine and you want to leave at least 150 MB for OS overhead. 1024-150 = 874 MB. A 2k image at float is 2048 across * 1556 high * 4 channels * 4 bytes-per-channel (float) = 50,987,008. About 51 MB for float 16 bit is half of that. Assuming you want to boost your memory for the script to 96 MB per proc, set cache.cacheMemory = 96; On a two processor machine this takes 192 MB. 874-192 = 682 MB. You want to have enough free RAM for 2 Viewer buffers at float 2K, so that's another 102 MB. 682-102 = 580 MB. You want to be able to keep at least 5 full 2K float images in the cache, thats 5*51 = 255 MB, so you set diskcache.cacheMemory = 256. 580-256 = 324 MB. This leaves 324 MB RAM left for flipbooks and whatever else you may be doing. At 1/3 proxy (still at float, though) you are at about 6 MB per frame, so you could get about 80 frames into a flipbook.
In short, give your favorite RAM manufacturer a call.