Field3D
|
#include <SparseFile.h>
Public Types | |
typedef std::list< SparseFile::CacheBlock > | CacheList |
Public Member Functions | |
template<class Data_T > | |
void | activateBlock (int fileId, int blockIdx) |
Called by SparseField when it's about to read from a block. This should not be called by the user, and may be removed from the public interface later. More... | |
float | cacheEfficiency () |
Computes the efficiency, the ratio of the number of blocks ever loaded to the number of loads. If this is <1, then there were reloads. More... | |
float | cacheFractionLoaded () |
Computes the ratio of blocks in the cache to the total number of blocks that have been loaded (including unloaded blocks) More... | |
float | cacheLoadsPerBlock () |
Computes the overall loaded-blocks-to-load ratio for cached files. More... | |
template<class Data_T > | |
void | decBlockRef (int fileId, int blockIdx) |
Decrements the usage reference count on the specified block, after its value is no longer being used This should not be called by the user, and may be removed from the public interface later. More... | |
bool | doLimitMemUse () const |
Returns whether to limit memory usage and do dynamic loading for sparse fields. More... | |
void | flushCache () |
Flushes the entire block cache for all files, should probably only be used for debugging. More... | |
template<class Data_T > | |
void | incBlockRef (int fileId, int blockIdx) |
Increments the usage reference count on the specified block, to prevent it from getting unloaded while it's still in use. This should not be called by the user, and may be removed from the public interface later. More... | |
long long int | memSize () const |
Returns the number of bytes used by the SparseFileManager itself. More... | |
long long | numLoadedBlocks () |
Returns the total number of blocks currently loaded into cache. More... | |
void | resetCacheStatistics () |
Resets block load. More... | |
void | setLimitMemUse (bool enabled) |
Sets whether to limit memory usage and do dynamic loading for sparse fields. More... | |
void | setMaxMemUse (float maxMemUse) |
Sets the maximum memory usage, in MB, by dynamically loaded sparse fields. More... | |
long long | totalLoadedBlocks () |
Returns the total number of blocks loaded (max 1 per block) into cache. More... | |
long long | totalLoads () |
Returns the total number of block loads in the cache. More... | |
Static Public Member Functions | |
static SparseFileManager & | singleton () |
Returns a reference to the singleton instance. More... | |
Protected Member Functions | |
template<class Data_T > | |
int | getNextId (const std::string filename, const std::string layerPath) |
Returns the id of the next cache item. This is stored in the SparseField in order to reference its fields at a later time. More... | |
template<class Data_T > | |
SparseFile::Reference< Data_T > * | reference (int index) |
Returns a reference to the Reference object with the given index. More... | |
template<class Data_T > | |
void | removeFieldFromCache (int refIdx) |
Private Member Functions | |
void | addBlockToCache (DataTypeEnum blockType, int fileId, int blockIdx) |
Adds the newly loaded block to the cache, managed by the paging algorithm. More... | |
template<class Data_T > | |
void | deallocateBlock (CacheList::iterator &it) |
Utility function to deallocate a single block. More... | |
template<class Data_T > | |
int64_t | deallocateBlock (const SparseFile::CacheBlock &cb) |
Utility function to attempt to deallocate a single block and advance the "hand". More... | |
void | deallocateBlocks (int64_t bytesNeeded) |
Utility function to reclaim the specified number of bytes by deallocating unneeded blocks. More... | |
SparseFileManager () | |
Private to prevent instantiation. More... | |
Private Attributes | |
CacheList | m_blockCacheList |
List of dynamically loaded blocks to be considered for unloading when the cache is full. Currently using Second-chance/Clock paging algorithm. For a description of the algorithm, look at: http://en.wikipedia.org/wiki/Page_replacement_algorithm#Second-chance. More... | |
SparseFile::FileReferences | m_fileData |
Vector containing information for each of the managed fields. The order matches the index stored in each SparseField::m_fileId. More... | |
bool | m_limitMemUse |
Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when true. More... | |
float | m_maxMemUse |
Max amount om memory to use in megabytes. More... | |
int64_t | m_maxMemUseInBytes |
Max amount om memory to use in bytes. More... | |
int64_t | m_memUse |
Current amount of memory in use in bytes. More... | |
boost::mutex | m_mutex |
Mutex to prevent multiple threads from deallocating blocks at the same time. More... | |
CacheList::iterator | m_nextBlock |
Pointer to the next block to test for unloading in the cache, the "hand" of the clock. More... | |
Static Private Attributes | |
static boost::scoped_ptr< SparseFileManager > | ms_singleton |
Pointer to singleton. More... | |
Friends | |
template<class Data_T > | |
class | SparseField |
Handles specifics about reading sparse fields from disk. Its primary use is to control sparse fields read using memory limiting (dynamic loading).
To enable the dynamic cache for a file, call setLimitMemUse(true) before opening the file. If you want other files to be fully loaded, call setLimitMemUse(false).
Example of how to use the cache manager to automatically unload sparse blocks from a f3d file:
SparseFileManager &sparseManager = SparseFileManager::singleton(); sparseManager.setLimitMemUse(true); // enables cache for files to be opened sparseManager.setMaxMemUse(1000.0); // sets cache to 1 GB Field3DInputFile cacheManagedFile; if (!cacheManagedFile.open(filename)) { Msg::print( "Couldn't open file: " + filename); return 1; } sparseManager.setLimitMemUse(false); // disables cache for other files
... // You can use the file normally, loading layers and then accessing ... // with const_iterator, value(), or empty block functions like ... // getBlockEmptyValue(). ... // Layers loaded from cacheManagedFile will be managed by the cache, ... // but other files you open will be fully loaded when opened because of ... // the setLimitMemUse(false) call.
Msg::print("blocks in cache: " + boost::lexical_cast<std::string>(sparseManager.numLoadedBlocks())); Msg::print("cache blocks ever loaded: " + boost::lexical_cast<std::string>(sparseManager.totalLoadedBlocks())); Msg::print("cache loads: " + boost::lexical_cast<std::string>(sparseManager.totalLoads())); Msg::print("cache fraction loaded: " + boost::lexical_cast<std::string>(sparseManager.cacheFractionLoaded())); Msg::print("cache loads per block: " + boost::lexical_cast<std::string>(sparseManager.cacheLoadsPerBlock())); Msg::print("cache efficiency: " + boost::lexical_cast<std::string>(sparseManager.cacheEfficiency()));
If you want to flush the cache manually instead of waiting for the process to end and clean up its memory:
sparseManager.flushCache(); sparseManager.resetCacheStatistics();
Definition at line 397 of file SparseFile.h.
typedef std::list<SparseFile::CacheBlock> SparseFileManager::CacheList |
Definition at line 407 of file SparseFile.h.
|
private |
Private to prevent instantiation.
Definition at line 282 of file SparseFile.cpp.
References m_blockCacheList, m_nextBlock, and setMaxMemUse().
|
static |
Returns a reference to the singleton instance.
Definition at line 66 of file SparseFile.cpp.
References ms_singleton.
Referenced by SparseField< Data_T >::addReference().
void SparseFileManager::setLimitMemUse | ( | bool | enabled | ) |
Sets whether to limit memory usage and do dynamic loading for sparse fields.
Definition at line 76 of file SparseFile.cpp.
References m_limitMemUse.
bool SparseFileManager::doLimitMemUse | ( | ) | const |
Returns whether to limit memory usage and do dynamic loading for sparse fields.
Definition at line 83 of file SparseFile.cpp.
References m_limitMemUse.
void SparseFileManager::setMaxMemUse | ( | float | maxMemUse | ) |
Sets the maximum memory usage, in MB, by dynamically loaded sparse fields.
Definition at line 90 of file SparseFile.cpp.
References m_maxMemUse, and m_maxMemUseInBytes.
Referenced by SparseFileManager().
void SparseFileManager::flushCache | ( | ) |
Flushes the entire block cache for all files, should probably only be used for debugging.
Definition at line 223 of file SparseFile.cpp.
References SparseFile::CacheBlock::blockType, DataTypeDouble, DataTypeFloat, DataTypeHalf, DataTypeUnknown, DataTypeVecDouble, DataTypeVecFloat, DataTypeVecHalf, m_blockCacheList, m_mutex, and m_nextBlock.
long long SparseFileManager::totalLoads | ( | ) |
Returns the total number of block loads in the cache.
Definition at line 292 of file SparseFile.cpp.
References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().
Referenced by cacheEfficiency(), and cacheLoadsPerBlock().
long long SparseFileManager::numLoadedBlocks | ( | ) |
Returns the total number of blocks currently loaded into cache.
Definition at line 325 of file SparseFile.cpp.
References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().
Referenced by cacheFractionLoaded().
long long SparseFileManager::totalLoadedBlocks | ( | ) |
Returns the total number of blocks loaded (max 1 per block) into cache.
Definition at line 358 of file SparseFile.cpp.
References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().
Referenced by cacheEfficiency(), cacheFractionLoaded(), and cacheLoadsPerBlock().
float SparseFileManager::cacheFractionLoaded | ( | ) |
Computes the ratio of blocks in the cache to the total number of blocks that have been loaded (including unloaded blocks)
Definition at line 391 of file SparseFile.cpp.
References detail::max(), numLoadedBlocks(), and totalLoadedBlocks().
float SparseFileManager::cacheLoadsPerBlock | ( | ) |
Computes the overall loaded-blocks-to-load ratio for cached files.
Definition at line 398 of file SparseFile.cpp.
References detail::max(), totalLoadedBlocks(), and totalLoads().
float SparseFileManager::cacheEfficiency | ( | ) |
Computes the efficiency, the ratio of the number of blocks ever loaded to the number of loads. If this is <1, then there were reloads.
Definition at line 405 of file SparseFile.cpp.
References detail::max(), totalLoadedBlocks(), and totalLoads().
void SparseFileManager::resetCacheStatistics | ( | ) |
Resets block load.
Definition at line 412 of file SparseFile.cpp.
References half, m_fileData, SparseFile::FileReferences::numRefs(), and SparseFile::FileReferences::ref().
long long int SparseFileManager::memSize | ( | ) | const |
Returns the number of bytes used by the SparseFileManager itself.
Definition at line 442 of file SparseFile.cpp.
References m_blockCacheList, m_fileData, m_mutex, and SparseFile::FileReferences::memSize().
void SparseFileManager::incBlockRef | ( | int | fileId, |
int | blockIdx | ||
) |
Increments the usage reference count on the specified block, to prevent it from getting unloaded while it's still in use. This should not be called by the user, and may be removed from the public interface later.
Definition at line 1298 of file SparseFile.h.
References m_fileData, SparseFile::FileReferences::ref(), and reference().
void SparseFileManager::decBlockRef | ( | int | fileId, |
int | blockIdx | ||
) |
Decrements the usage reference count on the specified block, after its value is no longer being used This should not be called by the user, and may be removed from the public interface later.
Definition at line 1311 of file SparseFile.h.
References m_fileData, SparseFile::FileReferences::ref(), and reference().
void SparseFileManager::activateBlock | ( | int | fileId, |
int | blockIdx | ||
) |
Called by SparseField when it's about to read from a block. This should not be called by the user, and may be removed from the public interface later.
Definition at line 1257 of file SparseFile.h.
References addBlockToCache(), deallocateBlocks(), m_fileData, m_limitMemUse, m_memUse, m_mutex, SparseFile::FileReferences::ref(), and reference().
Referenced by SparseField< Data_T >::const_iterator::operator->().
|
protected |
Returns a reference to the Reference object with the given index.
Definition at line 1248 of file SparseFile.h.
References m_fileData, and SparseFile::FileReferences::ref().
Referenced by activateBlock(), deallocateBlock(), decBlockRef(), incBlockRef(), and removeFieldFromCache().
|
protected |
Returns the id of the next cache item. This is stored in the SparseField in order to reference its fields at a later time.
Definition at line 1182 of file SparseFile.h.
References SparseFile::FileReferences::append(), m_fileData, and m_mutex.
|
protected |
Definition at line 1199 of file SparseFile.h.
References m_blockCacheList, m_fileData, m_memUse, m_mutex, m_nextBlock, SparseFile::FileReferences::ref(), reference(), and DataTypeTraits< T >::typeEnum().
|
private |
Adds the newly loaded block to the cache, managed by the paging algorithm.
Definition at line 260 of file SparseFile.cpp.
References m_blockCacheList, and m_nextBlock.
Referenced by activateBlock().
|
private |
Utility function to reclaim the specified number of bytes by deallocating unneeded blocks.
Definition at line 160 of file SparseFile.cpp.
References SparseFile::CacheBlock::blockType, DataTypeDouble, DataTypeFloat, DataTypeHalf, DataTypeUnknown, DataTypeVecDouble, DataTypeVecFloat, DataTypeVecHalf, m_blockCacheList, m_maxMemUseInBytes, m_memUse, m_mutex, and m_nextBlock.
Referenced by activateBlock().
|
private |
Utility function to attempt to deallocate a single block and advance the "hand".
Definition at line 99 of file SparseFile.cpp.
References SparseFile::CacheBlock::blockIdx, m_blockCacheList, m_fileData, m_memUse, m_nextBlock, SparseFile::FileReferences::ref(), reference(), and SparseFile::CacheBlock::refIdx.
|
private |
Utility function to deallocate a single block.
Definition at line 148 of file SparseFile.cpp.
References SparseFile::CacheBlock::blockIdx, m_blockCacheList, m_fileData, m_memUse, SparseFile::FileReferences::ref(), reference(), and SparseFile::CacheBlock::refIdx.
|
friend |
Definition at line 403 of file SparseFile.h.
|
staticprivate |
|
private |
Max amount om memory to use in megabytes.
Definition at line 517 of file SparseFile.h.
Referenced by setMaxMemUse().
|
private |
Max amount om memory to use in bytes.
Definition at line 520 of file SparseFile.h.
Referenced by deallocateBlocks(), and setMaxMemUse().
|
private |
Current amount of memory in use in bytes.
Definition at line 523 of file SparseFile.h.
Referenced by activateBlock(), deallocateBlock(), deallocateBlocks(), and removeFieldFromCache().
|
private |
Whether to limit memory use of sparse fields from disk. Enables the cache and dynamic loading when true.
Definition at line 527 of file SparseFile.h.
Referenced by activateBlock(), doLimitMemUse(), and setLimitMemUse().
|
private |
Vector containing information for each of the managed fields. The order matches the index stored in each SparseField::m_fileId.
Definition at line 531 of file SparseFile.h.
Referenced by activateBlock(), deallocateBlock(), decBlockRef(), getNextId(), incBlockRef(), memSize(), numLoadedBlocks(), reference(), removeFieldFromCache(), resetCacheStatistics(), totalLoadedBlocks(), and totalLoads().
|
private |
List of dynamically loaded blocks to be considered for unloading when the cache is full. Currently using Second-chance/Clock paging algorithm. For a description of the algorithm, look at: http://en.wikipedia.org/wiki/Page_replacement_algorithm#Second-chance.
Definition at line 538 of file SparseFile.h.
Referenced by addBlockToCache(), deallocateBlock(), deallocateBlocks(), flushCache(), memSize(), removeFieldFromCache(), and SparseFileManager().
|
private |
Pointer to the next block to test for unloading in the cache, the "hand" of the clock.
Definition at line 542 of file SparseFile.h.
Referenced by addBlockToCache(), deallocateBlock(), deallocateBlocks(), flushCache(), removeFieldFromCache(), and SparseFileManager().
|
mutableprivate |
Mutex to prevent multiple threads from deallocating blocks at the same time.
Definition at line 546 of file SparseFile.h.
Referenced by activateBlock(), deallocateBlocks(), flushCache(), getNextId(), memSize(), and removeFieldFromCache().