Set of options for snapshot manager. More...

Detailed Description

Set of options for snapshot manager.

This is a POD struct. Default destructor/copy-constructor/assignment operator work fine.

Definition at line 35 of file snapshot_options.hpp.

#include <snapshot_options.hpp>

Inheritance diagram for foedus::snapshot::SnapshotOptions:

[legend]

Collaboration diagram for foedus::snapshot::SnapshotOptions:

[legend]

Public Types
enum	Constants { kDefaultSnapshotTriggerPagePoolPercent = 100, kDefaultSnapshotIntervalMilliseconds = 60000, kDefaultLogMapperBucketKb = 1024, kDefaultLogMapperIoBufferMb = 64, kDefaultLogReducerBufferMb = 256, kDefaultLogReducerDumpIoBufferMb = 8, kDefaultLogReducerReadIoBufferKb = 1024, kDefaultSnapshotWriterPagePoolSizeMb = 128, kDefaultSnapshotWriterIntermediatePoolSizeMb = 16 }

Public Member Functions
	SnapshotOptions ()
	Constructs option values with default values. More...

std::string	convert_folder_path_pattern (int node) const
	converts folder_path_pattern_ into a string with the given node. More...

std::string	construct_snapshot_file_path (int snapshot_id, int node) const
	'folder_path'/snapshot_'snapshot-id'node'node-id'.data. More...

std::string	get_primary_folder_path () const
	Returns the path of first node, which is also used as the primary place to write out global files, such as snapshot metadata. More...

ErrorStack	load (tinyxml2::XMLElement *element) override
	Reads the content of this object from the given XML element. More...

ErrorStack	save (tinyxml2::XMLElement *element) const override
	Writes the content of this object to the given XML element. More...

const char *	get_tag_name () const override
	Returns an XML tag name for this object as a root element. More...

void	assign (const foedus::externalize::Externalizable *other) override
	Polymorphic assign operator. More...

Public Member Functions inherited from foedus::externalize::Externalizable
ErrorStack	load_from_string (const std::string &xml)
	Load the content of this object from the given XML string. More...

void	save_to_stream (std::ostream *ptr) const
	Invokes save() and directs the resulting XML text to the given stream. More...

ErrorStack	load_from_file (const fs::Path &path)
	Load the content of this object from the specified XML file. More...

ErrorStack	save_to_file (const fs::Path &path) const
	Atomically and durably writes out this object to the specified XML file. More...

Public Attributes
fs::FixedPath	folder_path_pattern_
	String pattern of path of snapshot folders in each NUMA node. More...

uint16_t	snapshot_trigger_page_pool_percent_
	When the main page pool runs under this percent (roughly calculated) of free pages, snapshot manager starts snapshotting to drop volatile pages even before the interval. More...

uint32_t	snapshot_interval_milliseconds_
	Interval in milliseconds to take snapshots. More...

uint32_t	log_mapper_bucket_kb_
	The size in KB of bucket (buffer for each partition) in mapper. More...

uint16_t	log_mapper_io_buffer_mb_
	The size in MB of IO buffer to read log files in mapper. More...

bool	log_mapper_sort_before_send_
	Whether to sort logs in mapper side before sending it to reducer. More...

uint32_t	log_reducer_buffer_mb_
	The size in MB of a buffer to store log entries in reducer (partition). More...

uint32_t	log_reducer_dump_io_buffer_mb_
	The size in MB of a buffer to write out sorted log entries in reducer to a temporary file. More...

uint32_t	log_reducer_read_io_buffer_kb_
	The size in KB of a buffer in reducer to read one temporary file. More...

uint32_t	snapshot_writer_page_pool_size_mb_
	The size in MB of one snapshot writer, which holds data pages modified in the snapshot and them sequentially dumps them to a file for each storage. More...

uint32_t	snapshot_writer_intermediate_pool_size_mb_
	The size in MB of additional page pool for one snapshot writer just for holding intermediate pages. More...

foedus::fs::DeviceEmulationOptions	emulation_
	Settings to emulate slower data device. More...

Friends
std::ostream &	operator<< (std::ostream &o, const SnapshotOptions &v)

Additional Inherited Members
Static Public Member Functions inherited from foedus::externalize::Externalizable
static ErrorStack	insert_comment (tinyxml2::XMLElement *element, const std::string &comment)

static ErrorStack	append_comment (tinyxml2::XMLElement *parent, const std::string &comment)

static ErrorStack	create_element (tinyxml2::XMLElement parent, const std::string &name, tinyxml2::XMLElement *out)

template<typename T >
static ErrorStack	add_element (tinyxml2::XMLElement *parent, const std::string &tag, const std::string &comment, T value)
	Only declaration in header. More...

template<uint MAXLEN, typename CHAR >
static ErrorStack	add_element (tinyxml2::XMLElement *parent, const std::string &tag, const std::string &comment, const assorted::FixedString< MAXLEN, CHAR > &value)

template<typename T >
static ErrorStack	add_element (tinyxml2::XMLElement *parent, const std::string &tag, const std::string &comment, const std::vector< T > &value)
	vector version More...

template<typename ENUM >
static ErrorStack	add_enum_element (tinyxml2::XMLElement *parent, const std::string &tag, const std::string &comment, ENUM value)
	enum version More...

static ErrorStack	add_child_element (tinyxml2::XMLElement *parent, const std::string &tag, const std::string &comment, const Externalizable &child)
	child Externalizable version More...

template<typename T >
static ErrorStack	get_element (tinyxml2::XMLElement parent, const std::string &tag, T out, bool optional=false, T value=0)
	Only declaration in header. More...

static ErrorStack	get_element (tinyxml2::XMLElement parent, const std::string &tag, std::string out, bool optional=false, const char *value="")
	string type is bit special. More...

template<uint MAXLEN, typename CHAR >
static ErrorStack	get_element (tinyxml2::XMLElement parent, const std::string &tag, assorted::FixedString< MAXLEN, CHAR > out, bool optional=false, const assorted::FixedString< MAXLEN, CHAR > &value=assorted::FixedString< MAXLEN, CHAR >())

template<typename ENUM >
static ErrorStack	get_enum_element (tinyxml2::XMLElement parent, const std::string &tag, ENUM out, bool optional=false, ENUM default_value=static_cast< ENUM >(0))
	enum version More...

template<typename T >
static ErrorStack	get_element (tinyxml2::XMLElement parent, const std::string &tag, std::vector< T > out, bool optional=false)
	vector version. More...

static ErrorStack	get_child_element (tinyxml2::XMLElement parent, const std::string &tag, Externalizable child, bool optional=false)
	child Externalizable version More...

Member Enumeration Documentation

enum foedus::snapshot::SnapshotOptions::Constants

Enumerator
kDefaultSnapshotTriggerPagePoolPercent
kDefaultSnapshotIntervalMilliseconds
kDefaultLogMapperBucketKb
kDefaultLogMapperIoBufferMb
kDefaultLogReducerBufferMb
kDefaultLogReducerDumpIoBufferMb
kDefaultLogReducerReadIoBufferKb
kDefaultSnapshotWriterPagePoolSizeMb
kDefaultSnapshotWriterIntermediatePoolSizeMb

Definition at line 36 of file snapshot_options.hpp.

                  {
     kDefaultSnapshotTriggerPagePoolPercent = 100,
     kDefaultSnapshotIntervalMilliseconds  = 60000,
     kDefaultLogMapperBucketKb             = 1024,
     kDefaultLogMapperIoBufferMb           = 64,
     kDefaultLogReducerBufferMb            = 256,
     kDefaultLogReducerDumpIoBufferMb      = 8,
     kDefaultLogReducerReadIoBufferKb      = 1024,
     kDefaultSnapshotWriterPagePoolSizeMb  = 128,
     kDefaultSnapshotWriterIntermediatePoolSizeMb  = 16,
   };

Constructor & Destructor Documentation

foedus::snapshot::SnapshotOptions::SnapshotOptions ( )

Constructs option values with default values.

Definition at line 26 of file snapshot_options.cpp.

References folder_path_pattern_, kDefaultLogMapperBucketKb, kDefaultLogMapperIoBufferMb, kDefaultLogReducerBufferMb, kDefaultLogReducerDumpIoBufferMb, kDefaultLogReducerReadIoBufferKb, kDefaultSnapshotIntervalMilliseconds, kDefaultSnapshotTriggerPagePoolPercent, kDefaultSnapshotWriterIntermediatePoolSizeMb, kDefaultSnapshotWriterPagePoolSizeMb, log_mapper_bucket_kb_, log_mapper_io_buffer_mb_, log_mapper_sort_before_send_, log_reducer_buffer_mb_, log_reducer_dump_io_buffer_mb_, log_reducer_read_io_buffer_kb_, snapshot_interval_milliseconds_, snapshot_trigger_page_pool_percent_, snapshot_writer_intermediate_pool_size_mb_, and snapshot_writer_page_pool_size_mb_.

                                  {
   folder_path_pattern_ = "snapshots/node_$NODE$";
   snapshot_trigger_page_pool_percent_ = kDefaultSnapshotTriggerPagePoolPercent;
   snapshot_interval_milliseconds_ = kDefaultSnapshotIntervalMilliseconds;
   log_mapper_bucket_kb_ = kDefaultLogMapperBucketKb;
   log_mapper_io_buffer_mb_ = kDefaultLogMapperIoBufferMb;
   log_mapper_sort_before_send_ = true;
   log_reducer_buffer_mb_ = kDefaultLogReducerBufferMb;
   log_reducer_dump_io_buffer_mb_ = kDefaultLogReducerDumpIoBufferMb;
   log_reducer_read_io_buffer_kb_ = kDefaultLogReducerReadIoBufferKb;
   snapshot_writer_page_pool_size_mb_ = kDefaultSnapshotWriterPagePoolSizeMb;
   snapshot_writer_intermediate_pool_size_mb_ = kDefaultSnapshotWriterIntermediatePoolSizeMb;
 }

Member Function Documentation

void foedus::snapshot::SnapshotOptions::assign ( const foedus::externalize::Externalizable * other )

inlineoverridevirtual

Polymorphic assign operator.

This should invoke operator= of the derived class.

Parameters

[in] other assigned value. It must be dynamic-castable to the assignee class.

Implements foedus::externalize::Externalizable.

Definition at line 163 of file snapshot_options.hpp.

std::string foedus::snapshot::SnapshotOptions::construct_snapshot_file_path	(	int	snapshot_id,
		int	node
	)		const

'folder_path'/snapshot_'snapshot-id'node'node-id'.data.

Definition at line 44 of file snapshot_options.cpp.

References convert_folder_path_pattern().

Referenced by foedus::cache::SnapshotFileSet::get_or_open_file().

                                                                                        {
   return convert_folder_path_pattern(node)
     + std::string("/snapshot_")
     + std::to_string(snapshot_id)
     + std::string("_")
     + std::to_string(node);
 }

Here is the call graph for this function:

Here is the caller graph for this function:

std::string foedus::snapshot::SnapshotOptions::convert_folder_path_pattern ( int node ) const

converts folder_path_pattern_ into a string with the given node.

Definition at line 40 of file snapshot_options.cpp.

References folder_path_pattern_, foedus::assorted::replace_all(), and foedus::assorted::FixedString< MAXLEN, CHAR >::str().

Referenced by construct_snapshot_file_path(), and get_primary_folder_path().

                                                                      {
   return assorted::replace_all(folder_path_pattern_.str(), "$NODE$", node);
 }

Here is the call graph for this function:

Here is the caller graph for this function:

std::string foedus::snapshot::SnapshotOptions::get_primary_folder_path ( ) const

inline

Returns the path of first node, which is also used as the primary place to write out global files, such as snapshot metadata.

Definition at line 159 of file snapshot_options.hpp.

References convert_folder_path_pattern().

                                                 {
     return convert_folder_path_pattern(0);
   }

Here is the call graph for this function:

const char* foedus::snapshot::SnapshotOptions::get_tag_name ( ) const

inlineoverridevirtual

Returns an XML tag name for this object as a root element.

We might want to give a different name for same externalizable objects, so this is used only when it is the root element of xml.

Implements foedus::externalize::Externalizable.

Definition at line 163 of file snapshot_options.hpp.

ErrorStack foedus::snapshot::SnapshotOptions::load ( tinyxml2::XMLElement * element )

overridevirtual

Reads the content of this object from the given XML element.

Parameters

[in] element the XML element that represents this object

Expect errors due to missing-elements, out-of-range values, etc.

Implements foedus::externalize::Externalizable.

Definition at line 53 of file snapshot_options.cpp.

References CHECK_ERROR, emulation_, EXTERNALIZE_LOAD_ELEMENT, folder_path_pattern_, foedus::externalize::Externalizable::get_child_element(), foedus::kRetOk, log_mapper_bucket_kb_, log_mapper_io_buffer_mb_, log_mapper_sort_before_send_, log_reducer_buffer_mb_, log_reducer_dump_io_buffer_mb_, log_reducer_read_io_buffer_kb_, snapshot_interval_milliseconds_, snapshot_trigger_page_pool_percent_, snapshot_writer_intermediate_pool_size_mb_, and snapshot_writer_page_pool_size_mb_.

                                                             {
   EXTERNALIZE_LOAD_ELEMENT(element, folder_path_pattern_);
   EXTERNALIZE_LOAD_ELEMENT(element, snapshot_trigger_page_pool_percent_);
   EXTERNALIZE_LOAD_ELEMENT(element, snapshot_interval_milliseconds_);
   EXTERNALIZE_LOAD_ELEMENT(element, log_mapper_bucket_kb_);
   EXTERNALIZE_LOAD_ELEMENT(element, log_mapper_io_buffer_mb_);
   EXTERNALIZE_LOAD_ELEMENT(element, log_mapper_sort_before_send_);
   EXTERNALIZE_LOAD_ELEMENT(element, log_reducer_buffer_mb_);
   EXTERNALIZE_LOAD_ELEMENT(element, log_reducer_dump_io_buffer_mb_);
   EXTERNALIZE_LOAD_ELEMENT(element, log_reducer_read_io_buffer_kb_);
   EXTERNALIZE_LOAD_ELEMENT(element, snapshot_writer_page_pool_size_mb_);
   EXTERNALIZE_LOAD_ELEMENT(element, snapshot_writer_intermediate_pool_size_mb_);
   CHECK_ERROR(get_child_element(element, "SnapshotDeviceEmulationOptions", &emulation_))
   return kRetOk;
 }

Here is the call graph for this function:

ErrorStack foedus::snapshot::SnapshotOptions::save ( tinyxml2::XMLElement * element ) const

overridevirtual

Writes the content of this object to the given XML element.

Parameters

[in] element the XML element that represents this object

Expect only out-of-memory error. We receive the XML element this object will represent, so this method does not determine the XML element name of itself. The parent object determines children's tag names because one parent object might have multiple child objects of the same type with different XML element name.

Implements foedus::externalize::Externalizable.

Definition at line 69 of file snapshot_options.cpp.

References foedus::externalize::Externalizable::add_child_element(), CHECK_ERROR, emulation_, EXTERNALIZE_SAVE_ELEMENT, folder_path_pattern_, foedus::externalize::Externalizable::insert_comment(), foedus::kRetOk, log_mapper_bucket_kb_, log_mapper_io_buffer_mb_, log_mapper_sort_before_send_, log_reducer_buffer_mb_, log_reducer_dump_io_buffer_mb_, log_reducer_read_io_buffer_kb_, snapshot_interval_milliseconds_, snapshot_trigger_page_pool_percent_, snapshot_writer_intermediate_pool_size_mb_, and snapshot_writer_page_pool_size_mb_.

                                                                   {
   CHECK_ERROR(insert_comment(element, "Set of options for snapshot manager"));
 
   EXTERNALIZE_SAVE_ELEMENT(element, folder_path_pattern_,
     "String pattern of path of snapshot folders in each NUMA node.\n"
     "This specifies the path of the folders to contain snapshot files in each NUMA node.\n"
     " A special placeholder $NODE$ will be replaced with the NUMA node number."
     " For example, /data/node_$NODE$ becomes /data/node_1 on node-1.");
   EXTERNALIZE_SAVE_ELEMENT(element, snapshot_trigger_page_pool_percent_,
     "When the main page pool runs under this percent (roughly calculated) of free pages,\n"
     " snapshot manager starts snapshotting to drop volatile pages even before the interval.");
   EXTERNALIZE_SAVE_ELEMENT(element, snapshot_interval_milliseconds_,
     "Interval in milliseconds to take snapshots.");
   EXTERNALIZE_SAVE_ELEMENT(element, log_mapper_bucket_kb_,
     "Size in KB of bucket (buffer for each partition) in mapper."
     " The larger, the less freuquently each mapper communicates with reducers."
     " 1024 (1MB) should be a good number.");
   EXTERNALIZE_SAVE_ELEMENT(element, log_mapper_io_buffer_mb_,
     "Size in MB of IO buffer to read log files in mapper."
     " This buffer is also the unit of batch processing in mapper.");
   EXTERNALIZE_SAVE_ELEMENT(element, log_mapper_sort_before_send_,
     "Whether to sort logs in mapper side before sending it to reducer.");
   EXTERNALIZE_SAVE_ELEMENT(element, log_reducer_buffer_mb_,
     "The size in MB of a buffer to store log entries in reducer (partition).");
   EXTERNALIZE_SAVE_ELEMENT(element, log_reducer_dump_io_buffer_mb_,
     "The size in MB of a buffer to write out sorted log entries in reducer to a temporary file.");
   EXTERNALIZE_SAVE_ELEMENT(element, log_reducer_read_io_buffer_kb_,
     "The size in KB of a buffer in reducer to read one temporary file. Note that the total"
     " memory consumption is this number times the number of temporary files. It's a merge-sort.");
   EXTERNALIZE_SAVE_ELEMENT(element, snapshot_writer_page_pool_size_mb_,
     "The size in MB of one snapshot writer, which holds data pages modified in the snapshot"
     " and them sequentially dumps them to a file for each storage.");
   EXTERNALIZE_SAVE_ELEMENT(element, snapshot_writer_intermediate_pool_size_mb_,
     "The size in MB of additional page pool for one snapshot writer just for holding"
     " intermediate pages.");
   CHECK_ERROR(add_child_element(element, "SnapshotDeviceEmulationOptions",
           "[Experiments-only] Settings to emulate slower data device", emulation_));
   return kRetOk;
 }

Here is the call graph for this function:

Friends And Related Function Documentation

std::ostream& operator<<	(	std::ostream &	o,
		const SnapshotOptions &	v
	)

friend

Definition at line 163 of file snapshot_options.hpp.

Member Data Documentation

foedus::fs::DeviceEmulationOptions foedus::snapshot::SnapshotOptions::emulation_

Settings to emulate slower data device.

Definition at line 147 of file snapshot_options.hpp.

Referenced by foedus::snapshot::LogMapper::handle_process(), load(), foedus::snapshot::SnapshotWriter::open(), and save().

fs::FixedPath foedus::snapshot::SnapshotOptions::folder_path_pattern_

String pattern of path of snapshot folders in each NUMA node.

This specifies the path of the folders to contain snapshot files in each NUMA node. A special placeholder $NODE$ will be replaced with the NUMA node number. For example, "/data/node_$NODE$" becomes "/data/node_1" on node-1.

It is optional. You can specify a fixed path without the patterns, which means you will use the same folder at all nodes. Even in that case, snapshot file names include uniquefiers, so it wouldn't cause any data corruption. It just makes things harder for poor sysadmins.

The default value is "snapshots/node_$NODE$".

Definition at line 67 of file snapshot_options.hpp.

Referenced by convert_folder_path_pattern(), load(), save(), and SnapshotOptions().

uint32_t foedus::snapshot::SnapshotOptions::log_mapper_bucket_kb_

The size in KB of bucket (buffer for each partition) in mapper.

The larger, the less freuquently each mapper communicates with reducers. 1024 (1MB) should be a good number.

Definition at line 87 of file snapshot_options.hpp.

Referenced by foedus::snapshot::LogMapper::initialize_once(), load(), save(), and SnapshotOptions().

uint16_t foedus::snapshot::SnapshotOptions::log_mapper_io_buffer_mb_

The size in MB of IO buffer to read log files in mapper.

This buffer is also the unit of batch processing in mapper, so this number should be sufficiently large. Maximum size is 1 << 15 MB (otherwise we can't represent log position in 4 bytes).

Definition at line 95 of file snapshot_options.hpp.

Referenced by foedus::snapshot::LogMapper::initialize_once(), load(), save(), and SnapshotOptions().

bool foedus::snapshot::SnapshotOptions::log_mapper_sort_before_send_

Whether to sort logs in mapper side before sending it to reducer.

Recuder anyway has to sort it again to merge multiple blocks, but probably it might reduce the work in reducer side (sorting "almost-sorted" array is faster depending on sort algo). default is true.

Definition at line 103 of file snapshot_options.hpp.

Referenced by load(), save(), and SnapshotOptions().

uint32_t foedus::snapshot::SnapshotOptions::log_reducer_buffer_mb_

The size in MB of a buffer to store log entries in reducer (partition).

Each reducer receives log entries from all mappers, so the right size is likely much larger than log_mapper_bucket_kb_.

Reducer sorts and dumps out this buffer to a file, then does merge-sort at the end. If this buffer can contain all the logs while snapshotting, it will not do any I/O thus be significanltly faster. If you have a big DRAM, you might want to specify a large number for that reason.

Definition at line 115 of file snapshot_options.hpp.

Referenced by foedus::snapshot::LogReducerRef::append_log_chunk(), foedus::soc::SharedMemoryRepo::calculate_node_memory_size(), foedus::snapshot::LogReducerRef::get_buffer_size(), load(), foedus::snapshot::LogReducer::LogReducer(), save(), and SnapshotOptions().

uint32_t foedus::snapshot::SnapshotOptions::log_reducer_dump_io_buffer_mb_

The size in MB of a buffer to write out sorted log entries in reducer to a temporary file.

Definition at line 120 of file snapshot_options.hpp.

Referenced by foedus::snapshot::LogReducer::initialize_once(), load(), save(), and SnapshotOptions().

uint32_t foedus::snapshot::SnapshotOptions::log_reducer_read_io_buffer_kb_

The size in KB of a buffer in reducer to read one temporary file.

Note that the total memory consumption is this number times the number of temporary files. It's a merge-sort.

Definition at line 127 of file snapshot_options.hpp.

Referenced by load(), save(), and SnapshotOptions().

uint32_t foedus::snapshot::SnapshotOptions::snapshot_interval_milliseconds_

Interval in milliseconds to take snapshots.

Default is one minute.

Definition at line 80 of file snapshot_options.hpp.

Referenced by load(), save(), and SnapshotOptions().

uint16_t foedus::snapshot::SnapshotOptions::snapshot_trigger_page_pool_percent_

When the main page pool runs under this percent (roughly calculated) of free pages, snapshot manager starts snapshotting to drop volatile pages even before the interval.

Default is 100 (no check).

Definition at line 74 of file snapshot_options.hpp.

Referenced by load(), save(), and SnapshotOptions().

uint32_t foedus::snapshot::SnapshotOptions::snapshot_writer_intermediate_pool_size_mb_

The size in MB of additional page pool for one snapshot writer just for holding intermediate pages.

We hold intermediate pages that have pointers to other pages separate from leaf pages. This page pool can be usually much smaller than snapshot_writer_page_pool_size_mb_.

Definition at line 144 of file snapshot_options.hpp.

Referenced by foedus::snapshot::LogReducer::initialize_once(), load(), save(), and SnapshotOptions().

uint32_t foedus::snapshot::SnapshotOptions::snapshot_writer_page_pool_size_mb_

The size in MB of one snapshot writer, which holds data pages modified in the snapshot and them sequentially dumps them to a file for each storage.

Ideally, this size should be more than the maximum size of data pages modifed in each storage in each snapshot. Note that the total memory consumption is this number times the number of reducers (nodes).

Definition at line 136 of file snapshot_options.hpp.

Referenced by foedus::snapshot::LogReducer::initialize_once(), load(), save(), and SnapshotOptions().

The documentation for this struct was generated from the following files:

/home/shino/foedus_code/foedus-core/include/foedus/snapshot/snapshot_options.hpp
/home/shino/foedus_code/foedus-core/src/foedus/snapshot/snapshot_options.cpp

Detailed Description

Public Types

Public Member Functions

Public Attributes

Friends

Additional Inherited Members

Member Enumeration Documentation

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation

Member Data Documentation