Represents one input stream of sorted log entries. More...

Detailed Description

Represents one input stream of sorted log entries.

At the end of reducer, the reducer reads dumped sorted run files and does the final apply phase to construct the new version of each storage's pages. This is done by invoking each storage's composer interface with this object as inputs. To avoid invoking composer for each log entry (which is very expensive), this object allows the composer to receive sufficiently large buffers and do sort-merge itself.

In-memory and On-disk: There are two implementations of this buffer. InMemorySortedBuffer and DumpFileSortedBuffer. See their comments for more details.

Current block: The buffer logically conveys log entries of one storage because each composer is responsible for only one storage. However, each sorted run file or in-memory buffer packs a bunch of storages. So, each buffer conveys begin/end positions of the current storage block where the relevant log entries start and end.

Absolute and Relative positions: Finally, the buffer fundamentally has a limited size. It can't contain all the data at once. Thus, we use absolute (byte position in entire file) and relative (byte position in buffer) positions. In in-memory buffer, these two are the same. To simplify, let "entire file size" or "file size" mean the file size in on-disk buffer, the buffer size in in-memory buffer.

Buffer window, offset, and byte positions

Definition at line 81 of file log_buffer.hpp.

#include <log_buffer.hpp>

Inheritance diagram for foedus::snapshot::SortedBuffer:

[legend]

Public Member Functions
	SortedBuffer (char *buffer, uint64_t buffer_size, uint64_t total_size)

virtual	~SortedBuffer ()

uint64_t	to_relative_pos (uint64_t absolute_pos) const

uint64_t	to_absolute_pos (uint64_t relative_pos) const

const char *	get_buffer () const
	Returns the buffer memory. More...

uint64_t	get_buffer_size () const
	Returns the size of buffer memory. More...

uint64_t	get_offset () const
	Returns the absolute byte position of the buffer's beginning in the entire file. More...

uint64_t	get_total_size () const
	Returns the total size of the underlying stream (eg file size). More...

storage::StorageId	get_cur_block_storage_id () const

uint32_t	get_cur_block_log_count () const

uint64_t	get_cur_block_abosulte_begin () const
	Current storage block's beginning in absolute byte position in the file. More...

uint64_t	get_cur_block_abosulte_end () const
	Current storage block's end in absolute byte position in the file. More...

uint32_t	get_cur_block_shortest_key_length () const
	Current storage block's shortest key length. More...

uint32_t	get_cur_block_longest_key_length () const
	Current storage block's longest key length. More...

void	set_current_block (storage::StorageId storage_id, uint32_t log_count, uint64_t abosulte_begin, uint64_t abosulte_end, uint32_t shortest_key_length, uint32_t longest_key_length)

bool	is_valid_current_block () const

void	invalidate_current_block ()

virtual ErrorCode	wind (uint64_t next_absolute_pos)=0
	Loads next data block to be consumed by the caller (composer). More...

virtual std::string	to_string () const =0
	Returns a short string that briefly describes this object. More...

virtual void	describe (std::ostream *o) const =0
	Writes out a detailed description of this object to stream. More...

void	assert_checks ()

Protected Member Functions
void	describe_base_elements (std::ostream *o) const

Protected Attributes
char *const	buffer_

const uint64_t	buffer_size_

uint64_t	offset_
	see get_offset() More...

const uint64_t	total_size_
	see get_total_size() More...

storage::StorageId	cur_block_storage_id_
	If this is zero, this buffer is not set for reading any block. More...

uint32_t	cur_block_log_count_

uint64_t	cur_block_abosulte_begin_
	see get_cur_block_abosulte_begin() More...

uint64_t	cur_block_abosulte_end_
	see get_cur_block_abosulte_end() More...

uint32_t	cur_block_shortest_key_length_
	additional statistics for masstree/hash More...

uint32_t	cur_block_longest_key_length_
	additional statistics for masstree/hash More...

Friends
std::ostream &	operator<< (std::ostream &o, const SortedBuffer &v)

Constructor & Destructor Documentation

foedus::snapshot::SortedBuffer::SortedBuffer	(	char *	buffer,
		uint64_t	buffer_size,
		uint64_t	total_size
	)

inline

Definition at line 83 of file log_buffer.hpp.

References assert_checks().

     : buffer_(buffer),
       buffer_size_(buffer_size),
       offset_(0),
       total_size_(total_size),
       cur_block_storage_id_(0),
       cur_block_log_count_(0),
       cur_block_abosulte_begin_(0),
       cur_block_abosulte_end_(0) {
     assert_checks();
   }

Here is the call graph for this function:

virtual foedus::snapshot::SortedBuffer::~SortedBuffer ( )

inlinevirtual

Definition at line 94 of file log_buffer.hpp.

94 {}

Member Function Documentation

void foedus::snapshot::SortedBuffer::assert_checks ( )

inline

Definition at line 177 of file log_buffer.hpp.

References ASSERT_ND, buffer_size_, cur_block_abosulte_begin_, cur_block_abosulte_end_, cur_block_log_count_, cur_block_longest_key_length_, cur_block_shortest_key_length_, cur_block_storage_id_, offset_, and total_size_.

Referenced by foedus::storage::sequential::StreamStatus::init(), foedus::snapshot::MergeSort::initialize_once(), set_current_block(), SortedBuffer(), and foedus::snapshot::DumpFileSortedBuffer::wind().

                        {
     ASSERT_ND(offset_ <= total_size_);
     if (cur_block_storage_id_ != 0) {
       ASSERT_ND(cur_block_log_count_ > 0);
       ASSERT_ND(cur_block_abosulte_begin_ <= cur_block_abosulte_end_);
       ASSERT_ND(cur_block_abosulte_end_ <= total_size_);
 
       // did we overlook the begin/end of the block?
       ASSERT_ND(cur_block_abosulte_begin_ <= offset_ + buffer_size_);
       ASSERT_ND(cur_block_abosulte_end_ >= offset_);
       ASSERT_ND(cur_block_shortest_key_length_ <= cur_block_longest_key_length_);
       ASSERT_ND(cur_block_shortest_key_length_ > 0);
       ASSERT_ND(cur_block_longest_key_length_ <= 0xFFFFU);
     }
   }

Here is the caller graph for this function:

virtual void foedus::snapshot::SortedBuffer::describe ( std::ostream * o ) const

pure virtual

Writes out a detailed description of this object to stream.

Implemented in foedus::snapshot::DumpFileSortedBuffer, and foedus::snapshot::InMemorySortedBuffer.

Referenced by foedus::snapshot::operator<<().

Here is the caller graph for this function:

void foedus::snapshot::SortedBuffer::describe_base_elements ( std::ostream * o ) const

protected

Definition at line 37 of file log_buffer.cpp.

References buffer_, buffer_size_, cur_block_abosulte_begin_, cur_block_abosulte_end_, cur_block_log_count_, cur_block_storage_id_, offset_, and total_size_.

Referenced by foedus::snapshot::InMemorySortedBuffer::describe(), and foedus::snapshot::DumpFileSortedBuffer::describe().

                                                                 {
   std::ostream& o = *optr;
   o << "<buffer_>" << reinterpret_cast<const void*>(buffer_) << "</buffer_>"
     << "<buffer_size_>" << buffer_size_ << "</buffer_size_>"
     << "<offset_>" << offset_ << "</offset_>"
     << "<total_size_>" << total_size_ << "</total_size_>"
     << "<cur_block_storage_id_>" << cur_block_storage_id_ << "</cur_block_storage_id_>"
     << "<cur_block_log_count_>" << cur_block_log_count_ << "</cur_block_log_count_>"
     << "<cur_block_abosulte_begin_>" << cur_block_abosulte_begin_ << "</cur_block_abosulte_begin_>"
     << "<cur_block_abosulte_end_>" << cur_block_abosulte_end_ << "</cur_block_abosulte_end_>";
 }

Here is the caller graph for this function:

const char* foedus::snapshot::SortedBuffer::get_buffer ( ) const

inline

Returns the buffer memory.

Definition at line 108 of file log_buffer.hpp.

References buffer_.

Referenced by foedus::storage::sequential::StreamStatus::init(), and foedus::snapshot::MergeSort::initialize_once().

108 { return buffer_; }

foedus::snapshot::SortedBuffer::buffer_

char *const buffer_

Definition: log_buffer.hpp:194

Here is the caller graph for this function:

uint64_t foedus::snapshot::SortedBuffer::get_buffer_size ( ) const

inline

Returns the size of buffer memory.

Definition at line 111 of file log_buffer.hpp.

References buffer_size_.

Referenced by foedus::storage::sequential::StreamStatus::init(), and foedus::snapshot::MergeSort::initialize_once().

111 { return buffer_size_; }

foedus::snapshot::SortedBuffer::buffer_size_

const uint64_t buffer_size_

Definition: log_buffer.hpp:195

Here is the caller graph for this function:

uint64_t foedus::snapshot::SortedBuffer::get_cur_block_abosulte_begin ( ) const

inline

Current storage block's beginning in absolute byte position in the file.

Definition at line 129 of file log_buffer.hpp.

References cur_block_abosulte_begin_.

Referenced by foedus::storage::sequential::StreamStatus::init(), and foedus::snapshot::MergeSort::initialize_once().

129 { return cur_block_abosulte_begin_; }

foedus::snapshot::SortedBuffer::cur_block_abosulte_begin_

uint64_t cur_block_abosulte_begin_

see get_cur_block_abosulte_begin()

Definition: log_buffer.hpp:205

Here is the caller graph for this function:

uint64_t foedus::snapshot::SortedBuffer::get_cur_block_abosulte_end ( ) const

inline

Current storage block's end in absolute byte position in the file.

Definition at line 131 of file log_buffer.hpp.

References cur_block_abosulte_end_.

Referenced by foedus::storage::sequential::StreamStatus::init(), and foedus::snapshot::MergeSort::initialize_once().

131 { return cur_block_abosulte_end_; }

foedus::snapshot::SortedBuffer::cur_block_abosulte_end_

uint64_t cur_block_abosulte_end_

see get_cur_block_abosulte_end()

Definition: log_buffer.hpp:207

Here is the caller graph for this function:

uint32_t foedus::snapshot::SortedBuffer::get_cur_block_log_count ( ) const

inline

Definition at line 126 of file log_buffer.hpp.

References cur_block_log_count_.

126 { return cur_block_log_count_; }

foedus::snapshot::SortedBuffer::cur_block_log_count_

uint32_t cur_block_log_count_

Definition: log_buffer.hpp:203

uint32_t foedus::snapshot::SortedBuffer::get_cur_block_longest_key_length ( ) const

inline

Current storage block's longest key length.

Definition at line 135 of file log_buffer.hpp.

References cur_block_longest_key_length_.

Referenced by foedus::snapshot::extract_longest_key_length().

135 { return cur_block_longest_key_length_; }

foedus::snapshot::SortedBuffer::cur_block_longest_key_length_

uint32_t cur_block_longest_key_length_

additional statistics for masstree/hash

Definition: log_buffer.hpp:211

Here is the caller graph for this function:

uint32_t foedus::snapshot::SortedBuffer::get_cur_block_shortest_key_length ( ) const

inline

Current storage block's shortest key length.

Definition at line 133 of file log_buffer.hpp.

References cur_block_shortest_key_length_.

Referenced by foedus::snapshot::extract_shortest_key_length().

133 { return cur_block_shortest_key_length_; }

foedus::snapshot::SortedBuffer::cur_block_shortest_key_length_

uint32_t cur_block_shortest_key_length_

additional statistics for masstree/hash

Definition: log_buffer.hpp:209

Here is the caller graph for this function:

storage::StorageId foedus::snapshot::SortedBuffer::get_cur_block_storage_id ( ) const

inline

Definition at line 125 of file log_buffer.hpp.

References cur_block_storage_id_.

125 { return cur_block_storage_id_; }

foedus::snapshot::SortedBuffer::cur_block_storage_id_

storage::StorageId cur_block_storage_id_

If this is zero, this buffer is not set for reading any block.

Definition: log_buffer.hpp:202

uint64_t foedus::snapshot::SortedBuffer::get_offset ( ) const

inline

Returns the absolute byte position of the buffer's beginning in the entire file.

For example, buffer size = 1MB, file size = 1GB. Now we are reading 3200KB-4224KB region. In that case, this returns 3200KB. This value changes when the caller invokes wind() to shifts the buffer window.

Definition at line 120 of file log_buffer.hpp.

References offset_.

Referenced by foedus::storage::sequential::StreamStatus::init(), and foedus::snapshot::MergeSort::initialize_once().

120 { return offset_; }

foedus::snapshot::SortedBuffer::offset_

uint64_t offset_

see get_offset()

Definition: log_buffer.hpp:197

Here is the caller graph for this function:

uint64_t foedus::snapshot::SortedBuffer::get_total_size ( ) const

inline

Returns the total size of the underlying stream (eg file size).

Definition at line 123 of file log_buffer.hpp.

References total_size_.

123 { return total_size_; }

foedus::snapshot::SortedBuffer::total_size_

const uint64_t total_size_

see get_total_size()

Definition: log_buffer.hpp:199

void foedus::snapshot::SortedBuffer::invalidate_current_block ( )

inline

Definition at line 153 of file log_buffer.hpp.

References cur_block_storage_id_.

153 { cur_block_storage_id_ = 0; }

foedus::snapshot::SortedBuffer::cur_block_storage_id_

storage::StorageId cur_block_storage_id_

If this is zero, this buffer is not set for reading any block.

Definition: log_buffer.hpp:202

bool foedus::snapshot::SortedBuffer::is_valid_current_block ( ) const

inline

Definition at line 152 of file log_buffer.hpp.

References cur_block_storage_id_.

152 { return cur_block_storage_id_ != 0; }

foedus::snapshot::SortedBuffer::cur_block_storage_id_

storage::StorageId cur_block_storage_id_

If this is zero, this buffer is not set for reading any block.

Definition: log_buffer.hpp:202

void foedus::snapshot::SortedBuffer::set_current_block	(	storage::StorageId	storage_id,
		uint32_t	log_count,
		uint64_t	abosulte_begin,
		uint64_t	abosulte_end,
		uint32_t	shortest_key_length,
		uint32_t	longest_key_length
	)

inline

Definition at line 137 of file log_buffer.hpp.

References assert_checks(), cur_block_abosulte_begin_, cur_block_abosulte_end_, cur_block_log_count_, cur_block_longest_key_length_, cur_block_shortest_key_length_, and cur_block_storage_id_.

                                            {
     cur_block_storage_id_ = storage_id;
     cur_block_log_count_ = log_count;
     cur_block_abosulte_begin_ = abosulte_begin;
     cur_block_abosulte_end_ = abosulte_end;
     cur_block_shortest_key_length_ = shortest_key_length;
     cur_block_longest_key_length_ = longest_key_length;
     assert_checks();
   }

Here is the call graph for this function:

uint64_t foedus::snapshot::SortedBuffer::to_absolute_pos ( uint64_t relative_pos ) const

inline

Definition at line 102 of file log_buffer.hpp.

References ASSERT_ND, buffer_size_, and offset_.

                                                            {
     ASSERT_ND(relative_pos < buffer_size_);
     return relative_pos + offset_;
   }

uint64_t foedus::snapshot::SortedBuffer::to_relative_pos ( uint64_t absolute_pos ) const

inline

Definition at line 96 of file log_buffer.hpp.

References ASSERT_ND, buffer_size_, and offset_.

Referenced by foedus::storage::sequential::StreamStatus::wind_stream().

                                                            {
     ASSERT_ND(absolute_pos >= offset_);
     ASSERT_ND(absolute_pos <= offset_ + buffer_size_);
     return absolute_pos - offset_;
   }

Here is the caller graph for this function:

virtual std::string foedus::snapshot::SortedBuffer::to_string ( ) const

pure virtual

Returns a short string that briefly describes this object.

Implemented in foedus::snapshot::DumpFileSortedBuffer, and foedus::snapshot::InMemorySortedBuffer.

virtual ErrorCode foedus::snapshot::SortedBuffer::wind ( uint64_t next_absolute_pos )

pure virtual

Loads next data block to be consumed by the caller (composer).

Parameters

[in] next_absolute_pos the absolute byte position that will be guaranteed to be within the buffer after this method call. Note that this position might not become 0-th byte position after this method call because the buffer might be backed by an aligned File I/O.

Precondition: next_absolute_pos <= get_total_size(); next_absolute_pos >= get_offset() : we allow foward-only iteration.; next_absolute_pos <= get_offset() + get_buffer_size() : we allow contiguous-read only.

Postcondition: get_offset() + get_buffer_size() > next_absolute_pos >= get_offset()

Returns: File I/O related errors only

Implemented in foedus::snapshot::DumpFileSortedBuffer, and foedus::snapshot::InMemorySortedBuffer.

Referenced by foedus::storage::sequential::StreamStatus::wind_stream().

Here is the caller graph for this function:

Friends And Related Function Documentation

std::ostream& operator<<	(	std::ostream &	o,
		const SortedBuffer &	v
	)

friend

Definition at line 32 of file log_buffer.cpp.

                                                              {
   v.describe(&o);
   return o;
 }

Member Data Documentation

char* const foedus::snapshot::SortedBuffer::buffer_

protected

Definition at line 194 of file log_buffer.hpp.

Referenced by describe_base_elements(), get_buffer(), and foedus::snapshot::DumpFileSortedBuffer::wind().

const uint64_t foedus::snapshot::SortedBuffer::buffer_size_

protected

Definition at line 195 of file log_buffer.hpp.

Referenced by assert_checks(), describe_base_elements(), foedus::snapshot::DumpFileSortedBuffer::DumpFileSortedBuffer(), get_buffer_size(), to_absolute_pos(), to_relative_pos(), foedus::snapshot::InMemorySortedBuffer::wind(), and foedus::snapshot::DumpFileSortedBuffer::wind().

uint64_t foedus::snapshot::SortedBuffer::cur_block_abosulte_begin_

protected

see get_cur_block_abosulte_begin()

Definition at line 205 of file log_buffer.hpp.

Referenced by assert_checks(), describe_base_elements(), get_cur_block_abosulte_begin(), and set_current_block().

uint64_t foedus::snapshot::SortedBuffer::cur_block_abosulte_end_

protected

see get_cur_block_abosulte_end()

Definition at line 207 of file log_buffer.hpp.

Referenced by assert_checks(), describe_base_elements(), get_cur_block_abosulte_end(), and set_current_block().

uint32_t foedus::snapshot::SortedBuffer::cur_block_log_count_

protected

Definition at line 203 of file log_buffer.hpp.

Referenced by assert_checks(), describe_base_elements(), get_cur_block_log_count(), and set_current_block().

uint32_t foedus::snapshot::SortedBuffer::cur_block_longest_key_length_

protected

additional statistics for masstree/hash

Definition at line 211 of file log_buffer.hpp.

Referenced by assert_checks(), get_cur_block_longest_key_length(), and set_current_block().

uint32_t foedus::snapshot::SortedBuffer::cur_block_shortest_key_length_

protected

additional statistics for masstree/hash

Definition at line 209 of file log_buffer.hpp.

Referenced by assert_checks(), get_cur_block_shortest_key_length(), and set_current_block().

storage::StorageId foedus::snapshot::SortedBuffer::cur_block_storage_id_

protected

If this is zero, this buffer is not set for reading any block.

Definition at line 202 of file log_buffer.hpp.

Referenced by assert_checks(), describe_base_elements(), get_cur_block_storage_id(), invalidate_current_block(), is_valid_current_block(), and set_current_block().

uint64_t foedus::snapshot::SortedBuffer::offset_

protected

see get_offset()

Definition at line 197 of file log_buffer.hpp.

Referenced by assert_checks(), describe_base_elements(), get_offset(), to_absolute_pos(), to_relative_pos(), and foedus::snapshot::DumpFileSortedBuffer::wind().

const uint64_t foedus::snapshot::SortedBuffer::total_size_

protected

see get_total_size()

Definition at line 199 of file log_buffer.hpp.

Referenced by assert_checks(), describe_base_elements(), foedus::snapshot::DumpFileSortedBuffer::DumpFileSortedBuffer(), get_total_size(), and foedus::snapshot::DumpFileSortedBuffer::wind().

The documentation for this class was generated from the following files:

/home/shino/foedus_code/foedus-core/include/foedus/snapshot/log_buffer.hpp
/home/shino/foedus_code/foedus-core/src/foedus/snapshot/log_buffer.cpp

Detailed Description

Public Member Functions

Protected Member Functions

Protected Attributes

Friends

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation

Member Data Documentation