Implementation of SortedBuffer that is backed by a dumped file. More...

Detailed Description

Implementation of SortedBuffer that is backed by a dumped file.

When the reducer's buffer becomes full, it sorts all entries in it and dumps it to a file (sorted run file). This stream sequentially reads from the file.

Definition at line 247 of file log_buffer.hpp.

#include <log_buffer.hpp>

Inheritance diagram for foedus::snapshot::DumpFileSortedBuffer:

[legend]

Collaboration diagram for foedus::snapshot::DumpFileSortedBuffer:

[legend]

Public Member Functions
	DumpFileSortedBuffer (fs::DirectIoFile *file, memory::AlignedMemorySlice io_buffer)
	wraps the given file as a buffer. More...

	~DumpFileSortedBuffer () override

std::string	to_string () const override
	Returns a short string that briefly describes this object. More...

ErrorCode	wind (uint64_t next_absolute_pos) override
	Loads next data block to be consumed by the caller (composer). More...

void	describe (std::ostream *o) const override
	Writes out a detailed description of this object to stream. More...

fs::DirectIoFile *	get_file () const

const memory::AlignedMemorySlice &	get_io_buffer () const

Public Member Functions inherited from foedus::snapshot::SortedBuffer
	SortedBuffer (char *buffer, uint64_t buffer_size, uint64_t total_size)

virtual	~SortedBuffer ()

uint64_t	to_relative_pos (uint64_t absolute_pos) const

uint64_t	to_absolute_pos (uint64_t relative_pos) const

const char *	get_buffer () const
	Returns the buffer memory. More...

uint64_t	get_buffer_size () const
	Returns the size of buffer memory. More...

uint64_t	get_offset () const
	Returns the absolute byte position of the buffer's beginning in the entire file. More...

uint64_t	get_total_size () const
	Returns the total size of the underlying stream (eg file size). More...

storage::StorageId	get_cur_block_storage_id () const

uint32_t	get_cur_block_log_count () const

uint64_t	get_cur_block_abosulte_begin () const
	Current storage block's beginning in absolute byte position in the file. More...

uint64_t	get_cur_block_abosulte_end () const
	Current storage block's end in absolute byte position in the file. More...

uint32_t	get_cur_block_shortest_key_length () const
	Current storage block's shortest key length. More...

uint32_t	get_cur_block_longest_key_length () const
	Current storage block's longest key length. More...

void	set_current_block (storage::StorageId storage_id, uint32_t log_count, uint64_t abosulte_begin, uint64_t abosulte_end, uint32_t shortest_key_length, uint32_t longest_key_length)

bool	is_valid_current_block () const

void	invalidate_current_block ()

void	assert_checks ()

Additional Inherited Members
Protected Member Functions inherited from foedus::snapshot::SortedBuffer
void	describe_base_elements (std::ostream *o) const

Protected Attributes inherited from foedus::snapshot::SortedBuffer
char *const	buffer_

const uint64_t	buffer_size_

uint64_t	offset_
	see get_offset() More...

const uint64_t	total_size_
	see get_total_size() More...

storage::StorageId	cur_block_storage_id_
	If this is zero, this buffer is not set for reading any block. More...

uint32_t	cur_block_log_count_

uint64_t	cur_block_abosulte_begin_
	see get_cur_block_abosulte_begin() More...

uint64_t	cur_block_abosulte_end_
	see get_cur_block_abosulte_end() More...

uint32_t	cur_block_shortest_key_length_
	additional statistics for masstree/hash More...

uint32_t	cur_block_longest_key_length_
	additional statistics for masstree/hash More...

Constructor & Destructor Documentation

foedus::snapshot::DumpFileSortedBuffer::DumpFileSortedBuffer	(	fs::DirectIoFile *	file,
		memory::AlignedMemorySlice	io_buffer
	)

wraps the given file as a buffer.

note that this object does NOT take ownership of anything given to this constructor. it doesn't delete/open file or memory. it's caller's responsibility (in other words, this object is just a view. so is InMemorySortedBuffer).

Definition at line 57 of file log_buffer.cpp.

References ASSERT_ND, foedus::snapshot::SortedBuffer::buffer_size_, and foedus::snapshot::SortedBuffer::total_size_.

   : SortedBuffer(
     reinterpret_cast<char*>(io_buffer.get_block()),
     io_buffer.get_size(),
     fs::file_size(file->get_path())),
     file_(file),
     io_buffer_(io_buffer) {
   ASSERT_ND(buffer_size_ % kAlignment == 0);
   ASSERT_ND(total_size_ % kAlignment == 0);
 }

foedus::snapshot::DumpFileSortedBuffer::~DumpFileSortedBuffer ( )

inlineoverride

Definition at line 255 of file log_buffer.hpp.

255 {}

Member Function Documentation

void foedus::snapshot::DumpFileSortedBuffer::describe ( std::ostream * o ) const

overridevirtual

Writes out a detailed description of this object to stream.

Implements foedus::snapshot::SortedBuffer.

Definition at line 73 of file log_buffer.cpp.

References foedus::snapshot::SortedBuffer::describe_base_elements().

                                                           {
   std::ostream& o = *optr;
   o << "<DumpFileSortedBuffer>";
   describe_base_elements(optr);
   o << "<file_>" << file_ << "</file_>";
   o << "<io_buffer_>" << io_buffer_ << "</io_buffer_>";
   o << "</DumpFileSortedBuffer>";
 }

Here is the call graph for this function:

fs::DirectIoFile* foedus::snapshot::DumpFileSortedBuffer::get_file ( ) const

inline

Definition at line 261 of file log_buffer.hpp.

261 { return file_; }

const memory::AlignedMemorySlice& foedus::snapshot::DumpFileSortedBuffer::get_io_buffer ( ) const

inline

Definition at line 262 of file log_buffer.hpp.

262 { return io_buffer_; }

std::string foedus::snapshot::DumpFileSortedBuffer::to_string ( ) const

overridevirtual

Returns a short string that briefly describes this object.

Implements foedus::snapshot::SortedBuffer.

Definition at line 69 of file log_buffer.cpp.

References foedus::fs::DirectIoFile::get_path(), and foedus::fs::Path::string().

                                                 {
   return std::string("DumpFileSortedBuffer: ") + file_->get_path().string();
 }

Here is the call graph for this function:

ErrorCode foedus::snapshot::DumpFileSortedBuffer::wind ( uint64_t next_absolute_pos )

overridevirtual

Loads next data block to be consumed by the caller (composer).

Parameters

[in] next_absolute_pos the absolute byte position that will be guaranteed to be within the buffer after this method call. Note that this position might not become 0-th byte position after this method call because the buffer might be backed by an aligned File I/O.

Precondition: next_absolute_pos <= get_total_size(); next_absolute_pos >= get_offset() : we allow foward-only iteration.; next_absolute_pos <= get_offset() + get_buffer_size() : we allow contiguous-read only.

Postcondition: get_offset() + get_buffer_size() > next_absolute_pos >= get_offset()

Returns: File I/O related errors only

Implements foedus::snapshot::SortedBuffer.

Definition at line 82 of file log_buffer.cpp.

References foedus::snapshot::SortedBuffer::assert_checks(), ASSERT_ND, foedus::snapshot::SortedBuffer::buffer_, foedus::snapshot::SortedBuffer::buffer_size_, CHECK_ERROR_CODE, foedus::fs::DirectIoFile::get_current_offset(), foedus::kErrorCodeInvalidParameter, foedus::kErrorCodeOk, foedus::snapshot::SortedBuffer::offset_, foedus::fs::DirectIoFile::read(), and foedus::snapshot::SortedBuffer::total_size_.

                                                                {
   ASSERT_ND(offset_ % kAlignment == 0);
   assert_checks();
   if (next_absolute_pos == offset_ || offset_ + buffer_size_ >= total_size_) {
     return kErrorCodeOk;  // nothing to do then
   } else if (next_absolute_pos < offset_ ||  // backward wind
     next_absolute_pos >= offset_ + buffer_size_ ||  // jumps more than one buffer
     next_absolute_pos >= total_size_) {  // jumps exceeding the end of file
     // this class doesn't support these operations. We shouldn't need them in any case.
     LOG(FATAL) << " wtf next_absolute_pos=" << next_absolute_pos << ", offset=" << offset_
       << ", buffer_size=" << buffer_size_ << ", total_size_=" << total_size_;
     return kErrorCodeInvalidParameter;
   }
 
   // suppose buf=64M and we have read second window(64M-128M) and now moving on to third window.
   // in the easiest case, current offset_=64M, next_absolute_pos=128M. we just read 64M.
   // but, probably next_absolute_pos=128M-alpha, further alpha might not be 4k-aligned.
   // the following code takes care of those cases.
   ASSERT_ND(file_->get_current_offset() == offset_ + buffer_size_);
   uint64_t retained_bytes
     = assorted::align<uint64_t, kAlignment>(offset_ + buffer_size_ - next_absolute_pos);
   ASSERT_ND(retained_bytes <= buffer_size_);
   std::memmove(buffer_, buffer_ + buffer_size_ - retained_bytes, retained_bytes);
 
   // we read as much as possible. note that "next_absolute_pos + buffer_size_" might become larger
   // than cur_block_abosulte_end_. It's fine, the next DumpFileSortedBuffer object for another
   // storage will take care (and make use) of the excessive bytes.
   uint64_t desired_reads = std::min(
     buffer_size_ - retained_bytes,
     total_size_ - (offset_ + buffer_size_));
   memory::AlignedMemorySlice sub_slice(io_buffer_, retained_bytes, desired_reads);
   CHECK_ERROR_CODE(file_->read(desired_reads, sub_slice));
   offset_ = offset_ + buffer_size_ - retained_bytes;
 
   ASSERT_ND(offset_ % kAlignment == 0);
   ASSERT_ND(next_absolute_pos >= offset_);
   assert_checks();
   return kErrorCodeOk;
 }

Here is the call graph for this function:

The documentation for this class was generated from the following files:

/home/shino/foedus_code/foedus-core/include/foedus/snapshot/log_buffer.hpp
/home/shino/foedus_code/foedus-core/src/foedus/snapshot/log_buffer.cpp

Detailed Description

Public Member Functions

Additional Inherited Members

Constructor & Destructor Documentation

Member Function Documentation