foedus::memory::PagePoolPimpl Class Reference

Pimpl object of PagePool. More...

Detailed Description

Pimpl object of PagePool.

A private pimpl object for PagePool. Do not include this header from a client program unless you know what you are doing.

Definition at line 74 of file page_pool_pimpl.hpp.

#include <page_pool_pimpl.hpp>

Inheritance diagram for foedus::memory::PagePoolPimpl:

Collaboration diagram for foedus::memory::PagePoolPimpl:

Public Member Functions
	PagePoolPimpl ()
void	attach (PagePoolControlBlock *control_block, void *memory, uint64_t memory_size, bool owns, bool rigorous_page_boundary_check)
ErrorStack	initialize_once () override
ErrorStack	uninitialize_once () override
ErrorCode	grab (uint32_t desired_grab_count, PagePoolOffsetChunk *chunk)
template<typename CHUNK >
void	release_impl (uint32_t desired_release_count, CHUNK *chunk)
void	release (uint32_t desired_release_count, PagePoolOffsetChunk *chunk)
void	release (uint32_t desired_release_count, PagePoolOffsetDynamicChunk *chunk)
void	release (uint32_t desired_release_count, PagePoolOffsetAndEpochChunk *chunk)
ErrorCode	grab_one (PagePoolOffset *offset)
void	release_one (PagePoolOffset offset)
const LocalPageResolver &	get_resolver () const
PagePool::Stat	get_stat () const
uint64_t &	free_pool_head ()
uint64_t	free_pool_head () const
uint64_t	get_free_pool_capacity () const
uint64_t	get_free_pool_count () const
void	increase_free_pool_count (uint64_t op)
void	decrease_free_pool_count (uint64_t op)
void	assert_free_pool () const
	Not thread safe. More...
std::string	get_debug_pool_name () const
void	set_debug_pool_name (const std::string &name)
Public Member Functions inherited from foedus::DefaultInitializable
	DefaultInitializable ()
virtual	~DefaultInitializable ()
	DefaultInitializable (const DefaultInitializable &)=delete
DefaultInitializable &	operator= (const DefaultInitializable &)=delete
ErrorStack	initialize () override final
	Typical implementation of Initializable::initialize() that provides initialize-once semantics. More...
ErrorStack	uninitialize () override final
	Typical implementation of Initializable::uninitialize() that provides uninitialize-once semantics. More...
bool	is_initialized () const override final
	Returns whether the object has been already initialized or not. More...
Public Member Functions inherited from foedus::Initializable
virtual	~Initializable ()

Public Attributes
PagePoolControlBlock *	control_block_
void *	memory_
	The whole memory region of the pool. More...
uint64_t	memory_size_
	Byte size of this page pool. More...
bool	owns_
	Whether this engine owns this page pool. More...
bool	rigorous_page_boundary_check_
	Whether to use mprotect() for page boundaries to detect bogus memory accesses. More...
LocalPageResolver	resolver_
	An object to resolve an offset in this page pool (thus local) to an actual pointer and vice versa. More...
storage::Page *	pool_base_
	Just an auxiliary variable to the beginning of the pool. More...
uint64_t	pool_size_
	Just an auxiliary variable of the size of pool. More...
uint64_t	pages_for_free_pool_
	This many first pages are used for free page maintainance. More...
PagePoolOffset *	free_pool_
	We maintain free pages as a simple circular queue. More...
uint64_t	free_pool_capacity_
	Size of free_pool_. More...

Friends
std::ostream &	operator<< (std::ostream &o, const PagePoolPimpl &v)

Constructor & Destructor Documentation

foedus::memory::PagePoolPimpl::PagePoolPimpl

(

)

Definition at line 40 of file page_pool_pimpl.cpp.

  : control_block_(nullptr),
    memory_(nullptr),
    memory_size_(0),
    owns_(false),
    rigorous_page_boundary_check_(false) {}

Member Function Documentation

void foedus::memory::PagePoolPimpl::assert_free_pool ( ) const

inline

Not thread safe.

Use it after taking a lock.

Definition at line 108 of file page_pool_pimpl.hpp.

References ASSERT_ND, free_pool_, free_pool_capacity_, free_pool_head(), get_free_pool_count(), pages_for_free_pool_, and pool_size_.

Referenced by grab(), initialize_once(), release_impl(), and uninitialize_once().

                                               {
#ifndef NDEBUG
    const uint64_t free_count = get_free_pool_count();
    const uint64_t free_head = free_pool_head();
    for (uint64_t i = 0; i < free_count; ++i) {
      uint64_t index = free_head + i;
      while (index >= free_pool_capacity_) {
        index -= free_pool_capacity_;
      }
      PagePoolOffset* address = free_pool_ + index;
      ASSERT_ND(*address >= pages_for_free_pool_);
      ASSERT_ND(*address < pool_size_);
    }
#endif  // NDEBUG
  }

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::memory::PagePoolPimpl::attach	(	PagePoolControlBlock *	control_block,
		void *	memory,
		uint64_t	memory_size,
		bool	owns,
		bool	rigorous_page_boundary_check
	)

Definition at line 47 of file page_pool_pimpl.cpp.

References foedus::assorted::FixedString< MAXLEN, CHAR >::clear(), control_block_, foedus::memory::PagePoolControlBlock::debug_pool_name_, free_pool_, free_pool_capacity_, foedus::assorted::int_div_ceil(), foedus::storage::kPageSize, memory_, memory_size_, owns_, pages_for_free_pool_, pool_base_, pool_size_, resolver_, and rigorous_page_boundary_check_.

Referenced by foedus::memory::PagePool::attach().

                                     {
  control_block_ = control_block;
  if (owns) {
    control_block_->debug_pool_name_.clear();
  }
  memory_ = memory;
  memory_size_ = memory_size;
  owns_ = owns;
  rigorous_page_boundary_check_ = rigorous_page_boundary_check;
  pool_base_ = reinterpret_cast<storage::Page*>(memory_);
  pool_size_ = memory_size_ / storage::kPageSize;

  uint64_t pointers_total_size = pool_size_ * sizeof(PagePoolOffset);
  pages_for_free_pool_ = assorted::int_div_ceil(pointers_total_size, storage::kPageSize);

  free_pool_ = reinterpret_cast<PagePoolOffset*>(memory_);
  free_pool_capacity_ = pool_size_ - pages_for_free_pool_;
  if (rigorous_page_boundary_check) {
    free_pool_capacity_ = free_pool_capacity_ / 2U;
  }
  resolver_ = LocalPageResolver(pool_base_, pages_for_free_pool_, pool_size_);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::memory::PagePoolPimpl::decrease_free_pool_count ( uint64_t  op )

inline

Definition at line 103 of file page_pool_pimpl.hpp.

References ASSERT_ND, control_block_, and foedus::memory::PagePoolControlBlock::free_pool_count_.

Referenced by grab(), and grab_one().

                                                  {
    ASSERT_ND(control_block_->free_pool_count_ >= op);
    control_block_->free_pool_count_ -= op;
  }

Here is the caller graph for this function:

uint64_t& foedus::memory::PagePoolPimpl::free_pool_head ( )

inline

Definition at line 98 of file page_pool_pimpl.hpp.

References control_block_, and foedus::memory::PagePoolControlBlock::free_pool_head_.

Referenced by assert_free_pool(), grab(), grab_one(), foedus::memory::operator<<(), release_impl(), and release_one().

{ return control_block_->free_pool_head_;}

Here is the caller graph for this function:

uint64_t foedus::memory::PagePoolPimpl::free_pool_head ( ) const

inline

Definition at line 99 of file page_pool_pimpl.hpp.

References control_block_, and foedus::memory::PagePoolControlBlock::free_pool_head_.

{ return control_block_->free_pool_head_;}

std::string foedus::memory::PagePoolPimpl::get_debug_pool_name ( ) const

inline

Definition at line 124 of file page_pool_pimpl.hpp.

References control_block_, foedus::memory::PagePoolControlBlock::debug_pool_name_, foedus::assorted::FixedString< MAXLEN, CHAR >::empty(), and foedus::assorted::FixedString< MAXLEN, CHAR >::str().

Referenced by foedus::memory::PagePool::get_debug_pool_name(), grab(), grab_one(), initialize_once(), foedus::memory::operator<<(), release_impl(), release_one(), and uninitialize_once().

                                                {
    if (control_block_) {
      if (!control_block_->debug_pool_name_.empty()) {
        return control_block_->debug_pool_name_.str();
      } else {
        return "Unnamed";
      }
    } else {
      return "Not-attached";
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

uint64_t foedus::memory::PagePoolPimpl::get_free_pool_capacity ( ) const

inline

Definition at line 100 of file page_pool_pimpl.hpp.

References free_pool_capacity_.

Referenced by foedus::memory::PagePool::get_free_pool_capacity().

{ return free_pool_capacity_; }

Here is the caller graph for this function:

uint64_t foedus::memory::PagePoolPimpl::get_free_pool_count ( ) const

inline

Definition at line 101 of file page_pool_pimpl.hpp.

References control_block_, and foedus::memory::PagePoolControlBlock::free_pool_count_.

Referenced by assert_free_pool(), get_stat(), grab(), grab_one(), foedus::memory::operator<<(), release_impl(), release_one(), and uninitialize_once().

{ return control_block_->free_pool_count_;}

Here is the caller graph for this function:

const LocalPageResolver& foedus::memory::PagePoolPimpl::get_resolver ( ) const

inline

Definition at line 96 of file page_pool_pimpl.hpp.

References resolver_.

Referenced by foedus::memory::PagePool::get_resolver().

{ return resolver_; }

Here is the caller graph for this function:

PagePool::Stat foedus::memory::PagePoolPimpl::get_stat

(

)

const

Definition at line 359 of file page_pool_pimpl.cpp.

References foedus::memory::PagePool::Stat::allocated_pages_, get_free_pool_count(), pages_for_free_pool_, pool_size_, and foedus::memory::PagePool::Stat::total_pages_.

Referenced by foedus::memory::PagePool::get_stat().

                                           {
  PagePool::Stat ret;
  ret.total_pages_ = pool_size_ - pages_for_free_pool_;
  ret.allocated_pages_ = ret.total_pages_ - get_free_pool_count();
  return ret;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorCode foedus::memory::PagePoolPimpl::grab	(	uint32_t	desired_grab_count,
		PagePoolOffsetChunk *	chunk
	)

Definition at line 191 of file page_pool_pimpl.cpp.

References assert_free_pool(), ASSERT_ND, foedus::memory::PagePoolOffsetChunk::capacity(), control_block_, decrease_free_pool_count(), free_pool_, free_pool_capacity_, free_pool_head(), get_debug_pool_name(), get_free_pool_count(), foedus::kErrorCodeMemoryNoFreePages, foedus::kErrorCodeOk, foedus::memory::PagePoolControlBlock::lock_, foedus::memory::PagePoolOffsetChunk::push_back(), foedus::memory::PagePoolOffsetChunk::size(), and UNLIKELY.

Referenced by foedus::memory::PagePool::grab().

                                                                                     {
  ASSERT_ND(chunk->size() + desired_grab_count <= chunk->capacity());
  VLOG(0) << get_debug_pool_name() << " - Grabbing " << desired_grab_count << " pages."
    << " free_pool_count_=" << get_free_pool_count()
    << "->"
    << (get_free_pool_count() >= desired_grab_count
      ? get_free_pool_count() - desired_grab_count
      : 0);
  soc::SharedMutexScope guard(&control_block_->lock_);
  uint64_t free_count = get_free_pool_count();
  if (UNLIKELY(free_count == 0)) {
    LOG(WARNING) << get_debug_pool_name() << " - No more free pages left in the pool";
    return kErrorCodeMemoryNoFreePages;
  }

  // grab from the head
  assert_free_pool();
  uint64_t grab_count = std::min<uint64_t>(desired_grab_count, free_count);
  PagePoolOffset* head = free_pool_ + free_pool_head();
  if (free_pool_head() + grab_count > free_pool_capacity_) {
    // wrap around
    uint64_t wrap_count = free_pool_capacity_ - free_pool_head();
    chunk->push_back(head, head + wrap_count);
    free_pool_head() = 0;
    decrease_free_pool_count(wrap_count);
    grab_count -= wrap_count;
    head = free_pool_;
  }

  // no wrap around (or no more wrap around)
  assert_free_pool();
  ASSERT_ND(free_pool_head() + grab_count <= free_pool_capacity_);
  chunk->push_back(head, head + grab_count);
  free_pool_head() += grab_count;
  decrease_free_pool_count(grab_count);
  assert_free_pool();
  return kErrorCodeOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorCode foedus::memory::PagePoolPimpl::grab_one

(

PagePoolOffset *

offset

)

Definition at line 230 of file page_pool_pimpl.cpp.

References ASSERT_ND, control_block_, decrease_free_pool_count(), free_pool_, free_pool_capacity_, free_pool_head(), get_debug_pool_name(), get_free_pool_count(), foedus::kErrorCodeMemoryNoFreePages, foedus::kErrorCodeOk, foedus::memory::PagePoolControlBlock::lock_, and UNLIKELY.

Referenced by foedus::memory::PagePool::grab_one().

                                                        {
  VLOG(1) << get_debug_pool_name()
    << " - Grabbing just one page. free_pool_count_=" << get_free_pool_count() << "->"
    << (get_free_pool_count() - 1);
  *offset = 0;
  soc::SharedMutexScope guard(&control_block_->lock_);
  uint64_t free_count = get_free_pool_count();
  if (UNLIKELY(free_count == 0)) {
    LOG(WARNING) << get_debug_pool_name() << " - No more free pages left in the pool";
    return kErrorCodeMemoryNoFreePages;
  }

  // grab from the head
  PagePoolOffset* head = free_pool_ + free_pool_head();
  if (free_pool_head() == free_pool_capacity_) {
    // wrap around
    free_pool_head() = 0;
    head = free_pool_;
  }

  // no wrap around (or no more wrap around)
  ASSERT_ND(free_pool_head() + 1 <= free_pool_capacity_);
  *offset = *head;
  ++free_pool_head();
  decrease_free_pool_count(1);
  return kErrorCodeOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::memory::PagePoolPimpl::increase_free_pool_count ( uint64_t  op )

inline

Definition at line 102 of file page_pool_pimpl.hpp.

References control_block_, and foedus::memory::PagePoolControlBlock::free_pool_count_.

Referenced by release_impl(), and release_one().

{ control_block_->free_pool_count_ += op; }

Here is the caller graph for this function:

ErrorStack foedus::memory::PagePoolPimpl::initialize_once ( )

overridevirtual

Implements foedus::DefaultInitializable.

Definition at line 75 of file page_pool_pimpl.cpp.

References foedus::assorted::ProtectedBoundary::acquire_protect(), assert_free_pool(), ASSERT_ND, control_block_, foedus::debugging::StopWatch::elapsed_ms(), free_pool_, free_pool_capacity_, foedus::memory::PagePoolControlBlock::free_pool_count_, foedus::memory::PagePoolControlBlock::free_pool_head_, get_debug_pool_name(), foedus::memory::PagePoolControlBlock::initialize(), foedus::kErrorCodeOk, foedus::kRetOk, foedus::assorted::os_error(), owns_, pages_for_free_pool_, pool_base_, pool_size_, foedus::assorted::ProtectedBoundary::reset(), rigorous_page_boundary_check_, and foedus::debugging::StopWatch::stop().

                                          {
  if (owns_) {
    LOG(INFO) << get_debug_pool_name()
      << " - total_pages=" << pool_size_ << ", pages_for_free_pool_=" << pages_for_free_pool_
      << ", boundary_check=" << rigorous_page_boundary_check_;
    control_block_->initialize();
    LOG(INFO) << get_debug_pool_name() << " - Constructing circular free pool...";
    // all pages after pages_for_free_pool_-th page is in the free pool at first
    if (!rigorous_page_boundary_check_) {
      ASSERT_ND(free_pool_capacity_ == pool_size_ - pages_for_free_pool_);
      for (uint64_t i = 0; i < free_pool_capacity_; ++i) {
        free_pool_[i] = pages_for_free_pool_ + i;
      }
    } else {
      // Use even-numbered pages only
      ASSERT_ND(free_pool_capacity_ == (pool_size_ - pages_for_free_pool_) / 2U);
      for (uint64_t i = 0; i < free_pool_capacity_; ++i) {
        free_pool_[i] = pages_for_free_pool_ + i * 2U;
      }

      LOG(INFO) << get_debug_pool_name() << " - mprotect()-ing odd-numbered pages...";
      debugging::StopWatch watch;
      bool error_reported = false;
      for (uint64_t i = 0; i < free_pool_capacity_; ++i) {
        uint64_t index = pages_for_free_pool_ + i * 2U + 1U;
        static_assert(sizeof(assorted::ProtectedBoundary) == sizeof(storage::Page), "Hoosh!");
        assorted::ProtectedBoundary* boundary = reinterpret_cast<assorted::ProtectedBoundary*>(
          pool_base_ + index);
        boundary->reset(std::to_string(index));
        ErrorCode mprotect_ret = boundary->acquire_protect();
        if (mprotect_ret != kErrorCodeOk && !error_reported) {
          LOG(ERROR) << get_debug_pool_name() << " - mprotect() failed: " << assorted::os_error();
          error_reported = true;
        }
      }
      watch.stop();
      LOG(INFO) << get_debug_pool_name() << " - mprotect()-ed " << free_pool_capacity_
        << " pages in " << watch.elapsed_ms() << "ms";
    }

    /* to enable this, we need a version for rigorous_page_boundary_check_
    // [experimental] randomize the free pool pointers so that we can evenly utilize all
    // memory banks
    if (false) {  // disabled for now
      // this should be an option...
      LOG(INFO) << get_debug_pool_name() << " - Randomizing free pool...";
      struct Randomizer {
        PagePoolOffset  offset_;
        uint32_t        rank_;
        static bool compare(const Randomizer& left, const Randomizer& right) {
          return left.rank_ < right.rank_;
        }
      };
      Randomizer* randomizers = new Randomizer[free_pool_capacity_];
      assorted::UniformRandom rnd(123456L);
      for (uint64_t i = 0; i < free_pool_capacity_; ++i) {
        randomizers[i].offset_ = pages_for_free_pool_ + i;
        randomizers[i].rank_ = rnd.next_uint32();
      }
      std::sort(randomizers, randomizers + free_pool_capacity_, Randomizer::compare);
      for (uint64_t i = 0; i < free_pool_capacity_; ++i) {
        free_pool_[i] = randomizers[i].offset_;
      }
      delete[] randomizers;
      LOG(INFO) << get_debug_pool_name() << " - Randomized free pool.";
    }
    */

    control_block_->free_pool_head_ = 0;
    control_block_->free_pool_count_ = free_pool_capacity_;
    LOG(INFO) << get_debug_pool_name() << " - Constructed circular free pool.";
    assert_free_pool();
  }

  return kRetOk;
}

Here is the call graph for this function:

void foedus::memory::PagePoolPimpl::release	(	uint32_t	desired_release_count,
		PagePoolOffsetChunk *	chunk
	)

Definition at line 302 of file page_pool_pimpl.cpp.

Referenced by foedus::memory::PagePool::release().

                                                                                      {
  release_impl<PagePoolOffsetChunk>(desired_release_count, chunk);
}

Here is the caller graph for this function:

void foedus::memory::PagePoolPimpl::release	(	uint32_t	desired_release_count,
		PagePoolOffsetDynamicChunk *	chunk
	)

Definition at line 305 of file page_pool_pimpl.cpp.

                                                                                             {
  release_impl<PagePoolOffsetDynamicChunk>(desired_release_count, chunk);
}

void foedus::memory::PagePoolPimpl::release	(	uint32_t	desired_release_count,
		PagePoolOffsetAndEpochChunk *	chunk
	)

Definition at line 308 of file page_pool_pimpl.cpp.

                                                                                              {
  release_impl<PagePoolOffsetAndEpochChunk>(desired_release_count, chunk);
}

template<typename CHUNK >

void foedus::memory::PagePoolPimpl::release_impl	(	uint32_t	desired_release_count,
		CHUNK *	chunk
	)

Definition at line 259 of file page_pool_pimpl.cpp.

References assert_free_pool(), ASSERT_ND, control_block_, free_pool_, free_pool_capacity_, free_pool_head(), get_debug_pool_name(), get_free_pool_count(), increase_free_pool_count(), and foedus::memory::PagePoolControlBlock::lock_.

                                                                             {
  ASSERT_ND(chunk->size() >= desired_release_count);
  VLOG(0) << get_debug_pool_name() << " - Releasing " << desired_release_count << " pages."
    << " free_pool_count_=" << get_free_pool_count() << "->"
    << (get_free_pool_count() + desired_release_count);
  soc::SharedMutexScope guard(&control_block_->lock_);
  uint64_t free_count = get_free_pool_count();
  if (free_count + desired_release_count > free_pool_capacity_) {
    // this can't happen unless something is wrong! This is a critical issue from which
    // we can't recover because page pool is inconsistent!
    LOG(ERROR) << get_debug_pool_name()
      << " - PagePoolPimpl::release() More than full free-pool. inconsistent state!"
        << " free_count/capacity/release_count=" << free_count << "/" << free_pool_capacity_
          << "/" << desired_release_count;
    // TASK(Hideaki) Do a duplicate-check here to identify the problemetic pages.
    // crash here only in debug mode. otherwise just log the error
    // ASSERT_ND(free_count + desired_release_count <= free_pool_capacity_);
    // TASK(Hideaki) need to figure out why we hit this.
    return;
  }

  // append to the tail
  assert_free_pool();
  uint64_t release_count = std::min<uint64_t>(desired_release_count, chunk->size());
  uint64_t tail = free_pool_head() + free_count;
  if (tail >= free_pool_capacity_) {
    tail -= free_pool_capacity_;
  }
  if (tail + release_count > free_pool_capacity_) {
    // wrap around
    uint32_t wrap_count = free_pool_capacity_ - tail;
    chunk->move_to(free_pool_ + tail, wrap_count);
    increase_free_pool_count(wrap_count);
    release_count -= wrap_count;
    tail = 0;
  }

  // no wrap around (or no more wrap around)
  ASSERT_ND(tail + release_count <= free_pool_capacity_);
  chunk->move_to(free_pool_ + tail, release_count);
  increase_free_pool_count(release_count);
  assert_free_pool();
}

Here is the call graph for this function:

void foedus::memory::PagePoolPimpl::release_one

(

PagePoolOffset

offset

)

Definition at line 312 of file page_pool_pimpl.cpp.

References ASSERT_ND, COERCE_ERROR, control_block_, ERROR_STACK, free_pool_, free_pool_capacity_, free_pool_head(), get_debug_pool_name(), get_free_pool_count(), increase_free_pool_count(), foedus::DefaultInitializable::is_initialized(), foedus::kErrorCodeMemoryDuplicatePage, foedus::memory::PagePoolControlBlock::lock_, and owns_.

Referenced by foedus::memory::PagePool::release_one().

                                                     {
  ASSERT_ND(is_initialized() || !owns_);
  VLOG(1) << get_debug_pool_name() << " - Releasing just one page. free_pool_count_="
    << get_free_pool_count() << "->" << (get_free_pool_count() + 1);
  soc::SharedMutexScope guard(&control_block_->lock_);
  uint64_t free_count = get_free_pool_count();
  if (free_count >= free_pool_capacity_) {
    // this can't happen unless something is wrong! This is a critical issue from which
    // we can't recover because page pool is inconsistent!
    LOG(ERROR) << get_debug_pool_name()
      << " - PagePoolPimpl::release_one() More than full free-pool. inconsistent state!";
    COERCE_ERROR(ERROR_STACK(kErrorCodeMemoryDuplicatePage));
  }

  // append to the tail
  uint64_t tail = free_pool_head() + free_count;
  if (tail >= free_pool_capacity_) {
    tail -= free_pool_capacity_;
  }
  if (tail == free_pool_capacity_) {
    // wrap around
    tail = 0;
  }

  // no wrap around (or no more wrap around)
  ASSERT_ND(tail + 1 <= free_pool_capacity_);
  free_pool_[tail] = offset;
  increase_free_pool_count(1);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::memory::PagePoolPimpl::set_debug_pool_name ( const std::string &  name )

inline

Definition at line 135 of file page_pool_pimpl.hpp.

References foedus::assorted::FixedString< MAXLEN, CHAR >::assign(), control_block_, and foedus::memory::PagePoolControlBlock::debug_pool_name_.

Referenced by foedus::memory::PagePool::set_debug_pool_name().

                                                                 {
    if (control_block_) {
      control_block_->debug_pool_name_.assign(name);
    }
  }

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::memory::PagePoolPimpl::uninitialize_once ( )

overridevirtual

Implements foedus::DefaultInitializable.

Definition at line 152 of file page_pool_pimpl.cpp.

References foedus::assorted::ProtectedBoundary::assert_boundary(), assert_free_pool(), ASSERT_ND, control_block_, foedus::debugging::StopWatch::elapsed_ms(), free_pool_capacity_, foedus::assorted::ProtectedBoundary::get_boundary_name(), get_debug_pool_name(), get_free_pool_count(), foedus::kErrorCodeOk, foedus::kRetOk, foedus::assorted::os_error(), owns_, pages_for_free_pool_, pool_base_, foedus::assorted::ProtectedBoundary::release_protect(), rigorous_page_boundary_check_, foedus::debugging::StopWatch::stop(), and foedus::memory::PagePoolControlBlock::uninitialize().

                                            {
  if (owns_) {
    assert_free_pool();
    if (rigorous_page_boundary_check_) {
      LOG(INFO) << get_debug_pool_name() << " - releasing mprotect() odd-numbered pages...";
      debugging::StopWatch watch;
      bool error_reported = false;
      for (uint64_t i = 0; i < free_pool_capacity_; ++i) {
        uint64_t index = pages_for_free_pool_ + i * 2U + 1U;
        assorted::ProtectedBoundary* boundary = reinterpret_cast<assorted::ProtectedBoundary*>(
          pool_base_ + index);
        ErrorCode mprotect_ret = boundary->release_protect();
        if (mprotect_ret != kErrorCodeOk && !error_reported) {
          LOG(ERROR) << get_debug_pool_name() << " - mprotect() failed: " << assorted::os_error();
          error_reported = true;
        }
        boundary->assert_boundary();
        ASSERT_ND(boundary->get_boundary_name() == std::to_string(index));
      }
      watch.stop();
      LOG(INFO) << get_debug_pool_name() << " - released mprotect() " << free_pool_capacity_
        << " pages in " << watch.elapsed_ms() << "ms";
    }

    uint64_t free_count = get_free_pool_count();
    if (free_count != free_pool_capacity_) {
      // This is not a memory leak as we anyway releases everything, but it's a smell of bug.
      LOG(WARNING) << get_debug_pool_name()
        << " - Page Pool has not received back all free pages by its uninitialization!!"
        << " count=" << free_count << ", capacity=" << free_pool_capacity_;
    } else {
      LOG(INFO) << get_debug_pool_name()
        << " - Page Pool has received back all free pages. No suspicious behavior.";
    }
    control_block_->uninitialize();
  }
  return kRetOk;
}

Here is the call graph for this function:

Friends And Related Function Documentation

std::ostream& operator<< ( std::ostream &  o, const PagePoolPimpl &  v  )

friend

Definition at line 343 of file page_pool_pimpl.cpp.

                                                              {
  o << "<PagePool>"
    << "<name_>" << v.get_debug_pool_name() << "</name_>"
    << "<memory_>" << v.memory_ << "</memory_>"
    << "<memory_size>" << v.memory_size_ << "</memory_size>"
    << "<owns_>" << v.owns_ << "</owns_>"
    << "<rigorous_page_boundary_check_>"
      << v.rigorous_page_boundary_check_ << "</rigorous_page_boundary_check_>"
    << "<pages_for_free_pool_>" << v.pages_for_free_pool_ << "</pages_for_free_pool_>"
    << "<free_pool_capacity_>" << v.free_pool_capacity_ << "</free_pool_capacity_>"
    << "<free_pool_head_>" << v.free_pool_head() << "</free_pool_head_>"
    << "<free_pool_count_>" << v.get_free_pool_count() << "</free_pool_count_>"
    << "</PagePool>";
  return o;
}

Member Data Documentation

PagePoolControlBlock* foedus::memory::PagePoolPimpl::control_block_

Definition at line 143 of file page_pool_pimpl.hpp.

Referenced by attach(), decrease_free_pool_count(), free_pool_head(), get_debug_pool_name(), get_free_pool_count(), grab(), grab_one(), increase_free_pool_count(), initialize_once(), release_impl(), release_one(), set_debug_pool_name(), and uninitialize_once().

PagePoolOffset* foedus::memory::PagePoolPimpl::free_pool_

We maintain free pages as a simple circular queue.

We append new/released pages to tail while we eat from head.

Definition at line 181 of file page_pool_pimpl.hpp.

Referenced by assert_free_pool(), attach(), grab(), grab_one(), initialize_once(), release_impl(), and release_one().

uint64_t foedus::memory::PagePoolPimpl::free_pool_capacity_

Size of free_pool_.

Immutable once initialized.

Definition at line 183 of file page_pool_pimpl.hpp.

Referenced by assert_free_pool(), attach(), get_free_pool_capacity(), foedus::memory::PagePool::get_recommended_pages_per_grab(), grab(), grab_one(), initialize_once(), foedus::memory::operator<<(), release_impl(), release_one(), and uninitialize_once().

void* foedus::memory::PagePoolPimpl::memory_

The whole memory region of the pool.

Definition at line 146 of file page_pool_pimpl.hpp.

Referenced by attach(), and foedus::memory::operator<<().

uint64_t foedus::memory::PagePoolPimpl::memory_size_

Byte size of this page pool.

Definition at line 149 of file page_pool_pimpl.hpp.

Referenced by attach(), foedus::memory::PagePool::get_memory_size(), and foedus::memory::operator<<().

bool foedus::memory::PagePoolPimpl::owns_

Whether this engine owns this page pool.

If not, initialize just attaches

Definition at line 152 of file page_pool_pimpl.hpp.

Referenced by attach(), initialize_once(), foedus::memory::operator<<(), release_one(), and uninitialize_once().

uint64_t foedus::memory::PagePoolPimpl::pages_for_free_pool_

This many first pages are used for free page maintainance.

This also means that "Page-0" never appears in our engine, thus we can use offset=0 as null. In other words, all offsets grabbed/released should be equal or larger than this value. Immutable once initialized.

Definition at line 175 of file page_pool_pimpl.hpp.

Referenced by assert_free_pool(), attach(), get_stat(), initialize_once(), foedus::memory::operator<<(), and uninitialize_once().

storage::Page* foedus::memory::PagePoolPimpl::pool_base_

Just an auxiliary variable to the beginning of the pool.

Same as memory_.get_block().

Definition at line 164 of file page_pool_pimpl.hpp.

Referenced by attach(), foedus::memory::PagePool::get_base(), initialize_once(), and uninitialize_once().

uint64_t foedus::memory::PagePoolPimpl::pool_size_

Just an auxiliary variable of the size of pool.

Same as memory_byte_size/kPageSize.

Definition at line 167 of file page_pool_pimpl.hpp.

Referenced by assert_free_pool(), attach(), get_stat(), and initialize_once().

LocalPageResolver foedus::memory::PagePoolPimpl::resolver_

An object to resolve an offset in this page pool (thus local) to an actual pointer and vice versa.

Definition at line 161 of file page_pool_pimpl.hpp.

Referenced by attach(), and get_resolver().

bool foedus::memory::PagePoolPimpl::rigorous_page_boundary_check_

Whether to use mprotect() for page boundaries to detect bogus memory accesses.

Similar to rigorous_memory_boundary_check_, but this one is about page pools. When true, we waste a half of volatile/snapshot page pool, putting mprotect in odd-numbered page indexes. The threads receive only the even-numbered indexes, so if any read/write overruns a page, a signal is fired. This makes the execution quite slower and wastes half of memory, so use it just for debugging. This flag also disables hugepages used for snapshot page pool for mprotect to work, so makes it really SLOW. Default is false.

Attention

To use this feature with large page pool sizes, you must increase max_map_count. sudo sysctl -w vm.max_map_count=2147483647 Otherwise you will get out-of-memory crashes. Linux's default value is only 65530.

Note

When FOEDUS is running on a valgrind process, you shouldn't turn this on because valgrind uses a hard-coded limit VG_N_SEGMENTS rather than vm.max_map_count. However, if you are sure the number of mprotect-ed pages are less than 64k, you can.

See also

foedus::assorted::ProtectedBoundary

Definition at line 155 of file page_pool_pimpl.hpp.

Referenced by attach(), initialize_once(), foedus::memory::operator<<(), and uninitialize_once().

---

The documentation for this class was generated from the following files:

• /home/shino/foedus_code/foedus-core/include/foedus/memory/page_pool_pimpl.hpp
• /home/shino/foedus_code/foedus-core/src/foedus/memory/page_pool_pimpl.cpp