foedus::memory Namespace Reference

Memory Manager, which controls memory allocations, deallocations, and sharing. More...

Detailed Description

Memory Manager, which controls memory allocations, deallocations, and sharing.

This package contains classes that control memory allocations, deallocations, and sharing.

Classes
class	AlignedMemory
	Represents one memory block aligned to actual OS/hardware pages. More...
struct	AlignedMemorySlice
	A slice of foedus::memory::AlignedMemory. More...
struct	AutoVolatilePageReleaseScope
	Automatically invokes a page offset acquired for volatile page. More...
class	DivvyupPageGrabBatch
	A helper class to grab a bunch of pages from multiple nodes in arbitrary fashion. More...
class	EngineMemory
	Repository of all memories dynamically acquired and shared within one database engine. More...
struct	GlobalVolatilePageResolver
	Resolves an offset in a volatile page pool to an actual pointer and vice versa. More...
struct	LocalPageResolver
	Resolves an offset in local (same NUMA node) page pool to a pointer and vice versa. More...
struct	MemoryOptions
	Set of options for memory manager. More...
class	NumaCoreMemory
	Repository of memories dynamically acquired within one CPU core (thread). More...
class	NumaNodeMemory
	Repository of memories dynamically acquired and shared within one NUMA node (socket). More...
class	NumaNodeMemoryRef
	A view of NumaNodeMemory for other SOCs and master engine. More...
class	PagePool
	Page pool for volatile read/write store (VolatilePage) and the read-only bufferpool (SnapshotPage). More...
struct	PagePoolControlBlock
	Shared data in PagePoolPimpl. More...
class	PagePoolOffsetAndEpochChunk
	Used to store an epoch value with each entry in PagePoolOffsetChunk. More...
class	PagePoolOffsetChunk
	To reduce the overhead of grabbing/releasing pages from pool, we pack this many pointers for each grab/release. More...
class	PagePoolOffsetDynamicChunk
	Used to point to an already existing array. More...
class	PagePoolPimpl
	Pimpl object of PagePool. More...
class	PageReleaseBatch
	A helper class to return a bunch of pages to individual nodes. More...
class	RoundRobinPageGrabBatch
	A helper class to grab a bunch of pages from multiple nodes in round-robin fashion per chunk. More...
struct	ScopedNumaPreferred
	Automatically sets and resets numa_set_preferred(). More...
class	SharedMemory
	Represents memory shared between processes. More...

Typedefs
typedef uint32_t	PagePoolOffset
	Offset in PagePool that compactly represents the page address (unlike 8 bytes pointer). More...

Functions
bool	is_1gb_hugepage_enabled ()
	Returns if 1GB hugepages were enabled. More...
void	_dummy_static_size_check__COUNTER__ ()
char *	alloc_mmap (uint64_t size, uint64_t alignment)
void *	alloc_mmap_1gb_pages (uint64_t size)
std::ostream &	operator<< (std::ostream &o, const AlignedMemory &v)
std::ostream &	operator<< (std::ostream &o, const AlignedMemorySlice &v)
int64_t	get_numa_node_size (int node)
std::ostream &	operator<< (std::ostream &o, const PagePool &v)
std::ostream &	operator<< (std::ostream &o, const PagePoolPimpl &v)
std::ostream &	operator<< (std::ostream &o, const SharedMemory &v)

Variables
const uint64_t	kHugepageSize = 1 << 21
	So far 2MB is the only page size available via Transparent Huge Page (THP). More...

Function Documentation

void foedus::memory::_dummy_static_size_check__COUNTER__ ( )

inline

Definition at line 417 of file page_pool.hpp.

char* foedus::memory::alloc_mmap	(	uint64_t	size,
		uint64_t	alignment
	)

Definition at line 64 of file aligned_memory.cpp.

References is_1gb_hugepage_enabled(), MAP_HUGE_1GB, MAP_HUGE_2MB, and foedus::assorted::os_error().

Referenced by foedus::memory::AlignedMemory::alloc(), and alloc_mmap_1gb_pages().

                                                    {
  // std::lock_guard<std::mutex> guard(mmap_allocate_mutex);
  // we don't use MAP_POPULATE because it will block here and also serialize hugepage allocation!
  // even if we run mmap in parallel, linux serializes the looooong population in all numa nodes.
  // lame. we will memset right after this.
  int pagesize;
  if (alignment >= (1ULL << 30)) {
    if (is_1gb_hugepage_enabled()) {
      pagesize = MAP_HUGE_1GB | MAP_HUGETLB;
    } else {
      pagesize = MAP_HUGE_2MB | MAP_HUGETLB;
    }
  } else if (alignment >= (1ULL << 21)) {
    pagesize = MAP_HUGE_2MB | MAP_HUGETLB;
  } else {
    pagesize = 0;
  }
  bool running_on_valgrind = RUNNING_ON_VALGRIND;
  if (running_on_valgrind) {
    // if this is running under valgrind, we have to avoid using hugepages due to a bug in valgrind.
    // When we are running on valgrind, we don't care performance anyway. So shouldn't matter.
    pagesize = 0;
  }
  char* ret = reinterpret_cast<char*>(::mmap(
    nullptr,
    size,
    PROT_READ | PROT_WRITE,
    MAP_ANONYMOUS | MAP_PRIVATE | pagesize,  // | MAP_NORESERVE
    -1,
    0));
  // Note: We previously used MAP_NORESERVE to explicitly say we don't want swapping,
  // but mmap with this flag causes SIGSEGV when there aren't enough hugepages.
  // In that case mmap doesn't return -1 because it just checks if VA space is mappable.
  // We still don't need swapping, but it won't hurt. sorta. Debuggability matters more.

  // when mmap() fails, it returns -1 (MAP_FAILED)
  if (ret == nullptr || ret == MAP_FAILED) {
    LOG(FATAL) << "mmap() failed. size=" << size << ", error=" << assorted::os_error()
      << ". This error usually means you don't have enough hugepages allocated."
      << " eg) sudo sh -c 'echo 196608 > /proc/sys/vm/nr_hugepages'";
  }
  return ret;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void* foedus::memory::alloc_mmap_1gb_pages

(

uint64_t

size

)

Definition at line 108 of file aligned_memory.cpp.

References alloc_mmap(), and ASSERT_ND.

Referenced by foedus::memory::AlignedMemory::alloc().

                                          {
  ASSERT_ND(size % (1ULL << 30) == 0);
  return alloc_mmap(size, 1ULL << 30);
}

Here is the call graph for this function:

Here is the caller graph for this function:

int64_t foedus::memory::get_numa_node_size

(

int

node

)

Definition at line 49 of file numa_node_memory.cpp.

References numa_available(), and numa_node_size().

Referenced by foedus::memory::NumaNodeMemory::initialize_once(), and foedus::memory::NumaNodeMemory::uninitialize_once().

                                     {
  if (::numa_available() < 0) {
    return 0;
  } else {
    return ::numa_node_size(node, nullptr);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool foedus::memory::is_1gb_hugepage_enabled

(

)

Returns if 1GB hugepages were enabled.

Definition at line 293 of file aligned_memory.cpp.

Referenced by alloc_mmap().

                               {
  // /proc/meminfo should have "Hugepagesize:    1048576 kB"
  // Unfortunately, sysinfo() doesn't provide this information. So, just read the whole file.
  // Alternatively, we can use gethugepagesizes(3) in libhugetlbs, but I don't want to add
  // a dependency just for that...
  std::ifstream file("/proc/meminfo");
  if (!file.is_open()) {
    return false;
  }

  std::string line;
  while (std::getline(file, line)) {
    if (line.find("Hugepagesize:") != std::string::npos) {
      break;
    }
  }
  file.close();
  if (line.find("1048576 kB") != std::string::npos) {
    return true;
  }
  return false;
}

Here is the caller graph for this function:

std::ostream& foedus::memory::operator<<	(	std::ostream &	o,
		const PagePool &	v
	)

Definition at line 148 of file page_pool.cpp.

                                                         {
  o << v.pimpl_;
  return o;
}

std::ostream& foedus::memory::operator<<	(	std::ostream &	o,
		const SharedMemory &	v
	)

Definition at line 245 of file shared_memory.cpp.

References foedus::memory::SharedMemory::get_block(), foedus::memory::SharedMemory::get_meta_path(), foedus::memory::SharedMemory::get_numa_node(), foedus::memory::SharedMemory::get_owner_pid(), foedus::memory::SharedMemory::get_shmid(), foedus::memory::SharedMemory::get_shmkey(), foedus::memory::SharedMemory::get_size(), and foedus::memory::SharedMemory::is_owned().

                                                             {
  o << "<SharedMemory>";
  o << "<meta_path>" << v.get_meta_path() << "</meta_path>";
  o << "<size>" << v.get_size() << "</size>";
  o << "<owned>" << v.is_owned() << "</owned>";
  o << "<owner_pid>" << v.get_owner_pid() << "</owner_pid>";
  o << "<numa_node>" << v.get_numa_node() << "</numa_node>";
  o << "<shmid>" << v.get_shmid() << "</shmid>";
  o << "<shmkey>" << v.get_shmkey() << "</shmkey>";
  o << "<address>" << reinterpret_cast<uintptr_t>(v.get_block()) << "</address>";
  o << "</SharedMemory>";
  return o;
}

Here is the call graph for this function:

std::ostream& foedus::memory::operator<<	(	std::ostream &	o,
		const AlignedMemory &	v
	)

Definition at line 253 of file aligned_memory.cpp.

References foedus::memory::AlignedMemory::get_alignment(), foedus::memory::AlignedMemory::get_alloc_type(), foedus::memory::AlignedMemory::get_block(), foedus::memory::AlignedMemory::get_numa_node(), foedus::memory::AlignedMemory::get_size(), foedus::memory::AlignedMemory::is_null(), foedus::memory::AlignedMemory::kNumaAllocInterleaved, foedus::memory::AlignedMemory::kNumaAllocOnnode, foedus::memory::AlignedMemory::kNumaMmapOneGbPages, and foedus::memory::AlignedMemory::kPosixMemalign.

                                                              {
  o << "<AlignedMemory>";
  o << "<is_null>" << v.is_null() << "</is_null>";
  o << "<size>" << v.get_size() << "</size>";
  o << "<alignment>" << v.get_alignment() << "</alignment>";
  o << "<alloc_type>" << v.get_alloc_type() << " (";
  switch (v.get_alloc_type()) {
    case AlignedMemory::kPosixMemalign:
      o << "kPosixMemalign";
      break;
    case AlignedMemory::kNumaAllocInterleaved:
      o << "kNumaAllocInterleaved";
      break;
    case AlignedMemory::kNumaAllocOnnode:
      o << "kNumaAllocOnnode";
      break;
    case AlignedMemory::kNumaMmapOneGbPages:
      o << "kNumaMmapOneGbPages";
      break;
    default:
      o << "Unknown";
  }
  o << ")</alloc_type>";
  o << "<numa_node>" << static_cast<int>(v.get_numa_node()) << "</numa_node>";
  o << "<address>" << v.get_block() << "</address>";
  o << "</AlignedMemory>";
  return o;
}

Here is the call graph for this function:

std::ostream& foedus::memory::operator<<	(	std::ostream &	o,
		const AlignedMemorySlice &	v
	)

Definition at line 282 of file aligned_memory.cpp.

References foedus::memory::AlignedMemorySlice::count_, foedus::memory::AlignedMemorySlice::memory_, and foedus::memory::AlignedMemorySlice::offset_.

                                                                   {
  o << "<AlignedMemorySlice>";
  o << "<offset>" << v.offset_ << "</offset>";
  o << "<count>" << v.count_ << "</count>";
  if (v.memory_) {
    o << *v.memory_;
  }
  o << "</AlignedMemorySlice>";
  return o;
}

std::ostream& foedus::memory::operator<<	(	std::ostream &	o,
		const PagePoolPimpl &	v
	)

Definition at line 343 of file page_pool_pimpl.cpp.

References foedus::memory::PagePoolPimpl::free_pool_capacity_, foedus::memory::PagePoolPimpl::free_pool_head(), foedus::memory::PagePoolPimpl::get_debug_pool_name(), foedus::memory::PagePoolPimpl::get_free_pool_count(), foedus::memory::PagePoolPimpl::memory_, foedus::memory::PagePoolPimpl::memory_size_, foedus::memory::PagePoolPimpl::owns_, foedus::memory::PagePoolPimpl::pages_for_free_pool_, and foedus::memory::PagePoolPimpl::rigorous_page_boundary_check_.

                                                              {
  o << "<PagePool>"
    << "<name_>" << v.get_debug_pool_name() << "</name_>"
    << "<memory_>" << v.memory_ << "</memory_>"
    << "<memory_size>" << v.memory_size_ << "</memory_size>"
    << "<owns_>" << v.owns_ << "</owns_>"
    << "<rigorous_page_boundary_check_>"
      << v.rigorous_page_boundary_check_ << "</rigorous_page_boundary_check_>"
    << "<pages_for_free_pool_>" << v.pages_for_free_pool_ << "</pages_for_free_pool_>"
    << "<free_pool_capacity_>" << v.free_pool_capacity_ << "</free_pool_capacity_>"
    << "<free_pool_head_>" << v.free_pool_head() << "</free_pool_head_>"
    << "<free_pool_count_>" << v.get_free_pool_count() << "</free_pool_count_>"
    << "</PagePool>";
  return o;
}

Here is the call graph for this function: