libfoedus-core
FOEDUS Core Library
foedus::memory::NumaNodeMemory Class Referencefinal

Repository of memories dynamically acquired and shared within one NUMA node (socket). More...

Detailed Description

Repository of memories dynamically acquired and shared within one NUMA node (socket).

One NumaNodeMemory corresponds to one foedus::thread::ThreadGroup. All threads in the thread group belong to the NUMA node, thus sharing memories between them must be efficient. So, all memories here are allocated/freed via numa_alloc_interleaved(), numa_alloc_onnode(), and numa_free() (except the user specifies to not use them).

Definition at line 48 of file numa_node_memory.hpp.

#include <numa_node_memory.hpp>

Inheritance diagram for foedus::memory::NumaNodeMemory:
Collaboration diagram for foedus::memory::NumaNodeMemory:

Public Member Functions

 NumaNodeMemory ()=delete
 
 NumaNodeMemory (Engine *engine, foedus::thread::ThreadGroupId numa_node)
 
ErrorStack initialize_once () override
 
ErrorStack uninitialize_once () override
 
foedus::thread::ThreadGroupId get_numa_node () const
 
PagePoolget_volatile_pool ()
 
PagePoolget_snapshot_pool ()
 
cache::CacheHashtableget_snapshot_cache_table ()
 
foedus::thread::ThreadLocalOrdinal get_core_memory_count () const
 
std::vector< NumaCoreMemory * > & get_core_memories ()
 
NumaCoreMemoryget_core_memory (foedus::thread::ThreadId id) const
 
NumaCoreMemoryget_core_memory (foedus::thread::ThreadLocalOrdinal ordinal) const
 
ErrorStack allocate_numa_memory_general (uint64_t size, uint64_t alignment, AlignedMemory *out) const
 Allocate a memory of the given size on this NUMA node. More...
 
ErrorStack allocate_numa_memory (uint64_t size, AlignedMemory *out) const
 
ErrorStack allocate_huge_numa_memory (uint64_t size, AlignedMemory *out) const
 
PagePoolOffsetChunkget_volatile_offset_chunk_memory_piece (foedus::thread::ThreadLocalOrdinal core_ordinal)
 
PagePoolOffsetChunkget_snapshot_offset_chunk_memory_piece (foedus::thread::ThreadLocalOrdinal core_ordinal)
 
AlignedMemorySlice get_log_buffer_memory_piece (log::LoggerId logger)
 
std::string dump_free_memory_stat () const
 Report rough statistics of free memory. More...
 
- Public Member Functions inherited from foedus::DefaultInitializable
 DefaultInitializable ()
 
virtual ~DefaultInitializable ()
 
 DefaultInitializable (const DefaultInitializable &)=delete
 
DefaultInitializableoperator= (const DefaultInitializable &)=delete
 
ErrorStack initialize () override final
 Typical implementation of Initializable::initialize() that provides initialize-once semantics. More...
 
ErrorStack uninitialize () override final
 Typical implementation of Initializable::uninitialize() that provides uninitialize-once semantics. More...
 
bool is_initialized () const override final
 Returns whether the object has been already initialized or not. More...
 
- Public Member Functions inherited from foedus::Initializable
virtual ~Initializable ()
 

Constructor & Destructor Documentation

foedus::memory::NumaNodeMemory::NumaNodeMemory ( )
delete
foedus::memory::NumaNodeMemory::NumaNodeMemory ( Engine engine,
foedus::thread::ThreadGroupId  numa_node 
)

Definition at line 41 of file numa_node_memory.cpp.

42  : engine_(engine),
43  numa_node_(numa_node),
44  cores_(engine_->get_options().thread_.thread_count_per_group_),
45  loggers_(engine_->get_options().log_.loggers_per_node_),
46  snapshot_cache_table_(nullptr) {
47 }
const EngineOptions & get_options() const
Definition: engine.cpp:39
ThreadLocalOrdinal thread_count_per_group_
Number of Thread in each ThreadGroup.
thread::ThreadOptions thread_
uint16_t loggers_per_node_
Number of loggers per NUMA node.
Definition: log_options.hpp:80

Member Function Documentation

ErrorStack foedus::memory::NumaNodeMemory::allocate_huge_numa_memory ( uint64_t  size,
AlignedMemory out 
) const
inline

Definition at line 88 of file numa_node_memory.hpp.

References allocate_numa_memory_general(), and foedus::memory::kHugepageSize.

Referenced by initialize_once().

88  {
90  }
const uint64_t kHugepageSize
So far 2MB is the only page size available via Transparent Huge Page (THP).
Definition: memory_id.hpp:50
ErrorStack allocate_numa_memory_general(uint64_t size, uint64_t alignment, AlignedMemory *out) const
Allocate a memory of the given size on this NUMA node.

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::memory::NumaNodeMemory::allocate_numa_memory ( uint64_t  size,
AlignedMemory out 
) const
inline

Definition at line 85 of file numa_node_memory.hpp.

References allocate_numa_memory_general().

Referenced by initialize_once(), foedus::memory::NumaCoreMemory::initialize_once(), and foedus::log::Logger::initialize_once().

85  {
86  return allocate_numa_memory_general(size, 1 << 12, out);
87  }
ErrorStack allocate_numa_memory_general(uint64_t size, uint64_t alignment, AlignedMemory *out) const
Allocate a memory of the given size on this NUMA node.

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::memory::NumaNodeMemory::allocate_numa_memory_general ( uint64_t  size,
uint64_t  alignment,
AlignedMemory out 
) const

Allocate a memory of the given size on this NUMA node.

Parameters
[in]sizebyte size of the memory to acquire
[in]alignmentalignment size
[out]outallocated memory is moved to object
Returns
Expect OUTOFMEMORY error.

Definition at line 201 of file numa_node_memory.cpp.

References foedus::memory::AlignedMemory::alloc(), ASSERT_ND, ERROR_STACK, foedus::Engine::get_options(), foedus::memory::AlignedMemory::is_null(), foedus::kErrorCodeOutofmemory, foedus::memory::kHugepageSize, foedus::memory::AlignedMemory::kNumaAllocOnnode, foedus::memory::AlignedMemory::kNumaMmapOneGbPages, foedus::kRetOk, foedus::EngineOptions::memory_, and foedus::memory::MemoryOptions::use_mmap_hugepages_.

Referenced by allocate_huge_numa_memory(), and allocate_numa_memory().

204  {
205  ASSERT_ND(out);
206  if (engine_->get_options().memory_.use_mmap_hugepages_ &&
207  alignment >= kHugepageSize
208  && size >= (1ULL << 30) * 8 / 10) {
209  LOG(INFO) << "This is a big memory allocation. Let's use the mmap hugepage (1GB pages)";
210  out->alloc(size, 1ULL << 30, AlignedMemory::kNumaMmapOneGbPages, numa_node_);
211  } else {
212  out->alloc(size, alignment, AlignedMemory::kNumaAllocOnnode, numa_node_);
213  }
214  if (out->is_null()) {
216  }
217  return kRetOk;
218 }
0x0001 : "GENERAL: Out of memory" .
Definition: error_code.hpp:105
numa_alloc_onnode() and numa_free().
#define ERROR_STACK(e)
Instantiates ErrorStack with the given foedus::error_code, creating an error stack with the current f...
const EngineOptions & get_options() const
Definition: engine.cpp:39
const uint64_t kHugepageSize
So far 2MB is the only page size available via Transparent Huge Page (THP).
Definition: memory_id.hpp:50
memory::MemoryOptions memory_
bool use_mmap_hugepages_
Whether to use non-transparent hugepages for big memories (1GB huge pages).
const ErrorStack kRetOk
Normal return value for no-error case.
#define ASSERT_ND(x)
A warning-free wrapper macro of assert() that has no performance effect in release mode even when 'x'...
Definition: assert_nd.hpp:72

Here is the call graph for this function:

Here is the caller graph for this function:

std::string foedus::memory::NumaNodeMemory::dump_free_memory_stat ( ) const

Report rough statistics of free memory.

Definition at line 220 of file numa_node_memory.cpp.

References foedus::memory::PagePool::Stat::allocated_pages_, foedus::memory::PagePool::get_stat(), and foedus::memory::PagePool::Stat::total_pages_.

220  {
221  std::stringstream ret;
222  PagePool::Stat volatile_stat = volatile_pool_.get_stat();
223  ret << " Volatile-Pool: " << volatile_stat.allocated_pages_ << " allocated pages, "
224  << volatile_stat.total_pages_ << " total pages, "
225  << (volatile_stat.total_pages_ - volatile_stat.allocated_pages_) << " free pages"
226  << std::endl;
227  PagePool::Stat snapshot_stat = snapshot_pool_.get_stat();
228  ret << " Snapshot-Pool: " << snapshot_stat.allocated_pages_ << " allocated pages, "
229  << snapshot_stat.total_pages_ << " total pages, "
230  << (snapshot_stat.total_pages_ - snapshot_stat.allocated_pages_) << " free pages"
231  << std::endl;
232  return ret.str();
233 }
Stat get_stat() const
Definition: page_pool.cpp:118

Here is the call graph for this function:

std::vector<NumaCoreMemory*>& foedus::memory::NumaNodeMemory::get_core_memories ( )
inline

Definition at line 66 of file numa_node_memory.hpp.

66 { return core_memories_; }
NumaCoreMemory* foedus::memory::NumaNodeMemory::get_core_memory ( foedus::thread::ThreadId  id) const
inline

Definition at line 67 of file numa_node_memory.hpp.

References foedus::thread::decompose_numa_local_ordinal().

Referenced by foedus::thread::ThreadPimpl::initialize_once(), and foedus::log::ThreadLogBuffer::initialize_once().

67  {
68  return core_memories_[foedus::thread::decompose_numa_local_ordinal(id)];
69  }
ThreadLocalOrdinal decompose_numa_local_ordinal(ThreadId global_id)
Extracts local ordinal from the given globally unique ID of Thread (core).
Definition: thread_id.hpp:139

Here is the call graph for this function:

Here is the caller graph for this function:

NumaCoreMemory* foedus::memory::NumaNodeMemory::get_core_memory ( foedus::thread::ThreadLocalOrdinal  ordinal) const
inline

Definition at line 70 of file numa_node_memory.hpp.

70  {
71  return core_memories_[ordinal];
72  }
foedus::thread::ThreadLocalOrdinal foedus::memory::NumaNodeMemory::get_core_memory_count ( ) const
inline

Definition at line 62 of file numa_node_memory.hpp.

References ASSERT_ND, and foedus::thread::kMaxThreadLocalOrdinal.

62  {
63  ASSERT_ND(core_memories_.size() <= foedus::thread::kMaxThreadLocalOrdinal);
64  return static_cast<foedus::thread::ThreadLocalOrdinal>(core_memories_.size());
65  }
const ThreadLocalOrdinal kMaxThreadLocalOrdinal
Maximum possible value of ThreadLocalOrdinal.
Definition: thread_id.hpp:69
uint8_t ThreadLocalOrdinal
Typedef for a local ID of Thread (core), which is NOT unique across NUMA nodes.
Definition: thread_id.hpp:58
#define ASSERT_ND(x)
A warning-free wrapper macro of assert() that has no performance effect in release mode even when 'x'...
Definition: assert_nd.hpp:72
AlignedMemorySlice foedus::memory::NumaNodeMemory::get_log_buffer_memory_piece ( log::LoggerId  logger)
inline

Definition at line 100 of file numa_node_memory.hpp.

Referenced by foedus::memory::NumaCoreMemory::initialize_once().

100  {
101  return log_buffer_memory_pieces_[logger];
102  }

Here is the caller graph for this function:

foedus::thread::ThreadGroupId foedus::memory::NumaNodeMemory::get_numa_node ( ) const
inline

Definition at line 55 of file numa_node_memory.hpp.

Referenced by foedus::memory::NumaCoreMemory::NumaCoreMemory().

55 { return numa_node_; }

Here is the caller graph for this function:

cache::CacheHashtable* foedus::memory::NumaNodeMemory::get_snapshot_cache_table ( )
inline

Definition at line 59 of file numa_node_memory.hpp.

Referenced by foedus::cache::CacheManagerPimpl::initialize_once(), and foedus::thread::ThreadPimpl::initialize_once().

59 { return snapshot_cache_table_; }

Here is the caller graph for this function:

PagePoolOffsetChunk* foedus::memory::NumaNodeMemory::get_snapshot_offset_chunk_memory_piece ( foedus::thread::ThreadLocalOrdinal  core_ordinal)
inline

Definition at line 96 of file numa_node_memory.hpp.

Referenced by foedus::memory::NumaCoreMemory::initialize_once().

97  {
98  return snapshot_offset_chunk_memory_pieces_[core_ordinal];
99  }

Here is the caller graph for this function:

PagePool* foedus::memory::NumaNodeMemory::get_snapshot_pool ( )
inline

Definition at line 58 of file numa_node_memory.hpp.

Referenced by foedus::cache::CacheManagerPimpl::initialize_once(), foedus::memory::NumaCoreMemory::initialize_once(), and foedus::thread::ThreadPimpl::initialize_once().

58 { return &snapshot_pool_; }

Here is the caller graph for this function:

PagePoolOffsetChunk* foedus::memory::NumaNodeMemory::get_volatile_offset_chunk_memory_piece ( foedus::thread::ThreadLocalOrdinal  core_ordinal)
inline

Definition at line 92 of file numa_node_memory.hpp.

Referenced by foedus::memory::NumaCoreMemory::initialize_once().

93  {
94  return volatile_offset_chunk_memory_pieces_[core_ordinal];
95  }

Here is the caller graph for this function:

PagePool* foedus::memory::NumaNodeMemory::get_volatile_pool ( )
inline

Definition at line 57 of file numa_node_memory.hpp.

Referenced by foedus::memory::NumaCoreMemory::initialize_once(), and foedus::thread::ThreadPimpl::initialize_once().

57 { return &volatile_pool_; }

Here is the caller graph for this function:

ErrorStack foedus::memory::NumaNodeMemory::initialize_once ( )
overridevirtual

Implements foedus::DefaultInitializable.

Definition at line 57 of file numa_node_memory.cpp.

References foedus::memory::AlignedMemory::alloc(), allocate_huge_numa_memory(), allocate_numa_memory(), ASSERT_ND, foedus::memory::PagePool::attach(), foedus::EngineOptions::cache_, CHECK_ERROR, foedus::memory::AlignedMemory::get_block(), foedus::memory::PagePool::get_memory_size(), foedus::soc::SharedMemoryRepo::get_node_memory_anchors(), foedus::memory::get_numa_node_size(), foedus::Engine::get_options(), foedus::soc::SocManager::get_shared_memory_repo(), foedus::memory::AlignedMemory::get_size(), foedus::Engine::get_soc_manager(), foedus::soc::SharedMemoryRepo::get_volatile_pool(), foedus::memory::PagePool::initialize(), foedus::memory::PagePool::is_initialized(), foedus::memory::AlignedMemory::kNumaAllocOnnode, foedus::storage::kPageSize, foedus::kRetOk, foedus::EngineOptions::memory_, foedus::memory::MemoryOptions::page_pool_size_mb_per_node_, foedus::memory::MemoryOptions::rigorous_page_boundary_check_, foedus::memory::PagePool::set_debug_pool_name(), foedus::cache::CacheOptions::snapshot_cache_size_mb_per_node_, and foedus::soc::NodeMemoryAnchors::volatile_pool_status_.

57  {
58  LOG(INFO) << "Initializing NumaNodeMemory for node " << static_cast<int>(numa_node_) << "."
59  << " BEFORE: numa_node_size=" << get_numa_node_size(numa_node_);
60 
61 
62  // volatile pool is placed on the shared memory
63  soc::SharedMemoryRepo* memory_repo = engine_->get_soc_manager()->get_shared_memory_repo();
64  uint64_t volatile_size =
65  static_cast<uint64_t>(engine_->get_options().memory_.page_pool_size_mb_per_node_) << 20;
66  volatile_pool_.attach(
67  memory_repo->get_node_memory_anchors(numa_node_)->volatile_pool_status_,
68  memory_repo->get_volatile_pool(numa_node_),
69  volatile_size,
70  true,
72  volatile_pool_.set_debug_pool_name(
73  std::string("VolatilePool-")
74  + std::to_string(static_cast<int>(numa_node_)));
75 
76  // snapshot pool is SOC-local
77  uint64_t snapshot_pool_bytes
78  = static_cast<uint64_t>(engine_->get_options().cache_.snapshot_cache_size_mb_per_node_) << 20;
80  // mprotect raises EINVAL if the underlying pages are hugepages.
81  LOG(INFO) << "rigorous_page_boundary_check_ is specified, so disabled hugepages.";
82  allocate_numa_memory(snapshot_pool_bytes, &snapshot_pool_memory_);
83  } else {
84  allocate_huge_numa_memory(snapshot_pool_bytes, &snapshot_pool_memory_);
85  }
86  snapshot_pool_control_block_.alloc(1 << 12, 1 << 12, AlignedMemory::kNumaAllocOnnode, numa_node_);
87  snapshot_pool_.attach(
88  reinterpret_cast<PagePoolControlBlock*>(snapshot_pool_control_block_.get_block()),
89  snapshot_pool_memory_.get_block(),
90  snapshot_pool_memory_.get_size(),
91  true,
93  snapshot_pool_.set_debug_pool_name(
94  std::string("SnapshotPool-")
95  + std::to_string(static_cast<int>(numa_node_)));
96 
97  CHECK_ERROR(volatile_pool_.initialize());
98  CHECK_ERROR(snapshot_pool_.initialize());
99 
100  // snapshot_pool_ consumes #pages * 4kb bytes of memory.
101  // CacheBucket is 16 bytes, so even with 32-fold (3% full hashtable), we spend only
102  // #pages * 0.5kb for hash buckets. This is a neligible overhead.
103  uint64_t cache_hashtable_buckets = (snapshot_pool_.get_memory_size() / storage::kPageSize) * 32;
104  snapshot_cache_table_ = new cache::CacheHashtable(cache_hashtable_buckets, numa_node_);
105  CHECK_ERROR(initialize_page_offset_chunk_memory());
106  CHECK_ERROR(initialize_log_buffers_memory());
107  for (auto ordinal = 0; ordinal < cores_; ++ordinal) {
108  CHECK_ERROR(initialize_core_memory(ordinal));
109  }
110  ASSERT_ND(volatile_pool_.is_initialized());
111  ASSERT_ND(snapshot_pool_.is_initialized());
112  ASSERT_ND(core_memories_.size() == cores_);
113  ASSERT_ND(volatile_offset_chunk_memory_pieces_.size() == cores_);
114  ASSERT_ND(snapshot_offset_chunk_memory_pieces_.size() == cores_);
115  ASSERT_ND(log_buffer_memory_pieces_.size() == cores_);
116 
117  LOG(INFO) << "Initialized NumaNodeMemory for node " << static_cast<int>(numa_node_) << "."
118  << " AFTER: numa_node_size=" << get_numa_node_size(numa_node_);
119  return kRetOk;
120 }
void attach(PagePoolControlBlock *control_block, void *memory, uint64_t memory_size, bool owns, bool rigorous_page_boundary_check)
Definition: page_pool.cpp:101
ErrorStack allocate_numa_memory(uint64_t size, AlignedMemory *out) const
numa_alloc_onnode() and numa_free().
int64_t get_numa_node_size(int node)
void alloc(uint64_t size, uint64_t alignment, AllocType alloc_type, int numa_node) noexcept
Allocate a memory, releasing the current memory if exists.
const EngineOptions & get_options() const
Definition: engine.cpp:39
memory::MemoryOptions memory_
ErrorStack initialize() override
Acquires resources in this object, usually called right after constructor.
Definition: page_pool.cpp:114
ErrorStack allocate_huge_numa_memory(uint64_t size, AlignedMemory *out) const
void * get_block() const
Returns the memory block.
uint64_t get_size() const
Returns the byte size of the memory block.
#define CHECK_ERROR(x)
This macro calls x and checks its returned value.
uint32_t page_pool_size_mb_per_node_
Size of the page pool in MB per each NUMA node.
const ErrorStack kRetOk
Normal return value for no-error case.
soc::SocManager * get_soc_manager() const
See SOC and IPC.
Definition: engine.cpp:59
uint64_t get_memory_size() const
Definition: page_pool.cpp:117
uint32_t snapshot_cache_size_mb_per_node_
Size of the snapshot cache in MB per each NUMA node.
bool is_initialized() const override
Returns whether the object has been already initialized or not.
Definition: page_pool.cpp:115
#define ASSERT_ND(x)
A warning-free wrapper macro of assert() that has no performance effect in release mode even when 'x'...
Definition: assert_nd.hpp:72
bool rigorous_page_boundary_check_
Whether to use mprotect() for page boundaries to detect bogus memory accesses.
const uint16_t kPageSize
A constant defining the page size (in bytes) of both snapshot pages and volatile pages.
Definition: storage_id.hpp:45
void set_debug_pool_name(const std::string &name)
Call this anytime after attach()
Definition: page_pool.cpp:122
cache::CacheOptions cache_
SharedMemoryRepo * get_shared_memory_repo()
Returns the shared memories maintained across SOCs.
Definition: soc_manager.cpp:38

Here is the call graph for this function:

ErrorStack foedus::memory::NumaNodeMemory::uninitialize_once ( )
overridevirtual

Implements foedus::DefaultInitializable.

Definition at line 175 of file numa_node_memory.cpp.

References foedus::ErrorStackBatch::emprace_back(), foedus::memory::get_numa_node_size(), foedus::memory::AlignedMemory::release_block(), SUMMARIZE_ERROR_BATCH, foedus::memory::PagePool::uninitialize(), and foedus::ErrorStackBatch::uninitialize_and_delete_all().

175  {
176  LOG(INFO) << "Uninitializing NumaNodeMemory for node " << static_cast<int>(numa_node_) << "."
177  << " BEFORE: numa_node_size=" << get_numa_node_size(numa_node_);
178 
179  ErrorStackBatch batch;
180  batch.uninitialize_and_delete_all(&core_memories_);
181  volatile_offset_chunk_memory_pieces_.clear();
182  volatile_offset_chunk_memory_.release_block();
183  snapshot_offset_chunk_memory_pieces_.clear();
184  snapshot_offset_chunk_memory_.release_block();
185  log_buffer_memory_pieces_.clear();
186  log_buffer_memory_.release_block();
187  if (snapshot_cache_table_) {
188  delete snapshot_cache_table_;
189  snapshot_cache_table_ = nullptr;
190  }
191  batch.emprace_back(volatile_pool_.uninitialize());
192  batch.emprace_back(snapshot_pool_.uninitialize());
193  snapshot_pool_memory_.release_block();
194  snapshot_pool_control_block_.release_block();
195 
196  LOG(INFO) << "Uninitialized NumaNodeMemory for node " << static_cast<int>(numa_node_) << "."
197  << " AFTER: numa_node_size=" << get_numa_node_size(numa_node_);
198  return SUMMARIZE_ERROR_BATCH(batch);
199 }
int64_t get_numa_node_size(int node)
void release_block()
Releases the memory block.
ErrorStack uninitialize() override
An idempotent method to release all resources of this object, if any.
Definition: page_pool.cpp:116
#define SUMMARIZE_ERROR_BATCH(x)
This macro calls ErrorStackBatch::summarize() with automatically provided parameters.

Here is the call graph for this function:


The documentation for this class was generated from the following files: