libfoedus-core
FOEDUS Core Library
Public Member Functions | Static Public Member Functions | Public Attributes | Friends | List of all members
foedus::storage::PartitionerMetadata Struct Referencefinal
FOEDUS Components » Storage Manager

Tiny metadata of partitioner for every storage used while log gleaning. More...

Detailed Description

Tiny metadata of partitioner for every storage used while log gleaning.

The metadata is tiny because it just points to a data block in a separate partitioner data block, which is variable-sized. Think of Masstree's partitioning information for example. we have to store keys, so we can't statically determine the size. We allocate an array of this object on shared memory so that all mappers/reducers can access the partitioner information.

Index-0 Entry
As storage-id 0 doesn't exist, we use the first entry as metadata of the data block. data_offset_ is the end of already-taken regions while data_size_ is the . When we initialize a new partitioner, we lock mutex_ and increment data_offset_.

Definition at line 209 of file partitioner.hpp.

#include <partitioner.hpp>

Collaboration diagram for foedus::storage::PartitionerMetadata:
Collaboration graph
[legend]

Public Member Functions

 PartitionerMetadata ()=delete
 
 ~PartitionerMetadata ()=delete
 
void initialize ()
 
void uninitialize ()
 
void clear_counts ()
 
void * locate_data (Engine *engine)
 Returns the partitioner data pointed from this metadata. More...
 
ErrorCode allocate_data (Engine *engine, soc::SharedMutexScope *locked, uint32_t data_size)
 Allocates a patitioner data in shared memory of the given size. More...
 

Static Public Member Functions

static PartitionerMetadata * get_metadata (Engine *engine, StorageId id)
 Returns the shared memory for the given storage ID. More...
 
static PartitionerMetadata * get_index0_metadata (Engine *engine)
 Returns the special index-0 entry that manages data block allocation for partitioners. More...
 

Public Attributes

soc::SharedMutex mutex_
 Serialize concurrent initialization of this partitioner. More...
 
bool valid_
 Whether this partitioner information (metadata+data) has been constructed. More...
 
uint32_t data_offset_
 Relative offset from the beginning of partitioner data block that points to variable-sized partitioner data. More...
 
uint32_t data_size_
 The size of the partitioner data. More...
 

Friends

std::ostream & operator<< (std::ostream &o, const PartitionerMetadata &v)
 

Constructor & Destructor Documentation

foedus::storage::PartitionerMetadata::PartitionerMetadata ( )
delete
foedus::storage::PartitionerMetadata::~PartitionerMetadata ( )
delete

Member Function Documentation

ErrorCode foedus::storage::PartitionerMetadata::allocate_data ( Engine *  engine,
soc::SharedMutexScope *  locked,
uint32_t  data_size 
)

Allocates a patitioner data in shared memory of the given size.

Precondition
!valid_ (if it's already constructed, why are we allocating again?)
locked->is_locked_by_me(), locked->get_mutex() == &mutex_ (the caller must own the mutex)
data_size > 0
Postcondition
data_offset_ is set and index0-entry's data_offset_ is incremented.
data_size_ == data_size The only possible error is memory running out (kErrorCodeStrPartitionerDataMemoryTooSmall).

Definition at line 61 of file partitioner.cpp.

References ASSERT_ND, data_offset_, data_size_, get_index0_metadata(), foedus::soc::SharedMutexScope::get_mutex(), foedus::soc::SharedMutexScope::is_locked_by_me(), foedus::kErrorCodeOk, foedus::kErrorCodeStrPartitionerDataMemoryTooSmall, mutex_, and valid_.

Referenced by foedus::storage::hash::HashPartitioner::design_partition(), foedus::storage::array::ArrayPartitioner::design_partition(), and foedus::storage::masstree::MasstreePartitioner::design_partition().

64  {
65  ASSERT_ND(!valid_);
66  ASSERT_ND(data_size > 0);
67  ASSERT_ND(data_size_ == 0);
68  ASSERT_ND(data_offset_ == 0);
69  ASSERT_ND(locked->get_mutex() == &mutex_);
70  ASSERT_ND(locked->is_locked_by_me());
71  PartitionerMetadata* index0 = get_index0_metadata(engine);
72  ASSERT_ND(index0->data_offset_ <= index0->data_size_);
73  soc::SharedMutexScope index0_scope(&index0->mutex_);
74  if (index0->data_offset_ + data_size > index0->data_size_) {
75  return kErrorCodeStrPartitionerDataMemoryTooSmall;
76  }
77  data_offset_ = index0->data_offset_;
78  data_size_ = data_size;
79  index0->data_offset_ += data_size;
80  return kErrorCodeOk;
81 }
foedus::storage::PartitionerMetadata::valid_
bool valid_
Whether this partitioner information (metadata+data) has been constructed.
Definition: partitioner.hpp:237
foedus::kErrorCodeStrPartitionerDataMemoryTooSmall
0x0825 : "STORAGE: Memory for Partitioners ran out during snapshot. Increase StorageOptions::partitio...
Definition: error_code.hpp:188
foedus::kErrorCodeOk
0 means no-error.
Definition: error_code.hpp:87
foedus::storage::PartitionerMetadata::get_index0_metadata
static PartitionerMetadata * get_index0_metadata(Engine *engine)
Returns the special index-0 entry that manages data block allocation for partitioners.
Definition: partitioner.cpp:45
foedus::storage::PartitionerMetadata::data_size_
uint32_t data_size_
The size of the partitioner data.
Definition: partitioner.hpp:246
foedus::storage::PartitionerMetadata::mutex_
soc::SharedMutex mutex_
Serialize concurrent initialization of this partitioner.
Definition: partitioner.hpp:232
foedus::storage::PartitionerMetadata::data_offset_
uint32_t data_offset_
Relative offset from the beginning of partitioner data block that points to variable-sized partitione...
Definition: partitioner.hpp:242
ASSERT_ND
#define ASSERT_ND(x)
A warning-free wrapper macro of assert() that has no performance effect in release mode even when 'x'...
Definition: assert_nd.hpp:72

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::storage::PartitionerMetadata::clear_counts ( )
inline

Definition at line 221 of file partitioner.hpp.

References data_offset_, data_size_, and valid_.

Referenced by initialize().

221  {
222  valid_ = false;
223  data_offset_ = 0;
224  data_size_ = 0;
225  }
foedus::storage::PartitionerMetadata::valid_
bool valid_
Whether this partitioner information (metadata+data) has been constructed.
Definition: partitioner.hpp:237
foedus::storage::PartitionerMetadata::data_size_
uint32_t data_size_
The size of the partitioner data.
Definition: partitioner.hpp:246
foedus::storage::PartitionerMetadata::data_offset_
uint32_t data_offset_
Relative offset from the beginning of partitioner data block that points to variable-sized partitione...
Definition: partitioner.hpp:242

Here is the caller graph for this function:

PartitionerMetadata * foedus::storage::PartitionerMetadata::get_index0_metadata ( Engine *  engine)
static

Returns the special index-0 entry that manages data block allocation for partitioners.

Definition at line 45 of file partitioner.cpp.

References foedus::soc::SharedMemoryRepo::get_global_memory_anchors(), foedus::soc::SocManager::get_shared_memory_repo(), foedus::Engine::get_soc_manager(), and foedus::soc::GlobalMemoryAnchors::partitioner_metadata_.

Referenced by allocate_data().

45  {
46  soc::GlobalMemoryAnchors* anchors
47  = engine->get_soc_manager()->get_shared_memory_repo()->get_global_memory_anchors();
48  return anchors->partitioner_metadata_;
49 }

Here is the call graph for this function:

Here is the caller graph for this function:

PartitionerMetadata * foedus::storage::PartitionerMetadata::get_metadata ( Engine *  engine,
StorageId  id 
)
static

Returns the shared memory for the given storage ID.

Precondition
id > 0

Definition at line 38 of file partitioner.cpp.

References ASSERT_ND, foedus::soc::SharedMemoryRepo::get_global_memory_anchors(), foedus::soc::SocManager::get_shared_memory_repo(), foedus::Engine::get_soc_manager(), and foedus::soc::GlobalMemoryAnchors::partitioner_metadata_.

Referenced by foedus::storage::array::ArrayComposeContext::ArrayComposeContext(), and foedus::storage::masstree::MasstreeComposer::drop_volatiles().

38  {
39  ASSERT_ND(id > 0);
40  soc::GlobalMemoryAnchors* anchors
41  = engine->get_soc_manager()->get_shared_memory_repo()->get_global_memory_anchors();
42  return anchors->partitioner_metadata_ + id;
43 }
ASSERT_ND
#define ASSERT_ND(x)
A warning-free wrapper macro of assert() that has no performance effect in release mode even when 'x'...
Definition: assert_nd.hpp:72

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::storage::PartitionerMetadata::initialize ( )
inline

Definition at line 214 of file partitioner.hpp.

References clear_counts(), foedus::soc::SharedMutex::initialize(), and mutex_.

Referenced by foedus::snapshot::SnapshotManagerPimpl::initialize_once().

214  {
215  mutex_.initialize();
216  clear_counts();
217  }
foedus::soc::SharedMutex::initialize
void initialize(bool recursive=false)
Definition: shared_mutex.cpp:29
foedus::storage::PartitionerMetadata::mutex_
soc::SharedMutex mutex_
Serialize concurrent initialization of this partitioner.
Definition: partitioner.hpp:232

Here is the call graph for this function:

Here is the caller graph for this function:

void * foedus::storage::PartitionerMetadata::locate_data ( Engine *  engine)

Returns the partitioner data pointed from this metadata.

Precondition
valid_
data_size_ > 0

Definition at line 51 of file partitioner.cpp.

References ASSERT_ND, data_offset_, data_size_, foedus::soc::SharedMemoryRepo::get_global_memory_anchors(), foedus::Engine::get_options(), foedus::soc::SocManager::get_shared_memory_repo(), foedus::Engine::get_soc_manager(), foedus::soc::GlobalMemoryAnchors::partitioner_data_, foedus::storage::StorageOptions::partitioner_data_memory_mb_, and foedus::EngineOptions::storage_.

Referenced by foedus::storage::array::ArrayComposeContext::ArrayComposeContext(), foedus::storage::array::ArrayPartitioner::ArrayPartitioner(), foedus::storage::hash::HashPartitioner::design_partition(), foedus::storage::array::ArrayPartitioner::design_partition(), foedus::storage::masstree::MasstreePartitioner::design_partition(), foedus::storage::masstree::MasstreeComposer::drop_volatiles(), foedus::storage::hash::HashPartitioner::HashPartitioner(), and foedus::storage::masstree::MasstreePartitioner::MasstreePartitioner().

51  {
52  ASSERT_ND(data_size_ > 0);
53  ASSERT_ND(data_offset_ + data_size_
54  <= engine->get_options().storage_.partitioner_data_memory_mb_ * (1ULL << 20));
55  soc::GlobalMemoryAnchors* anchors
56  = engine->get_soc_manager()->get_shared_memory_repo()->get_global_memory_anchors();
57  char* buffer = reinterpret_cast<char*>(anchors->partitioner_data_);
58  return buffer + data_offset_;
59 }
foedus::storage::PartitionerMetadata::data_size_
uint32_t data_size_
The size of the partitioner data.
Definition: partitioner.hpp:246
foedus::storage::PartitionerMetadata::data_offset_
uint32_t data_offset_
Relative offset from the beginning of partitioner data block that points to variable-sized partitione...
Definition: partitioner.hpp:242
ASSERT_ND
#define ASSERT_ND(x)
A warning-free wrapper macro of assert() that has no performance effect in release mode even when 'x'...
Definition: assert_nd.hpp:72

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::storage::PartitionerMetadata::uninitialize ( )
inline

Definition at line 218 of file partitioner.hpp.

References mutex_, and foedus::soc::SharedMutex::uninitialize().

Referenced by foedus::snapshot::SnapshotManagerPimpl::uninitialize_once().

218  {
219  mutex_.uninitialize();
220  }
foedus::storage::PartitionerMetadata::mutex_
soc::SharedMutex mutex_
Serialize concurrent initialization of this partitioner.
Definition: partitioner.hpp:232

Here is the call graph for this function:

Here is the caller graph for this function:

Friends And Related Function Documentation

std::ostream& operator<< ( std::ostream &  o,
const PartitionerMetadata &  v 
)
friend

Definition at line 148 of file partitioner.cpp.

148  {
149  o << "<PartitionerMetadata>"
150  << "<valid>" << v.valid_ << "</valid>"
151  << "<data_offset_>" << assorted::Hex(v.data_offset_) << "</data_offset_>"
152  << "<data_size_>" << assorted::Hex(v.data_size_) << "</data_size_>"
153  << "</PartitionerMetadata>";
154  return o;
155 }

Member Data Documentation

uint32_t foedus::storage::PartitionerMetadata::data_offset_

Relative offset from the beginning of partitioner data block that points to variable-sized partitioner data.

Definition at line 242 of file partitioner.hpp.

Referenced by allocate_data(), clear_counts(), foedus::storage::sequential::SequentialPartitioner::design_partition(), locate_data(), and foedus::storage::operator<<().

uint32_t foedus::storage::PartitionerMetadata::data_size_

The size of the partitioner data.

Definition at line 246 of file partitioner.hpp.

Referenced by allocate_data(), clear_counts(), foedus::storage::sequential::SequentialPartitioner::design_partition(), foedus::snapshot::SnapshotManagerPimpl::initialize_once(), locate_data(), and foedus::storage::operator<<().

soc::SharedMutex foedus::storage::PartitionerMetadata::mutex_

Serialize concurrent initialization of this partitioner.

This is taken only when valid_ is false or might be false. As far as one observes valid_==true, he doesn't have to take mutex.

Definition at line 232 of file partitioner.hpp.

Referenced by allocate_data(), foedus::storage::array::ArrayPartitioner::ArrayPartitioner(), foedus::storage::hash::HashPartitioner::design_partition(), foedus::storage::array::ArrayPartitioner::design_partition(), foedus::storage::masstree::MasstreePartitioner::design_partition(), foedus::storage::hash::HashPartitioner::HashPartitioner(), initialize(), foedus::snapshot::SnapshotManagerPimpl::initialize_once(), uninitialize(), and foedus::snapshot::SnapshotManagerPimpl::uninitialize_once().

bool foedus::storage::PartitionerMetadata::valid_

Whether this partitioner information (metadata+data) has been constructed.

When this is false, only valid_ and mutex_ can be safely accessed.

Definition at line 237 of file partitioner.hpp.

Referenced by allocate_data(), foedus::storage::array::ArrayComposeContext::ArrayComposeContext(), foedus::storage::array::ArrayPartitioner::ArrayPartitioner(), clear_counts(), foedus::storage::sequential::SequentialPartitioner::design_partition(), foedus::storage::hash::HashPartitioner::design_partition(), foedus::storage::array::ArrayPartitioner::design_partition(), foedus::storage::masstree::MasstreePartitioner::design_partition(), foedus::storage::masstree::MasstreeComposer::drop_volatiles(), foedus::storage::hash::HashPartitioner::HashPartitioner(), foedus::storage::masstree::MasstreePartitioner::MasstreePartitioner(), and foedus::storage::operator<<().

The documentation for this struct was generated from the following files: