foedus::storage::Composer Class Reference

Represents a logic to compose a new version of data pages for one storage. More...

Detailed Description

Represents a logic to compose a new version of data pages for one storage.

Overview

This object is one of the liaisons between Snapshot Manager module and Storage Manager module. It receives previous snapshot files and pre-sorted log entries from snapshot module, then applies a storage-specific implementation to convert them into a new version of data pages. Every interface is batched and completely separated from the normal transactional processing part. In fact, this object is not part of foedus::storage::Storage at all.

Composer's scope

One composer object is in charge of data pages that meet all of following criteria:

• In one storage
• In one partition (in one NUMA node)
• In one snapshot

None of these responsibilities is overlapping, so the job of composer is totally independent from other composers except the root page of the storage.

Inputs

Every composer receives the following when constructed.

• Corresponding Partitioner object that tells what pages this composer is responsible for.
• Pre-allocated and reused working memory (assured to be on the same NUMA node).
• Pre-sorted stream(s) of log entries (foedus::snapshot::SortedBuffer).
• Snapshot writer to allocate pages and write them out to a snapshot file.
• Most recent snapshot files.

Outputs

Composers emit the following data when it's done.

• Composed data pages, which are written to the snapshot file by the snapshot writer.
• For each storage and for each second-level page that is pointed from the root page, the snapshot pointer and relevant pointer information (eg key range). We call this information as root-info and store them in a tentative page. This is required to construct the root page at the end of snapshotting.

Installing Composed Pages

At the end of snapshotting, composers install pointers to the snapshot pages they composed. These are written to the snapshot pointer part of DualPagePointer so that transactions can start using the snapshot pages. Composers also drop volatile pointers if possible, reducing pressures to volatile page pool. This volatile-drop is carefully done after pausing all transactions because we have to make sure no transactions are newly installing a volatile page while we are dropping its parent.

Shared memory, No virtual methods

Like Partitioner, no virtual methods allowed. We just do switch-case.

Definition at line 86 of file composer.hpp.

#include <composer.hpp>

Classes
struct	ComposeArguments
	Arguments for compose() More...
struct	ConstructRootArguments
	Arguments for construct_root() More...
struct	DropResult
	Retrun value of drop_volatiles() More...
struct	DropVolatilesArguments
	Arguments for drop_volatiles() More...

Public Member Functions
	Composer (Engine *engine, StorageId storage_id)
Engine *	get_engine ()
StorageId	get_storage_id () const
StorageType	get_storage_type () const
ErrorStack	compose (const ComposeArguments &args)
	Construct snapshot pages from sorted run files of one storage. More...
ErrorStack	construct_root (const ConstructRootArguments &args)
	Construct root page(s) for one storage based on the ouputs of compose(). More...
DropResult	drop_volatiles (const DropVolatilesArguments &args)
	Drops volatile pages that have not been modified since the snapshotted epoch. More...
void	drop_root_volatile (const DropVolatilesArguments &args)
	This is additionally called when no partitions observed any new modifications. More...

Friends
std::ostream &	operator<< (std::ostream &o, const Composer &v)

Constructor & Destructor Documentation

foedus::storage::Composer::Composer	(	Engine *	engine,
		StorageId	storage_id
	)

Definition at line 55 of file composer.cpp.

  : engine_(engine),
    storage_id_(storage_id),
    storage_type_(engine_->get_storage_manager()->get_storage(storage_id_)->meta_.type_) {}

Member Function Documentation

ErrorStack foedus::storage::Composer::compose

(

const ComposeArguments &

args

)

Construct snapshot pages from sorted run files of one storage.

Definition at line 61 of file composer.cpp.

References foedus::storage::array::ArrayComposer::compose(), foedus::storage::hash::HashComposer::compose(), foedus::storage::masstree::MasstreeComposer::compose(), foedus::storage::sequential::SequentialComposer::compose(), foedus::storage::kArrayStorage, foedus::storage::kHashStorage, foedus::storage::kMasstreeStorage, foedus::kRetOk, and foedus::storage::kSequentialStorage.

                                                         {
  switch (storage_type_) {
    case kArrayStorage: return array::ArrayComposer(this).compose(args);
    case kHashStorage: return hash::HashComposer(this).compose(args);
    case kSequentialStorage: return sequential::SequentialComposer(this).compose(args);
    case kMasstreeStorage: return masstree::MasstreeComposer(this).compose(args);
    default:
      return kRetOk;
  }
}

Here is the call graph for this function:

ErrorStack foedus::storage::Composer::construct_root

(

const ConstructRootArguments &

args

)

Construct root page(s) for one storage based on the ouputs of compose().

When all reducers complete, the gleaner invokes this method to construct new root page(s) for the storage. This

Definition at line 72 of file composer.cpp.

References foedus::storage::array::ArrayComposer::construct_root(), foedus::storage::hash::HashComposer::construct_root(), foedus::storage::masstree::MasstreeComposer::construct_root(), foedus::storage::sequential::SequentialComposer::construct_root(), foedus::storage::kArrayStorage, foedus::storage::kHashStorage, foedus::storage::kMasstreeStorage, foedus::kRetOk, and foedus::storage::kSequentialStorage.

                                                                      {
  switch (storage_type_) {
    case kArrayStorage: return array::ArrayComposer(this).construct_root(args);
    case kHashStorage: return hash::HashComposer(this).construct_root(args);
    case kSequentialStorage: return sequential::SequentialComposer(this).construct_root(args);
    case kMasstreeStorage: return masstree::MasstreeComposer(this).construct_root(args);
    default:
      return kRetOk;
  }
}

Here is the call graph for this function:

void foedus::storage::Composer::drop_root_volatile

(

const DropVolatilesArguments &

args

)

This is additionally called when no partitions observed any new modifications.

Only in this case, we can drop the root volatile page. Further, we can also drop all the descendent volatile pages safely in this case. Remember these methods are called within xct pausing.

Definition at line 94 of file composer.cpp.

References foedus::storage::array::ArrayComposer::drop_root_volatile(), foedus::storage::hash::HashComposer::drop_root_volatile(), foedus::storage::masstree::MasstreeComposer::drop_root_volatile(), foedus::storage::kArrayStorage, foedus::storage::kHashStorage, foedus::storage::kMasstreeStorage, and foedus::storage::kSequentialStorage.

Referenced by foedus::snapshot::SnapshotManagerPimpl::drop_volatile_pages().

                                                                            {
  switch (storage_type_) {
    case kArrayStorage:
      array::ArrayComposer(this).drop_root_volatile(args);
      return;
    case kHashStorage:
      hash::HashComposer(this).drop_root_volatile(args);
      return;
    case kSequentialStorage:
      // Sequential storage already dropped it. Nothing to do.
      return;
    case kMasstreeStorage:
      masstree::MasstreeComposer(this).drop_root_volatile(args);
      return;
    default:
      return;
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

Composer::DropResult foedus::storage::Composer::drop_volatiles

(

const DropVolatilesArguments &

args

)

Drops volatile pages that have not been modified since the snapshotted epoch.

This is called after pausing transaction executions, so this method does not worry about concurrent reads/writes while running this. Otherwise this method becomes very complex and/or expensive. It's just milliseconds for each several minutes, so should be fine to pause transactions. Also, this method is best-effort in many aspects. It might not drop some volatile pages that were not logically modified. In long run, it will be done at next snapshot, so it's okay to be opportunistic.

Definition at line 83 of file composer.cpp.

References foedus::storage::array::ArrayComposer::drop_volatiles(), foedus::storage::hash::HashComposer::drop_volatiles(), foedus::storage::masstree::MasstreeComposer::drop_volatiles(), foedus::storage::sequential::SequentialComposer::drop_volatiles(), foedus::storage::kArrayStorage, foedus::storage::kHashStorage, foedus::storage::kMasstreeStorage, and foedus::storage::kSequentialStorage.

Referenced by foedus::snapshot::SnapshotManagerPimpl::drop_volatile_pages_parallel().

                                                                              {
  switch (storage_type_) {
    case kArrayStorage:  return array::ArrayComposer(this).drop_volatiles(args);
    case kHashStorage:  return hash::HashComposer(this).drop_volatiles(args);
    case kSequentialStorage: return sequential::SequentialComposer(this).drop_volatiles(args);
    case kMasstreeStorage: return masstree::MasstreeComposer(this).drop_volatiles(args);
    default:
      return DropResult(args);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

Engine* foedus::storage::Composer::get_engine ( )

inline

Definition at line 90 of file composer.hpp.

{ return engine_; }

StorageId foedus::storage::Composer::get_storage_id ( ) const

inline

Definition at line 91 of file composer.hpp.

Referenced by foedus::storage::operator<<().

{ return storage_id_; }

Here is the caller graph for this function:

StorageType foedus::storage::Composer::get_storage_type ( ) const

inline

Definition at line 92 of file composer.hpp.

Referenced by foedus::storage::operator<<().

{ return storage_type_; }

Here is the caller graph for this function:

Friends And Related Function Documentation

std::ostream& operator<< ( std::ostream &  o, const Composer &  v  )

friend

Definition at line 39 of file composer.cpp.

                                                         {
  o << "<Composer>"
    << "<storage_id_>" << v.get_storage_id() << "</storage_id_>"
    << "<storage_type>" << to_storage_type_name(v.get_storage_type()) << "</storage_type>"
    << "</Composer>";
  return o;
}

---

The documentation for this class was generated from the following files:

• /home/shino/foedus_code/foedus-core/include/foedus/storage/composer.hpp
• /home/shino/foedus_code/foedus-core/src/foedus/storage/composer.cpp