foedus::cache::SnapshotFileSet Class Reference

Holds a set of read-only file objects for snapshot files. More...

Detailed Description

Holds a set of read-only file objects for snapshot files.

In essense, this is a map<snapshot_id, map<node_id, DirectIoFile*> >. Each thread memory internally maintains this file set. This is because DirectIoFile (or underlying linux read()) cannot be concurrently used. Each thread thus obtains its own file descriptors using this object. As it's thread-local, no synchronization is needed in this object.

This design might hit the maximum number of file descriptors per process. Check cat /proc/sys/fs/file-max if that happens. Google how to change it (soft AND hard limits).

Todo:

So far we really use std::map. But, this is not ideal in terms of performance. node-id is up to 256, snapshots are almost always very few, so we can do array-based something.

Definition at line 52 of file snapshot_file_set.hpp.

#include <snapshot_file_set.hpp>

Inheritance diagram for foedus::cache::SnapshotFileSet:

Collaboration diagram for foedus::cache::SnapshotFileSet:

Public Member Functions
	SnapshotFileSet (Engine *engine)
ErrorStack	initialize_once () override
ErrorStack	uninitialize_once () override
void	close_all ()
Engine *	get_engine ()
	SnapshotFileSet ()=delete
	SnapshotFileSet (const SnapshotFileSet &other)=delete
SnapshotFileSet &	operator= (const SnapshotFileSet &other)=delete
ErrorCode	get_or_open_file (storage::SnapshotPagePointer page_pointer, fs::DirectIoFile **out)
ErrorCode	get_or_open_file (snapshot::SnapshotId snapshot_id, thread::ThreadGroupId node_id, fs::DirectIoFile **out)
ErrorCode	read_page (storage::SnapshotPagePointer page_id, void *out)
ErrorCode	read_pages (storage::SnapshotPagePointer page_id_begin, uint32_t page_count, void *out)
	Read contiguous pages in one shot. More...
Public Member Functions inherited from foedus::DefaultInitializable
	DefaultInitializable ()
virtual	~DefaultInitializable ()
	DefaultInitializable (const DefaultInitializable &)=delete
DefaultInitializable &	operator= (const DefaultInitializable &)=delete
ErrorStack	initialize () override final
	Typical implementation of Initializable::initialize() that provides initialize-once semantics. More...
ErrorStack	uninitialize () override final
	Typical implementation of Initializable::uninitialize() that provides uninitialize-once semantics. More...
bool	is_initialized () const override final
	Returns whether the object has been already initialized or not. More...
Public Member Functions inherited from foedus::Initializable
virtual	~Initializable ()

Friends
std::ostream &	operator<< (std::ostream &o, const SnapshotFileSet &v)

Constructor & Destructor Documentation

foedus::cache::SnapshotFileSet::SnapshotFileSet ( Engine *  engine )

explicit

Definition at line 34 of file snapshot_file_set.cpp.

                                               : engine_(engine) {
}

foedus::cache::SnapshotFileSet::SnapshotFileSet ( )

delete

foedus::cache::SnapshotFileSet::SnapshotFileSet ( const SnapshotFileSet &  other )

delete

Member Function Documentation

void foedus::cache::SnapshotFileSet::close_all

(

)

Definition at line 47 of file snapshot_file_set.cpp.

Referenced by uninitialize_once().

                                {
  for (auto& files_in_a_snapshot : files_) {
    auto& values = files_in_a_snapshot.second;
    for (auto& file : values) {
      file.second->close();
      delete file.second;
    }
    values.clear();
  }
  files_.clear();
}

Here is the caller graph for this function:

Engine* foedus::cache::SnapshotFileSet::get_engine ( )

inline

Definition at line 59 of file snapshot_file_set.hpp.

{ return engine_; }

ErrorCode foedus::cache::SnapshotFileSet::get_or_open_file ( storage::SnapshotPagePointer  page_pointer, fs::DirectIoFile **  out  )

inline

Definition at line 65 of file snapshot_file_set.hpp.

References foedus::storage::extract_numa_node_from_snapshot_pointer(), and foedus::storage::extract_snapshot_id_from_snapshot_pointer().

Referenced by read_page(), and read_pages().

                                                                                            {
    snapshot::SnapshotId snapshot_id
      = storage::extract_snapshot_id_from_snapshot_pointer(page_pointer);
    thread::ThreadGroupId node_id = storage::extract_numa_node_from_snapshot_pointer(page_pointer);
    return get_or_open_file(snapshot_id, node_id, out);
  }

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorCode foedus::cache::SnapshotFileSet::get_or_open_file	(	snapshot::SnapshotId	snapshot_id,
		thread::ThreadGroupId	node_id,
		fs::DirectIoFile **	out
	)

Definition at line 59 of file snapshot_file_set.cpp.

References ASSERT_ND, foedus::snapshot::SnapshotOptions::construct_snapshot_file_path(), foedus::Engine::get_options(), foedus::kErrorCodeOk, and foedus::EngineOptions::snapshot_.

                        {
  *out = nullptr;
  auto snapshot = files_.find(snapshot_id);
  if (snapshot == files_.end()) {
    files_.insert(
      std::pair<snapshot::SnapshotId, std::map< thread::ThreadGroupId, fs::DirectIoFile* > >(
        snapshot_id, std::map< thread::ThreadGroupId, fs::DirectIoFile* >()));
    snapshot = files_.find(snapshot_id);
  }
  ASSERT_ND(snapshot != files_.end());
  ASSERT_ND(snapshot->first == snapshot_id);
  auto& the_map = snapshot->second;
  auto node = the_map.find(node_id);
  if (node != the_map.end()) {
    *out = node->second;
    return kErrorCodeOk;
  } else {
    fs::Path path(engine_->get_options().snapshot_.construct_snapshot_file_path(
      snapshot_id,
      node_id));
    fs::DirectIoFile* file = new fs::DirectIoFile(path);
    ErrorCode open_error = file->open(true, false, false, false);
    if (open_error != kErrorCodeOk) {
      delete file;
      return open_error;
    }
    std::pair< thread::ThreadGroupId, fs::DirectIoFile* > entry(node_id, file);
    the_map.insert(entry);
    *out = file;
    return kErrorCodeOk;
  }
}

Here is the call graph for this function:

ErrorStack foedus::cache::SnapshotFileSet::initialize_once ( )

overridevirtual

Implements foedus::DefaultInitializable.

Definition at line 37 of file snapshot_file_set.cpp.

References foedus::kRetOk.

                                            {
  return kRetOk;
}

SnapshotFileSet& foedus::cache::SnapshotFileSet::operator= ( const SnapshotFileSet &  other )

delete

ErrorCode foedus::cache::SnapshotFileSet::read_page	(	storage::SnapshotPagePointer	page_id,
		void *	out
	)

Definition at line 95 of file snapshot_file_set.cpp.

References ASSERT_ND, CHECK_ERROR_CODE, foedus::storage::extract_local_page_id_from_snapshot_pointer(), get_or_open_file(), foedus::fs::DirectIoFile::kDirectIoSeekSet, foedus::kErrorCodeOk, foedus::fs::DirectIoFile::read_raw(), and foedus::fs::DirectIoFile::seek().

Referenced by foedus::storage::array::ArrayComposer::construct_root(), foedus::storage::hash::HashComposer::construct_root(), foedus::storage::sequential::SequentialComposer::construct_root(), foedus::storage::masstree::MasstreeStoragePimpl::debugout_single_thread_follow(), foedus::storage::masstree::MasstreePartitioner::design_partition(), foedus::storage::hash::TmpSnashotPage::init(), foedus::memory::EngineMemory::load_one_volatile_page(), foedus::storage::hash::ComposedBinsMergedStream::open_path(), foedus::thread::ThreadPimpl::read_a_snapshot_page(), and foedus::storage::StorageManagerPimpl::reinitialize_for_recovered_snapshot().

                                                                                  {
  fs::DirectIoFile* file;
  CHECK_ERROR_CODE(get_or_open_file(page_id, &file));
  storage::SnapshotLocalPageId local_page_id
    = storage::extract_local_page_id_from_snapshot_pointer(page_id);
  CHECK_ERROR_CODE(
    file->seek(local_page_id * sizeof(storage::Page), fs::DirectIoFile::kDirectIoSeekSet));
  CHECK_ERROR_CODE(file->read_raw(sizeof(storage::Page), out));
  ASSERT_ND(reinterpret_cast<storage::Page*>(out)->get_header().page_id_ == page_id);
  return kErrorCodeOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorCode foedus::cache::SnapshotFileSet::read_pages	(	storage::SnapshotPagePointer	page_id_begin,
		uint32_t	page_count,
		void *	out
	)

Read contiguous pages in one shot.

Definition at line 107 of file snapshot_file_set.cpp.

References ASSERT_ND, CHECK_ERROR_CODE, foedus::storage::extract_local_page_id_from_snapshot_pointer(), get_or_open_file(), foedus::fs::DirectIoFile::kDirectIoSeekSet, foedus::kErrorCodeOk, foedus::fs::DirectIoFile::read_raw(), and foedus::fs::DirectIoFile::seek().

Referenced by foedus::storage::hash::ComposedBinsBuffer::next_pages(), and foedus::thread::ThreadPimpl::read_snapshot_pages().

             {
  fs::DirectIoFile* file;
  CHECK_ERROR_CODE(get_or_open_file(page_id_begin, &file));
  storage::SnapshotLocalPageId local_page_id_begin
    = storage::extract_local_page_id_from_snapshot_pointer(page_id_begin);
  CHECK_ERROR_CODE(
    file->seek(local_page_id_begin * sizeof(storage::Page), fs::DirectIoFile::kDirectIoSeekSet));
  CHECK_ERROR_CODE(file->read_raw(sizeof(storage::Page) * page_count, out));
#ifndef NDEBUG
  storage::Page* pages = reinterpret_cast<storage::Page*>(out);
  for (uint32_t i = 0; i < page_count; ++i) {
    ASSERT_ND(pages[i].get_header().page_id_ == page_id_begin + i);
  }
#endif  // NDEBUG
  return kErrorCodeOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::cache::SnapshotFileSet::uninitialize_once ( )

overridevirtual

Implements foedus::DefaultInitializable.

Definition at line 41 of file snapshot_file_set.cpp.

References close_all(), and SUMMARIZE_ERROR_BATCH.

                                              {
  ErrorStackBatch batch;
  close_all();
  return SUMMARIZE_ERROR_BATCH(batch);
}

Here is the call graph for this function:

Friends And Related Function Documentation

std::ostream& operator<< ( std::ostream &  o, const SnapshotFileSet &  v  )

friend

Definition at line 127 of file snapshot_file_set.cpp.

                                                                {
  o << "<SnapshotFileSet>";
  for (const auto& snapshot : v.files_) {
    o << "<snapshot id=\"" << snapshot.first << "\">";
    for (const auto& entry : snapshot.second) {
      o << "<node id=\"" << entry.first
        << "\" fd=\"" << entry.second->get_descriptor() << "\" />";
    }
    o << "</snapshot>";
  }
  return o;
}

---

The documentation for this class was generated from the following files:

• /home/shino/foedus_code/foedus-core/include/foedus/cache/snapshot_file_set.hpp
• /home/shino/foedus_code/foedus-core/src/foedus/cache/snapshot_file_set.cpp