foedus::soc::SocManagerPimpl Class Reference

Pimpl object of SocManager. More...

Detailed Description

Pimpl object of SocManager.

A private pimpl object for SocManager. Do not include this header from a client program unless you know what you are doing.

Definition at line 43 of file soc_manager_pimpl.hpp.

#include <soc_manager_pimpl.hpp>

Inheritance diagram for foedus::soc::SocManagerPimpl:

Collaboration diagram for foedus::soc::SocManagerPimpl:

Public Member Functions
	SocManagerPimpl ()=delete
	SocManagerPimpl (Engine *engine)
ErrorStack	initialize_once () override
ErrorStack	uninitialize_once () override
void	report_engine_fatal_error ()
ErrorStack	initialize_master ()
	Called as part of initialize_once() if this is a master engine. More...
ErrorStack	initialize_child ()
	Called as part of initialize_once() if this is a child SOC engine. More...
ErrorStack	launch_emulated_children ()
	Launch emulated children as threads. More...
ErrorStack	launch_forked_children ()
	Launch children via fork. More...
ErrorStack	launch_spawned_children ()
	Launch children via spawn. More...
ErrorStack	wait_for_child_attach ()
	Wait for child SOCs to start up and at least finish attaching shared memory. More...
ErrorStack	wait_for_child_terminate ()
	Wait for child SOCs to terminate. More...
ErrorStack	wait_for_master_status (MasterEngineStatus::StatusCode target_status)
	Wait for master engine to finish upto the specified status. More...
ErrorStack	wait_for_master_module (bool init, ModuleType module)
ErrorStack	wait_for_children_module (bool init, ModuleType module)
void	emulated_child_main (SocId node)
	Main routine of emulated SOCs. More...
int	forked_child_main (SocId node)
	Main routine of forked SOCs. More...
Public Member Functions inherited from foedus::DefaultInitializable
	DefaultInitializable ()
virtual	~DefaultInitializable ()
	DefaultInitializable (const DefaultInitializable &)=delete
DefaultInitializable &	operator= (const DefaultInitializable &)=delete
ErrorStack	initialize () override final
	Typical implementation of Initializable::initialize() that provides initialize-once semantics. More...
ErrorStack	uninitialize () override final
	Typical implementation of Initializable::uninitialize() that provides uninitialize-once semantics. More...
bool	is_initialized () const override final
	Returns whether the object has been already initialized or not. More...
Public Member Functions inherited from foedus::Initializable
virtual	~Initializable ()

Static Public Member Functions
static void	spawned_child_main (const std::vector< proc::ProcAndName > &procedures)
	Main routine of spawned SOCs. More...
static ErrorStack	child_main_common (EngineType engine_type, Upid master_upid, Eid master_eid, SocId node, const std::vector< proc::ProcAndName > &procedures)

Public Attributes
Engine *const	engine_
SharedMemoryRepo	memory_repo_
std::vector< std::thread >	child_emulated_threads_
	Threads that emulate child SOCs. More...
std::vector< Engine * >	child_emulated_engines_
	And their engines. More...
Upid	child_upids_ [kMaxSocs]
	Process IDs of child SOCs. More...

Constructor & Destructor Documentation

foedus::soc::SocManagerPimpl::SocManagerPimpl ( )

delete

foedus::soc::SocManagerPimpl::SocManagerPimpl ( Engine *  engine )

inlineexplicit

Definition at line 46 of file soc_manager_pimpl.hpp.

: engine_(engine) {}

Member Function Documentation

ErrorStack foedus::soc::SocManagerPimpl::child_main_common ( EngineType  engine_type, Upid  master_upid, Eid  master_eid, SocId  node, const std::vector< proc::ProcAndName > &  procedures  )

static

Definition at line 594 of file soc_manager_pimpl.cpp.

References foedus::soc::SharedMemoryRepo::change_child_status(), CHECK_ERROR, COERCE_ERROR, foedus::proc::ProcManager::describe_registered_procs(), foedus::Engine::get_proc_manager(), foedus::Engine::get_soc_manager(), foedus::Engine::initialize(), foedus::ErrorStack::is_error(), foedus::kChildEmulated, foedus::kRetOk, foedus::soc::MasterEngineStatus::kRunning, foedus::soc::ChildEngineStatus::kRunning, foedus::soc::ChildEngineStatus::kTerminated, foedus::soc::MasterEngineStatus::kWaitingForChildTerminate, foedus::soc::ChildEngineStatus::kWaitingForMasterInitialization, foedus::proc::ProcManager::local_register(), memory_repo_, foedus::Engine::uninitialize(), and wait_for_master_status().

Referenced by emulated_child_main(), forked_child_main(), and spawned_child_main().

                                                  {
  thread::NumaThreadScope scope(node);

  // In order to make sure child processes die as soon as the parent dies,
  // we use prctl.
  if (engine_type != kChildEmulated) {
    ::prctl(PR_SET_PDEATHSIG, SIGHUP);
  }

  Engine soc_engine(engine_type, master_upid, master_eid, node);
  ErrorStack init_error = soc_engine.initialize();
  if (init_error.is_error()) {
    std::cerr << "[FOEDUS-Child] Failed to initialize child SOC-" << node
      << ". error=" << init_error
      << " This is an unrecoverable error. This process quits shortly" << std::endl;
    CHECK_ERROR(soc_engine.uninitialize());
    return init_error;
  }

  SocManagerPimpl* soc_this = soc_engine.get_soc_manager()->pimpl_;
  SharedMemoryRepo& soc_memory = soc_this->memory_repo_;

  // after initialize(), we can safely use glog.
  LOG(INFO) << "The SOC engine-" << node << " was initialized.";

  // Add the given procedures.
  proc::ProcManager* procm = soc_engine.get_proc_manager();
  for (const proc::ProcAndName& proc_and_name : procedures) {
    COERCE_ERROR(procm->local_register(proc_and_name));
  }

  LOG(INFO) << "Added user procedures: " << procm->describe_registered_procs()
    << ". Waiting for master engine's initialization...";
  soc_memory.change_child_status(node, ChildEngineStatus::kWaitingForMasterInitialization);
  COERCE_ERROR(soc_this->wait_for_master_status(MasterEngineStatus::kRunning));
  LOG(INFO) << "The SOC engine-" << node << " detected that master engine has started"
    << " running.";
  soc_memory.change_child_status(node, ChildEngineStatus::kRunning);
  COERCE_ERROR(soc_this->wait_for_master_status(MasterEngineStatus::kWaitingForChildTerminate));

  LOG(INFO) << "Stopping the SOC engine-" << node;
  soc_memory.change_child_status(node, ChildEngineStatus::kTerminated);
  ErrorStack uninit_error = soc_engine.uninitialize();
  if (uninit_error.is_error()) {
    LOG(ERROR) << "Error while uninitializing SOC engine-" << node << ": " << uninit_error;
    return uninit_error;
  }

  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::soc::SocManagerPimpl::emulated_child_main

(

SocId

node

)

Main routine of emulated SOCs.

Definition at line 462 of file soc_manager_pimpl.cpp.

References child_main_common(), engine_, foedus::Engine::get_master_eid(), foedus::Engine::get_master_upid(), foedus::proc::ProcManager::get_pre_registered_procedures(), foedus::Engine::get_proc_manager(), foedus::ErrorStack::is_error(), and foedus::kChildEmulated.

Referenced by launch_emulated_children().

                                                    {
  Upid master_upid = engine_->get_master_upid();
  Eid master_eid = engine_->get_master_eid();
  // We can reuse procedures pre-registered in master. Good for being a thread.
  const auto& procedures = engine_->get_proc_manager()->get_pre_registered_procedures();
  ErrorStack ret = child_main_common(kChildEmulated, master_upid, master_eid, node, procedures);
  if (ret.is_error()) {
    std::cerr << "[FOEDUS-Child] Emulated SOC-" << node
      << " exits with an error: " << ret << std::endl;
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

int foedus::soc::SocManagerPimpl::forked_child_main

(

SocId

node

)

Main routine of forked SOCs.

Returns

exit code as a process

Definition at line 503 of file soc_manager_pimpl.cpp.

References child_main_common(), engine_, foedus::Engine::get_master_eid(), foedus::Engine::get_master_upid(), foedus::proc::ProcManager::get_pre_registered_procedures(), foedus::Engine::get_proc_manager(), foedus::ErrorStack::is_error(), and foedus::kChildForked.

Referenced by launch_forked_children().

                                                 {
  Upid master_upid = engine_->get_master_upid();
  Eid master_eid = engine_->get_master_eid();
  // These are pre-registered before fork(), so still we can reuse.
  const auto& procedures = engine_->get_proc_manager()->get_pre_registered_procedures();
  ErrorStack ret = child_main_common(kChildForked, master_upid, master_eid, node, procedures);
  if (ret.is_error()) {
    std::cerr << "[FOEDUS-Child] Forked SOC-" << node
      << " exits with an error: " << ret << std::endl;
    return EXIT_FAILURE;
  } else {
    return EXIT_SUCCESS;
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::initialize_child

(

)

Called as part of initialize_once() if this is a child SOC engine.

Definition at line 128 of file soc_manager_pimpl.cpp.

References foedus::soc::SharedMemoryRepo::attach_shared_memories(), foedus::soc::SharedMemoryRepo::change_child_status(), COERCE_ERROR, engine_, foedus::Engine::get_master_eid(), foedus::Engine::get_master_upid(), foedus::Engine::get_nonconst_options(), foedus::Engine::get_soc_id(), foedus::ErrorStack::is_error(), foedus::kRetOk, foedus::soc::ChildEngineStatus::kSharedMemoryAttached, foedus::soc::MasterEngineStatus::kSharedMemoryReservedReclamation, memory_repo_, and wait_for_master_status().

Referenced by initialize_once().

                                             {
  Upid master_upid = engine_->get_master_upid();
  Eid master_eid = engine_->get_master_eid();
  SocId soc_id = engine_->get_soc_id();
  pid_t pid = ::getpid();
  ErrorStack attach_error = memory_repo_.attach_shared_memories(
    master_upid,
    master_eid,
    soc_id,
    engine_->get_nonconst_options());
  if (attach_error.is_error()) {
    std::cerr << "[FOEDUS-Child] MasterUpid=" << master_upid << ", ChildPid=" << pid
      << ", Node=" << soc_id << ". Failed to attach shared memory. error=" << attach_error
      << " This is an unrecoverable error. This process quits shortly" << std::endl;
    ::_exit(1);
  }
  memory_repo_.change_child_status(soc_id, ChildEngineStatus::kSharedMemoryAttached);

  // child engines could just move on, but it's safer to wait for at least the reclamation
  // of shared memory by master engine.
  COERCE_ERROR(wait_for_master_status(MasterEngineStatus::kSharedMemoryReservedReclamation));
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::initialize_master

(

)

Called as part of initialize_once() if this is a master engine.

Definition at line 86 of file soc_manager_pimpl.cpp.

References foedus::soc::SharedMemoryRepo::allocate_shared_memories(), CHECK_ERROR, child_emulated_engines_, child_emulated_threads_, foedus::soc::NodeMemoryAnchors::child_status_memory_, child_upids_, foedus::soc::SharedMemoryRepo::deallocate_shared_memories(), engine_, foedus::soc::SharedMemoryRepo::get_global_memory_anchors(), foedus::Engine::get_master_eid(), foedus::Engine::get_master_upid(), foedus::soc::SharedMemoryRepo::get_node_memory_anchors(), foedus::Engine::get_options(), foedus::thread::ThreadOptions::group_count_, foedus::soc::MasterEngineStatus::initialized_modules_, foedus::soc::ChildEngineStatus::initialized_modules_, foedus::ErrorStack::is_error(), foedus::kChildForked, foedus::kChildLocalSpawned, foedus::kChildRemoteSpawned, foedus::kDummyTail, foedus::soc::ChildEngineStatus::kInitial, foedus::kInvalid, foedus::kRetOk, foedus::soc::MasterEngineStatus::kSharedMemoryAllocated, launch_emulated_children(), launch_forked_children(), launch_spawned_children(), foedus::soc::GlobalMemoryAnchors::master_status_memory_, memory_repo_, foedus::EngineOptions::soc_, foedus::soc::SocOptions::soc_type_, foedus::fs::status(), foedus::soc::MasterEngineStatus::status_code_, foedus::soc::ChildEngineStatus::status_code_, foedus::EngineOptions::thread_, foedus::soc::MasterEngineStatus::uninitialized_modules_, and foedus::soc::ChildEngineStatus::uninitialized_modules_.

Referenced by initialize_once().

                                              {
  ErrorStack alloc_error = memory_repo_.allocate_shared_memories(
    engine_->get_master_upid(),
    engine_->get_master_eid(),
    engine_->get_options());
  if (alloc_error.is_error()) {
    memory_repo_.deallocate_shared_memories();
    return alloc_error;
  }

  // shared memory allocated. now launch child SOCs
  child_emulated_engines_.clear();
  child_emulated_threads_.clear();
  std::memset(child_upids_, 0, sizeof(child_upids_));

  // set initial value of initialize/uninitialize status now
  {
    MasterEngineStatus* status = memory_repo_.get_global_memory_anchors()->master_status_memory_;
    status->status_code_ = MasterEngineStatus::kSharedMemoryAllocated;
    status->initialized_modules_ = kInvalid;
    status->uninitialized_modules_ = kDummyTail;
  }
  uint16_t soc_count = engine_->get_options().thread_.group_count_;
  for (uint16_t node = 0; node < soc_count; ++node) {
    ChildEngineStatus* status = memory_repo_.get_node_memory_anchors(node)->child_status_memory_;
    status->status_code_ = ChildEngineStatus::kInitial;
    status->initialized_modules_ = kInvalid;
    status->uninitialized_modules_ = kDummyTail;
  }

  EngineType soc_type = engine_->get_options().soc_.soc_type_;
  if (soc_type == kChildForked) {
    CHECK_ERROR(launch_forked_children());
  } else if (soc_type == kChildLocalSpawned || soc_type == kChildRemoteSpawned) {
    // so far remote spawn does local spawn
    CHECK_ERROR(launch_spawned_children());
  } else {
    CHECK_ERROR(launch_emulated_children());
  }
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::initialize_once ( )

overridevirtual

Implements foedus::DefaultInitializable.

Definition at line 49 of file soc_manager_pimpl.cpp.

References engine_, initialize_child(), initialize_master(), and foedus::Engine::is_master().

                                            {
  if (engine_->is_master()) {
    return initialize_master();
  } else {
    return initialize_child();
  }
}

Here is the call graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::launch_emulated_children

(

)

Launch emulated children as threads.

Definition at line 445 of file soc_manager_pimpl.cpp.

References CHECK_ERROR, child_emulated_engines_, child_emulated_threads_, emulated_child_main(), engine_, foedus::Engine::get_options(), foedus::thread::ThreadOptions::group_count_, foedus::kRetOk, foedus::EngineOptions::thread_, and wait_for_child_attach().

Referenced by initialize_master().

                                                     {
  uint16_t soc_count = engine_->get_options().thread_.group_count_;
  for (uint16_t node = 0; node < soc_count; ++node) {
    child_emulated_engines_.push_back(nullptr);
  }
  // from now on child_emulated_engines_ doesn't grow/shrink
  for (uint16_t node = 0; node < soc_count; ++node) {
    child_emulated_threads_.emplace_back(std::thread(
      &SocManagerPimpl::emulated_child_main,
      this,
      node));
  }

  CHECK_ERROR(wait_for_child_attach());
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::launch_forked_children

(

)

Launch children via fork.

Definition at line 479 of file soc_manager_pimpl.cpp.

References foedus::soc::SharedMemoryRepo::change_master_status(), CHECK_ERROR, child_upids_, foedus::soc::SharedMemoryRepo::deallocate_shared_memories(), engine_, ERROR_STACK, forked_child_main(), foedus::Engine::get_options(), foedus::thread::ThreadOptions::group_count_, foedus::kErrorCodeSocForkFailed, foedus::soc::MasterEngineStatus::kFatalError, foedus::kRetOk, memory_repo_, foedus::assorted::os_error(), foedus::EngineOptions::thread_, and wait_for_child_attach().

Referenced by initialize_master().

                                                   {
  uint16_t soc_count = engine_->get_options().thread_.group_count_;
  for (uint16_t node = 0; node < soc_count; ++node) {
    pid_t pid = ::fork();
    if (pid == -1) {
      // failed to fork! immediately release shared memory and return.
      memory_repo_.change_master_status(MasterEngineStatus::kFatalError);
      memory_repo_.deallocate_shared_memories();
      std::cerr << "[FOEDUS] Failed to fork child SOC. error=" << assorted::os_error() << std::endl;
      // the already forked children will shortly notice that the parent failed and exits.
      return ERROR_STACK(kErrorCodeSocForkFailed);
    } else if (pid == 0) {
      // child
      int child_ret = forked_child_main(node);
      ::_exit(child_ret);
    } else {
      // parent. move on.
      child_upids_[node] = pid;
    }
  }
  CHECK_ERROR(wait_for_child_attach());
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::launch_spawned_children

(

)

Launch children via spawn.

Definition at line 523 of file soc_manager_pimpl.cpp.

References foedus::soc::SharedMemoryRepo::change_master_status(), CHECK_ERROR, child_upids_, foedus::soc::SocOptions::convert_spawn_executable_pattern(), foedus::soc::SocOptions::convert_spawn_ld_library_path_pattern(), foedus::soc::SharedMemoryRepo::deallocate_shared_memories(), engine_, ERROR_STACK, foedus::Engine::get_master_eid(), foedus::Engine::get_master_upid(), foedus::Engine::get_options(), foedus::thread::ThreadOptions::group_count_, foedus::kErrorCodeSocSpawnFailed, foedus::soc::MasterEngineStatus::kFatalError, foedus::kRetOk, memory_repo_, foedus::assorted::os_error(), foedus::EngineOptions::soc_, foedus::EngineOptions::thread_, and wait_for_child_attach().

Referenced by initialize_master().

                                                    {
  Upid master_upid = engine_->get_master_upid();
  Eid master_eid = engine_->get_master_eid();
  uint16_t soc_count = engine_->get_options().thread_.group_count_;
  for (uint16_t node = 0; node < soc_count; ++node) {
    posix_spawn_file_actions_t file_actions;
    posix_spawnattr_t attr;
    ::posix_spawn_file_actions_init(&file_actions);
    ::posix_spawnattr_init(&attr);
    std::string executable = engine_->get_options().soc_.convert_spawn_executable_pattern(node);
    std::string ld_path = engine_->get_options().soc_.convert_spawn_ld_library_path_pattern(node);
    std::string ld_env("LD_LIBRARY_PATH=");
    ld_env += ld_path;
    std::string pid_env("FOEDUS_MASTER_UPID=");
    pid_env += std::to_string(master_upid);
    std::string eid_env("FOEDUS_MASTER_EID=");
    eid_env += std::to_string(master_eid);
    std::string soc_id_env("FOEDUS_SOC_ID=");
    soc_id_env += std::to_string(node);

    char* const argv[] = { const_cast<char*>(executable.c_str()), nullptr};
    char* const envp[] = {
      const_cast<char*>(ld_env.c_str()),
      const_cast<char*>(pid_env.c_str()),
      const_cast<char*>(eid_env.c_str()),
      const_cast<char*>(soc_id_env.c_str()),
      nullptr};

    pid_t child_pid;
    int ret = ::posix_spawn(&child_pid, executable.c_str(), &file_actions, &attr, argv, envp);
    if (ret == -1) {
      // failed to spawn! immediately release shared memory and return.
      memory_repo_.change_master_status(MasterEngineStatus::kFatalError);
      memory_repo_.deallocate_shared_memories();
      std::cerr << "[FOEDUS] Failed to spawn child SOC. error="
        << assorted::os_error() << std::endl;
      // the already spawned children will shortly notice that the parent failed and exits.
      return ERROR_STACK(kErrorCodeSocSpawnFailed);
    }
    child_upids_[node] = child_pid;
  }
  CHECK_ERROR(wait_for_child_attach());
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::soc::SocManagerPimpl::report_engine_fatal_error

(

)

Definition at line 67 of file soc_manager_pimpl.cpp.

References foedus::soc::MasterEngineStatus::change_status_atomic(), foedus::soc::ChildEngineStatus::change_status_atomic(), engine_, foedus::soc::SharedMemoryRepo::get_global_memory_anchors(), foedus::soc::SharedMemoryRepo::get_node_memory_anchors(), foedus::Engine::get_soc_id(), foedus::DefaultInitializable::is_initialized(), foedus::Engine::is_master(), foedus::soc::MasterEngineStatus::kFatalError, foedus::soc::ChildEngineStatus::kFatalError, foedus::soc::GlobalMemoryAnchors::master_status_memory_, memory_repo_, and foedus::fs::status().

Referenced by foedus::soc::SocManager::report_engine_fatal_error().

                                                {
  if (!is_initialized()) {
      std::cerr << "[FOEDUS] Shared memory not initialized yet. Can't report errors" << std::endl;
      return;
  }
  if (engine_->is_master()) {
    MasterEngineStatus* status = memory_repo_.get_global_memory_anchors()->master_status_memory_;
    if (status) {
      status->change_status_atomic(MasterEngineStatus::kFatalError);
    }
  } else {
    ChildEngineStatus* status = memory_repo_.get_node_memory_anchors(
      engine_->get_soc_id())->child_status_memory_;
    if (status) {
      status->change_status_atomic(ChildEngineStatus::kFatalError);
    }
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void foedus::soc::SocManagerPimpl::spawned_child_main ( const std::vector< proc::ProcAndName > &  procedures )

static

Main routine of spawned SOCs.

This is invoked from user's main(), so it's static.

Definition at line 568 of file soc_manager_pimpl.cpp.

References child_main_common(), foedus::ErrorStack::is_error(), and foedus::kChildLocalSpawned.

Referenced by foedus::soc::SocManager::trap_spawned_soc_main().

                                                                                       {
  const char* master_upid_str = std::getenv("FOEDUS_MASTER_UPID");
  const char* master_eid_str = std::getenv("FOEDUS_MASTER_EID");
  const char* soc_id_str = std::getenv("FOEDUS_SOC_ID");
  if (master_upid_str == nullptr || master_eid_str == nullptr || soc_id_str == nullptr) {
    return;  // not launched as an SOC engine. exit
  }

  Upid master_upid = std::atoll(master_upid_str);
  Eid master_eid = std::atoll(master_eid_str);
  SocId node = std::atol(master_upid_str);
  ErrorStack ret = child_main_common(kChildLocalSpawned, master_upid, master_eid, node, procedures);
  if (ret.is_error()) {
    std::cerr << "[FOEDUS-Child] Spawned SOC-" << node
      << " exits with an error: " << ret << std::endl;
    ::_exit(EXIT_FAILURE);
  } else {
    ::_exit(EXIT_SUCCESS);
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::uninitialize_once ( )

overridevirtual

Implements foedus::DefaultInitializable.

Definition at line 57 of file soc_manager_pimpl.cpp.

References foedus::soc::SharedMemoryRepo::change_master_status(), CHECK_ERROR, foedus::soc::SharedMemoryRepo::deallocate_shared_memories(), engine_, foedus::soc::SharedMemoryRepo::get_global_memory(), foedus::Engine::is_master(), foedus::soc::MasterEngineStatus::kTerminated, memory_repo_, SUMMARIZE_ERROR_BATCH, and wait_for_child_terminate().

                                              {
  ErrorStackBatch batch;
  if (engine_->is_master() && memory_repo_.get_global_memory() != nullptr) {
    CHECK_ERROR(wait_for_child_terminate());  // wait for children to terminate
    memory_repo_.change_master_status(MasterEngineStatus::kTerminated);
  }
  memory_repo_.deallocate_shared_memories();
  return SUMMARIZE_ERROR_BATCH(batch);
}

Here is the call graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::wait_for_child_attach

(

)

Wait for child SOCs to start up and at least finish attaching shared memory.

Definition at line 152 of file soc_manager_pimpl.cpp.

References foedus::soc::SharedMemoryRepo::change_master_status(), child_upids_, foedus::soc::SharedMemoryRepo::deallocate_shared_memories(), engine_, ERROR_STACK, foedus::soc::SharedMemoryRepo::get_child_status(), foedus::Engine::get_options(), foedus::thread::ThreadOptions::group_count_, foedus::kChildEmulated, foedus::kErrorCodeSocLaunchTimeout, foedus::kErrorCodeSocShmAttachFailed, foedus::soc::MasterEngineStatus::kFatalError, foedus::soc::ChildEngineStatus::kInitial, foedus::kRetOk, foedus::soc::ChildEngineStatus::kSharedMemoryAttached, foedus::soc::MasterEngineStatus::kSharedMemoryReservedReclamation, foedus::soc::SharedMemoryRepo::mark_for_release(), memory_repo_, foedus::assorted::os_error(), foedus::EngineOptions::soc_, foedus::soc::SocOptions::soc_type_, foedus::assorted::spinlock_yield(), foedus::fs::status(), and foedus::EngineOptions::thread_.

Referenced by launch_emulated_children(), launch_forked_children(), and launch_spawned_children().

                                                  {
  // As of this method, glog is not initialized. So, use std::cerr only.
  // When this method begins, the shared memory is in the most fragile state, not marked for
  // reclamation but needs to be open for shmget. Be careful in this method!
  // We avoid even assertions in this method.
  // As soon as something unusual happens, we conservatively release the shared memory.
  uint16_t soc_count = engine_->get_options().thread_.group_count_;

  // robustly wait until all children attached the shared memory.
  // mark-for-reclaim as soon as possible.
  const uint32_t kIntervalMillisecond = 20;
  const uint32_t kTimeoutMillisecond = 10000;
  uint32_t trials = 0;
  bool child_as_process = engine_->get_options().soc_.soc_type_ != kChildEmulated;
  while (true) {
    assorted::spinlock_yield();
    std::this_thread::sleep_for(std::chrono::milliseconds(kIntervalMillisecond));
    bool error_happened = false;
    bool remaining = false;
    for (uint16_t node = 0; node < soc_count; ++node) {
      ChildEngineStatus::StatusCode child_status = memory_repo_.get_child_status(node);
      switch (child_status) {
      case ChildEngineStatus::kInitial:
        remaining = true;
        break;
      case ChildEngineStatus::kSharedMemoryAttached:
        break;
      default:
        // all other statuses are unexpected
        error_happened = true;
        break;
      }
      if (child_as_process) {
        // if we launched the child as a process, let's also check with waitpid()
        // this doesn't do anything if they are emulated children
        if (!error_happened && child_upids_[node] != 0) {
          int status = 0;
          pid_t wait_ret = ::waitpid(child_upids_[node], &status, WNOHANG | __WALL);
          if (wait_ret == -1) {
            std::cerr << "[FOEDUS] FATAL! waitpid() for child-process " << child_upids_[node]
              << " failed. os error=" << assorted::os_error() << std::endl;
            error_happened = true;
            break;
          } else if (wait_ret != 0) {
            std::cerr << "[FOEDUS] FATAL! child-process " << child_upids_[node] << " has exit"
              << " unexpectedly. status=" << status << std::endl;
            error_happened = true;
            break;
          }
        }
      }
    }

    if (error_happened) {
      std::cerr << "[FOEDUS] FATAL! Some child failed to attach shared memory." << std::endl;
      memory_repo_.change_master_status(MasterEngineStatus::kFatalError);
      memory_repo_.deallocate_shared_memories();
      return ERROR_STACK(kErrorCodeSocShmAttachFailed);
    } else if (!remaining) {
      break;  // done!
    } else if ((++trials) * kIntervalMillisecond > kTimeoutMillisecond) {
      std::cerr << "[FOEDUS] FATAL! Timeout happend while waiting for child SOCs to start up."
        " Probably child SOC(s) hanged or did not trap SOC execution (if spawned)." << std::endl;
      memory_repo_.change_master_status(MasterEngineStatus::kFatalError);
      memory_repo_.deallocate_shared_memories();
      return ERROR_STACK(kErrorCodeSocLaunchTimeout);
    }
  }

  // as soon as all children ack-ed, mark the shared memory for release.
  // no one will newly issue shmget.
  memory_repo_.change_master_status(MasterEngineStatus::kSharedMemoryReservedReclamation);
  memory_repo_.mark_for_release();  // now it's safe. closed attaching and marked for reclaim.
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::wait_for_child_terminate

(

)

Wait for child SOCs to terminate.

Definition at line 228 of file soc_manager_pimpl.cpp.

References foedus::soc::SharedMemoryRepo::change_master_status(), child_emulated_threads_, child_upids_, engine_, ERROR_STACK, foedus::soc::SharedMemoryRepo::get_child_status(), foedus::Engine::get_options(), foedus::thread::ThreadOptions::group_count_, foedus::kChildEmulated, foedus::kErrorCodeSocTerminateFailed, foedus::kErrorCodeSocTerminateTimeout, foedus::soc::MasterEngineStatus::kFatalError, foedus::soc::ChildEngineStatus::kFatalError, foedus::kRetOk, foedus::soc::ChildEngineStatus::kRunning, foedus::soc::ChildEngineStatus::kTerminated, memory_repo_, foedus::EngineOptions::soc_, foedus::soc::SocOptions::soc_type_, foedus::assorted::spinlock_yield(), foedus::fs::status(), and foedus::EngineOptions::thread_.

Referenced by uninitialize_once().

                                                     {
  uint16_t soc_count = engine_->get_options().thread_.group_count_;
  const uint32_t kIntervalMillisecond = 20;
  const uint32_t kTimeoutMillisecond = 10000;
  uint32_t trials = 0;
  bool child_as_process = engine_->get_options().soc_.soc_type_ != kChildEmulated;
  while (true) {
    assorted::spinlock_yield();
    std::this_thread::sleep_for(std::chrono::milliseconds(kIntervalMillisecond));
    bool remaining = false;
    for (uint16_t node = 0; node < soc_count; ++node) {
      ChildEngineStatus::StatusCode child_status = memory_repo_.get_child_status(node);
      if (!child_as_process) {
        if (child_status == ChildEngineStatus::kTerminated
          || child_status == ChildEngineStatus::kFatalError) {
          if (child_emulated_threads_[node].joinable()) {
            child_emulated_threads_[node].join();
          }
        } else {
          if (child_emulated_threads_[node].joinable()) {
            remaining = true;
          }
        }
        continue;
      }
      if (child_status == ChildEngineStatus::kRunning) {
        // also check with waitpid(). if the child process terminated, it's fine.
        if (child_upids_[node] != 0) {
          int status = 0;
          pid_t wait_ret = ::waitpid(child_upids_[node], &status, WNOHANG | __WALL);
          if (wait_ret == -1) {
            // this is okay, too. the process has already terminated
          } else if (wait_ret == 0) {
            remaining = true;
          } else if (WIFSIGNALED(status)) {
            std::cerr << "[FOEDUS] ERROR! child-process " << child_upids_[node] << " has been"
              << " terminated by signal. status=" << status << std::endl;
            memory_repo_.change_master_status(MasterEngineStatus::kFatalError);
            return ERROR_STACK(kErrorCodeSocTerminateFailed);
          }
        }
      }
    }

    if (!remaining) {
      break;  // done!
    } else if ((++trials) * kIntervalMillisecond > kTimeoutMillisecond) {
      std::cerr << "[FOEDUS] ERROR! Timeout happend while waiting for child SOCs to terminate."
        " Probably child SOC(s) hanged or did not trap SOC execution (if spawned)." << std::endl;
      memory_repo_.change_master_status(MasterEngineStatus::kFatalError);
      return ERROR_STACK(kErrorCodeSocTerminateTimeout);
    }
  }

  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::wait_for_children_module	(	bool	init,
		ModuleType	module
	)

Definition at line 360 of file soc_manager_pimpl.cpp.

References ASSERT_ND, foedus::soc::NodeMemoryAnchors::child_status_memory_, child_upids_, engine_, ERROR_STACK, foedus::soc::SharedMemoryRepo::get_node_memory_anchors(), foedus::Engine::get_options(), foedus::thread::ThreadOptions::group_count_, foedus::soc::ChildEngineStatus::initialized_modules_, foedus::kChildEmulated, foedus::kErrorCodeSocChildInitFailed, foedus::kErrorCodeSocChildUninitFailed, foedus::soc::ChildEngineStatus::kFatalError, foedus::kRetOk, foedus::kSoc, foedus::assorted::memory_fence_acq_rel(), memory_repo_, foedus::EngineOptions::soc_, foedus::soc::SocOptions::soc_type_, foedus::assorted::spinlock_yield(), foedus::fs::status(), foedus::soc::ChildEngineStatus::status_code_, foedus::EngineOptions::thread_, and foedus::soc::ChildEngineStatus::uninitialized_modules_.

Referenced by foedus::soc::SocManager::wait_for_children_module().

                                                                                  {
  ASSERT_ND(desired != kSoc);  // this method assumes SOC manager is active
  uint16_t soc_count = engine_->get_options().thread_.group_count_;
  bool child_as_process = engine_->get_options().soc_.soc_type_ != kChildEmulated;

  // TASK(Hideaki) should be a function in soc manager
  // We also check if the child died unexpectedly
  const uint32_t kWarnSleeps = 400;
  for (uint32_t count = 0;; ++count) {
    if (count > 0 && count % kWarnSleeps == 0) {
      LOG(WARNING) << "Suspiciously long wait for child " << (init ? "" : "un") << "initializing"
        << " module-" << desired << ". count=" << count;
    }
    assorted::spinlock_yield();
    std::this_thread::sleep_for(std::chrono::milliseconds(5));
    assorted::memory_fence_acq_rel();
    bool error_happened = false;
    bool remaining = false;
    for (uint16_t node = 0; node < soc_count; ++node) {
      soc::ChildEngineStatus* status
        = memory_repo_.get_node_memory_anchors(node)->child_status_memory_;
      if (status->status_code_ == soc::ChildEngineStatus::kFatalError) {
        error_happened = true;
        break;
      }
      if (init) {
        if (status->initialized_modules_ == desired) {
          continue;  // ok
        } else if (static_cast<int>(status->initialized_modules_)
            > static_cast<int>(desired)) {
          LOG(ERROR) << "[FOEDUS] child init went too far??";
          error_happened = true;
          break;
        }
      } else {
        if (status->uninitialized_modules_ == desired) {
          continue;  // ok
        } else if (static_cast<int>(status->uninitialized_modules_)
            < static_cast<int>(desired)) {
          LOG(ERROR) << "[FOEDUS] ERROR! child uninit went too far??";
          error_happened = true;
          break;
        }
      }
      remaining = true;
      if (child_as_process) {
        // TASK(Hideaki) check child process status with waitpid
        if (child_upids_[node] != 0) {
          int status = 0;
          pid_t wait_ret = ::waitpid(child_upids_[node], &status, WNOHANG | __WALL);
          if (wait_ret == -1) {
            // this is okay, too. the process has already terminated
            error_happened = true;
            LOG(ERROR) << "waitpid() while waiting for child module status failed";
            break;
          } else if (wait_ret != 0) {
            // this is okay
            error_happened = true;
            LOG(ERROR) << "child process has already exit while waiting for child module status";
            break;
          }
        }
      }
    }

    if (error_happened) {
      LOG(ERROR) << "Error encountered in wait_for_children_module";
      if (init) {
        return ERROR_STACK(kErrorCodeSocChildInitFailed);
      } else {
        return ERROR_STACK(kErrorCodeSocChildUninitFailed);
      }
    } else if (!remaining) {
      break;
    }
  }
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::wait_for_master_module	(	bool	init,
		ModuleType	module
	)

Definition at line 327 of file soc_manager_pimpl.cpp.

References ERROR_STACK, foedus::soc::SharedMemoryRepo::get_global_memory_anchors(), foedus::soc::MasterEngineStatus::initialized_modules_, foedus::kErrorCodeSocChildInitFailed, foedus::kErrorCodeSocChildUninitFailed, foedus::soc::MasterEngineStatus::kFatalError, foedus::kRetOk, foedus::soc::MasterEngineStatus::kTerminated, foedus::soc::GlobalMemoryAnchors::master_status_memory_, foedus::assorted::memory_fence_acq_rel(), memory_repo_, foedus::assorted::spinlock_yield(), foedus::fs::status(), foedus::soc::MasterEngineStatus::status_code_, and foedus::soc::MasterEngineStatus::uninitialized_modules_.

Referenced by foedus::soc::SocManager::wait_for_master_module().

                                                                                {
  const uint32_t kWarnSleeps = 400;
  for (uint32_t count = 0;; ++count) {
    if (count > 0 && count % kWarnSleeps == 0) {
      LOG(WARNING) << "Suspiciously long wait for master " << (init ? "" : "un") << "initializing"
        << " module-" << desired << ". count=" << count;
    }
    soc::MasterEngineStatus* status
      = memory_repo_.get_global_memory_anchors()->master_status_memory_;
    if (status->status_code_ == MasterEngineStatus::kFatalError
      || status->status_code_ == MasterEngineStatus::kTerminated) {
      LOG(ERROR) << "Master apparently died while wait_for_master_module";
      if (init) {
        return ERROR_STACK(kErrorCodeSocChildInitFailed);
      } else {
        return ERROR_STACK(kErrorCodeSocChildUninitFailed);
      }
    }
    assorted::spinlock_yield();
    assorted::memory_fence_acq_rel();
    ModuleType cur;
    if (init) {
      cur = status->initialized_modules_;
    } else {
      cur = status->uninitialized_modules_;
    }
    if (cur == desired) {
      break;
    }
    std::this_thread::sleep_for(std::chrono::milliseconds(5));
  }
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

ErrorStack foedus::soc::SocManagerPimpl::wait_for_master_status

(

MasterEngineStatus::StatusCode

target_status

)

Wait for master engine to finish upto the specified status.

Definition at line 286 of file soc_manager_pimpl.cpp.

References ASSERT_ND, engine_, ERROR_STACK, foedus::soc::SharedMemoryRepo::get_master_status(), foedus::Engine::is_master(), foedus::kErrorCodeSocMasterDied, foedus::kErrorCodeSocMasterUnexpectedState, foedus::soc::MasterEngineStatus::kFatalError, foedus::kRetOk, memory_repo_, and foedus::assorted::spinlock_yield().

Referenced by child_main_common(), and initialize_child().

                                                                                             {
  ASSERT_ND(!engine_->is_master());
  const uint32_t kIntervalMillisecond = 10;
//  bool child_as_process = engine_->get_options().soc_.soc_type_ != kChildEmulated;
  while (true) {
    assorted::spinlock_yield();
    std::this_thread::sleep_for(std::chrono::milliseconds(kIntervalMillisecond));
    MasterEngineStatus::StatusCode master_status = memory_repo_.get_master_status();
    if (master_status == target_status) {
      break;  // wait done
    } else if (master_status == MasterEngineStatus::kFatalError) {
      return ERROR_STACK(kErrorCodeSocMasterDied);
    } else if (static_cast<int>(master_status) > static_cast<int>(target_status)) {
      return ERROR_STACK(kErrorCodeSocMasterUnexpectedState);
    }
/*
  this doesn't work because waitpid is only for child processes.
  As an alternative to die when parent dies, we use prctl().
    if (child_as_process) {
      // Also check parent's process status.
      Upid master_upid = engine_->get_master_upid();
      int status = 0;
      pid_t wait_ret = ::waitpid(master_upid, &status, WNOHANG);
      if (wait_ret == -1) {
        std::cerr << "[FOEDUS-Child] FATAL! waitpid() for master-process " << master_upid
          << " failed. os error:" << assorted::os_error() << std::endl;
        return ERROR_STACK(kErrorCodeSocMasterDied);
      } else if (WIFEXITED(status)) {
        std::cerr << "[FOEDUS-Child] FATAL! master-process " << master_upid << " has exit"
          << " unexpectedly. status=" << status << std::endl;
        return ERROR_STACK(kErrorCodeSocMasterDied);
      } else if (WIFSIGNALED(status)) {
        std::cerr << "[FOEDUS-Child] FATAL! master-process " << master_upid << " has been"
          << " terminated by signal. status=" << status << std::endl;
        return ERROR_STACK(kErrorCodeSocMasterDied);
      }
    }
*/
  }
  return kRetOk;
}

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

std::vector< Engine* > foedus::soc::SocManagerPimpl::child_emulated_engines_

And their engines.

These pointers become invalid when the threads quit

Definition at line 95 of file soc_manager_pimpl.hpp.

Referenced by initialize_master(), and launch_emulated_children().

std::vector<std::thread> foedus::soc::SocManagerPimpl::child_emulated_threads_

Threads that emulate child SOCs.

Used when this is a master engine and also children are emulated.

Definition at line 93 of file soc_manager_pimpl.hpp.

Referenced by initialize_master(), launch_emulated_children(), and wait_for_child_terminate().

Upid foedus::soc::SocManagerPimpl::child_upids_[kMaxSocs]

Process IDs of child SOCs.

Used when this is a master engine and also children are not emulated.

Definition at line 101 of file soc_manager_pimpl.hpp.

Referenced by initialize_master(), launch_forked_children(), launch_spawned_children(), wait_for_child_attach(), wait_for_child_terminate(), and wait_for_children_module().

Engine* const foedus::soc::SocManagerPimpl::engine_

Definition at line 86 of file soc_manager_pimpl.hpp.

Referenced by emulated_child_main(), forked_child_main(), foedus::soc::SocManager::get_shared_user_memory_size(), initialize_child(), initialize_master(), initialize_once(), launch_emulated_children(), launch_forked_children(), launch_spawned_children(), report_engine_fatal_error(), uninitialize_once(), wait_for_child_attach(), wait_for_child_terminate(), wait_for_children_module(), and wait_for_master_status().

SharedMemoryRepo foedus::soc::SocManagerPimpl::memory_repo_

Definition at line 87 of file soc_manager_pimpl.hpp.

Referenced by child_main_common(), foedus::soc::SocManager::get_shared_memory_repo(), foedus::soc::SocManager::get_shared_user_memory(), initialize_child(), initialize_master(), launch_forked_children(), launch_spawned_children(), report_engine_fatal_error(), uninitialize_once(), wait_for_child_attach(), wait_for_child_terminate(), wait_for_children_module(), wait_for_master_module(), and wait_for_master_status().

---

The documentation for this class was generated from the following files:

• /home/shino/foedus_code/foedus-core/include/foedus/soc/soc_manager_pimpl.hpp
• /home/shino/foedus_code/foedus-core/src/foedus/soc/soc_manager_pimpl.cpp