Thread and Thread-Group, which abstracts NUMA-core/node and provides API to attach/detach tasks to pooled threads. More...

Detailed Description

Thread and Thread-Group, which abstracts NUMA-core/node and provides API to attach/detach tasks to pooled threads.

This package is analogous to std::thread or boost::thread with a few additional features specific to our database engine. Among all, the main difference is that this package is fully aware of NUMA (Non Uniform Memory Access) architecture in hardware.

Worker Thread vs NUMA-Core: In our library, each worker thread (a thread that runs user/system transactions) runs on exactly one NUMA core (using numa_run_on_node()), and each NUMA core runs at most one worker thread (except when the user runs multiple engines). Hence, we use the words "thread" and "NUMA-core" interchangablly. The one-to-one mapping is to save thread-context switches and processor cache misses.

Thread and ThreadGroup: Thread Memory Hierarchy: NUMA cores have a hierarchy in hardware, which is explained in Memory Hierarchy. Our thread representations reflect the hierarchy; Thread=NUMA Core, and ThreadGroup=NUMA Node. Unfortunately, the terminology "thread group" has a different meaning in some other context, but I couldn't come with a better name.

Thread Pool and Thread Impersonation: We pre-allocate worker threads in the engine and user-programs impersonate one of the threads to run transactions. For more details, see Thread-Pool and Impersonation.

C++11 and public headers: We do NOT allow C++11 features in public headers; see C++11 Keywords in Public Headers. However, C++11's std::thread and related classes are the only cross-platform library that has no dependencies (boost::thread is NOT header-only). Hence, we use C++11's std::thread but encapsulates the details in implementations (cpp) and non-public header files (-impl.hpp).

Collaboration diagram for Thread and Thread-Group:

Modules
	Thread-Pool and Impersonation
	APIs to pool and impersonate worker threads in libfoedus-core.

Files
file	fwd.hpp
	Forward declarations of classes in thread package.

file	thread_id.hpp
	Typedefs of ID types used in thread package.

Classes
struct	foedus::memory::NumaCoreMemory::SmallThreadLocalMemoryPieces
	Packs pointers to pieces of small_thread_local_memory_. More...

class	foedus::memory::NumaCoreMemory
	Repository of memories dynamically acquired within one CPU core (thread). More...

class	foedus::memory::NumaNodeMemory
	Repository of memories dynamically acquired and shared within one NUMA node (socket). More...

class	foedus::thread::ConditionVariable
	An analogue of pthread's condition variable and std::condition_variable to avoid glibc's bug in pthread_cond_broadcast/signal (thus in notify_all/one in turn). More...

struct	foedus::thread::NumaThreadScope
	Pin the current thread to the given NUMA node in this object's scope. More...

class	foedus::thread::Rendezvous
	The frequently appearing triplet of condition_varible, "signal" flag for spurious wakeup, and mutex for a one-time single-producer multiple-consumer event synchronization. More...

class	foedus::thread::StoppableThread
	The frequently appearing quartet of std::thread, condition_varible, stop-request flag, and mutex. More...

class	foedus::thread::Thread
	Represents one thread running on one NUMA core. More...

class	foedus::thread::ThreadGroup
	Represents a group of pre-allocated threads running in one NUMA node. More...

struct	foedus::thread::ThreadOptions
	Set of options about threads and thread-groups. More...

class	foedus::thread::ThreadPimpl
	Pimpl object of Thread. More...

class	foedus::thread::ThreadPimplMcsAdaptor< RW_BLOCK >
	Implements McsAdaptorConcept over ThreadPimpl. More...

class	foedus::thread::ThreadRef
	A view of Thread object for other SOCs and master engine. More...

class	foedus::thread::ThreadGroupRef
	A view of Thread group object for other SOCs and master engine. More...

Typedefs
typedef uint8_t	foedus::thread::ThreadGroupId
	Typedef for an ID of ThreadGroup (NUMA node). More...

typedef uint8_t	foedus::thread::ThreadLocalOrdinal
	Typedef for a local ID of Thread (core), which is NOT unique across NUMA nodes. More...

typedef uint16_t	foedus::thread::ThreadId
	Typedef for a global ID of Thread (core), which is unique across NUMA nodes. More...

typedef uint16_t	foedus::thread::ThreadGlobalOrdinal
	Typedef for a globally and contiguously numbered ID of thread. More...

typedef uint64_t	foedus::thread::ThreadTicket
	Typedef for a monotonically increasing ticket for thread impersonation. More...

Enumerations
enum	foedus::thread::ThreadPolicy { foedus::thread::kScheduleOther = 0, foedus::thread::kScheduleFifo = 1, foedus::thread::kScheduleRr = 2, foedus::thread::kScheduleBatch = 3, foedus::thread::kScheduleIdle = 5 }
	Thread policy for worker threads. More...

enum	foedus::thread::ThreadPriority { foedus::thread::kPriorityIdle = 0, foedus::thread::kPriorityLowest = 1, foedus::thread::kPriorityDefault = 50, foedus::thread::kPriorityHighest = 99 }
	Thread priority for worker threads. More...

enum	foedus::thread::ThreadStatus { foedus::thread::kNotInitialized = 0, foedus::thread::kWaitingForTask, foedus::thread::kWaitingForExecution, foedus::thread::kRunningTask, foedus::thread::kWaitingForClientRelease, foedus::thread::kWaitingForTerminate, foedus::thread::kTerminated }
	Impersonation status of each worker thread. More...

Functions
ThreadId	foedus::thread::compose_thread_id (ThreadGroupId node, ThreadLocalOrdinal local_core)
	Returns a globally unique ID of Thread (core) for the given node and ordinal in the node. More...

ThreadGroupId	foedus::thread::decompose_numa_node (ThreadId global_id)
	Extracts NUMA node ID from the given globally unique ID of Thread (core). More...

ThreadLocalOrdinal	foedus::thread::decompose_numa_local_ordinal (ThreadId global_id)
	Extracts local ordinal from the given globally unique ID of Thread (core). More...

ThreadGlobalOrdinal	foedus::thread::to_global_ordinal (ThreadId thread_id, uint8_t threads_per_nodes)
	Calculate ThreadGlobalOrdinal from ThreadId. More...

Variables
const ThreadGroupId	foedus::thread::kMaxThreadGroupId = 0xFF
	Maximum possible value of ThreadGroupId. More...

const ThreadLocalOrdinal	foedus::thread::kMaxThreadLocalOrdinal = 0xFF
	Maximum possible value of ThreadLocalOrdinal. More...

const ThreadId	foedus::thread::kMaxThreadId = 0xFFFF
	Maximum possible value of ThreadId. More...

Class Documentation

struct foedus::memory::NumaCoreMemory::SmallThreadLocalMemoryPieces

Packs pointers to pieces of small_thread_local_memory_.

Definition at line 49 of file numa_core_memory.hpp.

Class Members
char *	sysxct_workspace_memory_
char *	xct_lock_free_read_access_memory_
char *	xct_lock_free_write_access_memory_
char *	xct_page_version_memory_
char *	xct_pointer_access_memory_
char *	xct_read_access_memory_
char *	xct_write_access_memory_

Typedef Documentation

foedus::thread::ThreadGlobalOrdinal

Typedef for a globally and contiguously numbered ID of thread.

ThreadId is not contiguous because its high byte is NUMA node, and that is what we use in most places. However, sometimes we need a contiguously numberd ID of thread, such as when we allocate an array of objects, one for each thread. This number is used in such cases. Be extra careful, both ThreadId and this are uint16_t, so compiler won't warn anything if we use them incorrectly.

Definition at line 98 of file thread_id.hpp.

foedus::thread::ThreadGroupId

Typedef for an ID of ThreadGroup (NUMA node).

Currently, we assume there are at most 256 NUMA nodes.

Definition at line 38 of file thread_id.hpp.

foedus::thread::ThreadId

Typedef for a global ID of Thread (core), which is unique across NUMA nodes.

This is a composite of ThreadGroupId (high 1 byte) and ThreadLocalOrdinal (low 1 byte). For example, if there are 2 NUMA nodes and 8 cores each: thread-0=node-0 core-0, thread-1=node-0 core-1, ..., thread-256=node-1 core-0,...

Definition at line 80 of file thread_id.hpp.

foedus::thread::ThreadLocalOrdinal

Typedef for a local ID of Thread (core), which is NOT unique across NUMA nodes.

Currently, we assume there are at most 256 cores per NUMA node.

Definition at line 58 of file thread_id.hpp.

foedus::thread::ThreadTicket

Typedef for a monotonically increasing ticket for thread impersonation.

For every impersonation, the thread increments this value in its control block. Each session receives the ticket and checks the ticket whenever it checks the status of the thread. This is required to avoid double-free and modifying input/output data of other sessions.

Definition at line 117 of file thread_id.hpp.

Enumeration Type Documentation

enum foedus::thread::ThreadPolicy

Thread policy for worker threads.

The values are compatible with pthread's values.

See also: http://man7.org/linux/man-pages/man3/pthread_getschedparam.3.html

Enumerator
kScheduleOther	SCHED_OTHER.
kScheduleFifo	SCHED_FIFO.
kScheduleRr	SCHED_RR.
kScheduleBatch	SCHED_BATCH.
kScheduleIdle	SCHED_IDLE.

Definition at line 166 of file thread_id.hpp.

                   {
   kScheduleOther = 0,
   kScheduleFifo = 1,
   kScheduleRr = 2,
   kScheduleBatch = 3,
   kScheduleIdle = 5,
   // no SCHED_DEADLINE. Too new (linux 3.14~).
 };

enum foedus::thread::ThreadPriority

Thread priority for worker threads.

The values are compatible with pthread's values. Depending on policy, the lowest-highest might be overwritten by what sched_get_priority_max()/min returns.

See also: http://man7.org/linux/man-pages/man3/pthread_getschedparam.3.html

Enumerator
kPriorityIdle
kPriorityLowest
kPriorityDefault
kPriorityHighest

Definition at line 186 of file thread_id.hpp.

                     {
   kPriorityIdle = 0,
   kPriorityLowest = 1,
   kPriorityDefault = 50,
   kPriorityHighest = 99,
 };

enum foedus::thread::ThreadStatus

Impersonation status of each worker thread.

The transition is basically only to next one. Exceptions are:

Every state might jump to kTerminated for whatever reason.
kWaitingForClientRelease goes back to kWaitingForTask when the client picks the result up and closes the session.

Enumerator
kNotInitialized	Initial state. The thread does nothing in this state
kWaitingForTask	Idle state, receiving a new task.
kWaitingForExecution	A client has set a next task. The thread has not picked it up yet.
kRunningTask	The thread has picked the task up and is now running.
kWaitingForClientRelease	The thread has completed the task and set the result. The client has not picked it up yet.
kWaitingForTerminate	The thread is requested to terminate.
kTerminated	The thread has terminated (either error or normal, check the result to differentiate them).

Definition at line 203 of file thread_id.hpp.

                   {
   kNotInitialized = 0,
   kWaitingForTask,
   kWaitingForExecution,
   kRunningTask,
   kWaitingForClientRelease,
   kWaitingForTerminate,
   kTerminated,
 };

Function Documentation

ThreadId foedus::thread::compose_thread_id	(	ThreadGroupId	node,
		ThreadLocalOrdinal	local_core
	)

inline

Returns a globally unique ID of Thread (core) for the given node and ordinal in the node.

Definition at line 123 of file thread_id.hpp.

Referenced by foedus::storage::sequential::SequentialStoragePimpl::drop(), foedus::storage::sequential::SequentialComposer::drop_volatiles(), foedus::storage::sequential::SequentialStoragePimpl::for_every_page(), foedus::thread::ThreadGroup::initialize_once(), foedus::log::LogManagerPimpl::initialize_once(), foedus::memory::NumaCoreMemory::NumaCoreMemory(), and foedus::thread::ThreadGroupRef::ThreadGroupRef().

                                                                                      {
   return (node << 8) | local_core;
 }

Here is the caller graph for this function:

ThreadLocalOrdinal foedus::thread::decompose_numa_local_ordinal ( ThreadId global_id )

inline

Extracts local ordinal from the given globally unique ID of Thread (core).

Definition at line 139 of file thread_id.hpp.

Referenced by foedus::memory::NumaNodeMemory::get_core_memory(), foedus::storage::sequential::SequentialStoragePimpl::get_head_pointer(), foedus::xct::McsMockAdaptor< RW_BLOCK >::get_other_thread(), foedus::storage::sequential::SequentialStoragePimpl::get_tail_pointer(), foedus::thread::ThreadPoolPimpl::get_thread(), foedus::soc::SharedMemoryRepo::get_thread_memory_anchors(), foedus::thread::ThreadPoolPimpl::get_thread_ref(), foedus::log::Logger::initialize_once(), and foedus::thread::to_global_ordinal().

                                                                            {
   return global_id & 0xFF;
 }

Here is the caller graph for this function:

ThreadGroupId foedus::thread::decompose_numa_node ( ThreadId global_id )

inline

Extracts NUMA node ID from the given globally unique ID of Thread (core).

Definition at line 131 of file thread_id.hpp.

Referenced by foedus::storage::sequential::SequentialStoragePimpl::get_head_pointer(), foedus::thread::ThreadRef::get_numa_node(), foedus::thread::Thread::get_numa_node(), foedus::xct::McsMockAdaptor< RW_BLOCK >::get_other_thread(), foedus::storage::sequential::SequentialStoragePimpl::get_tail_pointer(), foedus::thread::ThreadPoolPimpl::get_thread(), foedus::soc::SharedMemoryRepo::get_thread_memory_anchors(), foedus::thread::ThreadPoolPimpl::get_thread_ref(), foedus::thread::ThreadPimpl::handle_tasks(), and foedus::thread::to_global_ordinal().

                                                              {
   return (global_id >> 8) & 0xFF;
 }

Here is the caller graph for this function:

ThreadGlobalOrdinal foedus::thread::to_global_ordinal	(	ThreadId	thread_id,
		uint8_t	threads_per_nodes
	)

inline

Calculate ThreadGlobalOrdinal from ThreadId.

Definition at line 147 of file thread_id.hpp.

References foedus::thread::decompose_numa_local_ordinal(), and foedus::thread::decompose_numa_node().

Referenced by foedus::thread::ThreadGroup::initialize_once().

                                                                                             {
   return decompose_numa_node(thread_id) * threads_per_nodes
     + decompose_numa_local_ordinal(thread_id);
 }

Here is the call graph for this function:

Here is the caller graph for this function:

Variable Documentation

foedus::thread::kMaxThreadGroupId = 0xFF

Maximum possible value of ThreadGroupId.

Definition at line 49 of file thread_id.hpp.

foedus::thread::kMaxThreadId = 0xFFFF

Maximum possible value of ThreadId.

Definition at line 105 of file thread_id.hpp.

foedus::thread::kMaxThreadLocalOrdinal = 0xFF

Maximum possible value of ThreadLocalOrdinal.

Definition at line 69 of file thread_id.hpp.

Referenced by foedus::memory::NumaNodeMemory::get_core_memory_count().

Detailed Description

Modules

Files

Classes

Typedefs

Enumerations

Functions

Variables

Class Documentation

Typedef Documentation

Enumeration Type Documentation

Function Documentation

Variable Documentation