libfoedus-core
FOEDUS Core Library
foedus::log Namespace Reference

Log Manager, which writes out transactional logs. More...

Detailed Description

Log Manager, which writes out transactional logs.

This package contains classes that control transactional logging.

Decentralized Logging
Unlike traditional log manager in DBMS, this log manager is decentralized, meaning each log writer writes to its own file concurrently. This eliminates the bottleneck in log manager when there are a large number of cores. The basic idea to guarantee serializability is the epoch-based commit protocol, which does the check on all loggers before returning the results to client in a way similar to group-commit.
Thread-private log buffer
Each ThreadLoadBugger instance maintains a thread-local log buffer that is filled by the thread without any synchronization or blocking. The logger collects them and writes them out to log files. A single log writer handles one or more transactional threads (cores), and a single NUMA node hosts one or more log writers.
See also
foedus::log::ThreadLogBuffer
Log Writer
Each Logger instance writes out files suffixed with ordinal (eg ".0", ".1"...). The older logs files are deactivated and deleted after log gleaner consumes them.
See also
foedus::log::Logger
Global Durable Epoch
An important job of Log Manager is to maintain the global durable epoch. Log manager is the module to determine when it's safe to advance the global durable epoch. It makes sure all loggers flushed their logs up to the epoch, invoked required fsync(), and also takes a savepoint before it announces the new global durable epoch.
See also
foedus::log::LogManager

Classes

struct  BaseLogType
 Base class for log type. More...
 
struct  EngineLogType
 Base class for log type of engine-wide operation. More...
 
struct  EpochHistory
 Represents an event where a logger switched its epoch. More...
 
struct  EpochMarkerLogType
 A log type to declare a switch of epoch in a logger or the engine. More...
 
struct  FillerLogType
 A dummy log type to fill up a sector in log files. More...
 
class  Logger
 A log writer that writes out buffered logs to stable storages. More...
 
struct  LoggerControlBlock
 Shared data of Logger. More...
 
class  LoggerRef
 A view of Logger object for other SOCs and master engine. More...
 
struct  LogHeader
 A common header part for all log types. More...
 
class  LogManager
 Log Manager class that provides API to write/read transaction logs. More...
 
struct  LogManagerControlBlock
 Shared data in LogManagerPimpl. More...
 
class  LogManagerPimpl
 Pimpl object of LogManager. More...
 
struct  LogOptions
 Set of options for log manager. More...
 
struct  LogRange
 a contiguous range of log entries that might span multiple files. More...
 
class  MetaLogBuffer
 A single log buffer for metadata (eg create/drop storage). More...
 
struct  MetaLogControlBlock
 Control block for MetaLogBuffer and MetaLogger. More...
 
class  MetaLogger
 A log writer for metadata operation. More...
 
struct  RecordLogType
 Base class for log type of record-wise operation. More...
 
struct  StorageLogType
 Base class for log type of storage-wide operation. More...
 
struct  ThreadEpockMark
 A thread-buffer's epoch marker, which indicates where a thread switched an epoch. More...
 
class  ThreadLogBuffer
 A thread-local log buffer. More...
 
struct  ThreadLogBufferMeta
 Metadata part of ThreadLogBuffer, without the actual buffer which is way way larger. More...
 

Typedefs

typedef uint16_t LoggerId
 Typedef for an ID of Logger. More...
 
typedef uint32_t LogFileOrdinal
 Ordinal of log files (eg "log.0", "log.1"). More...
 

Enumerations

enum  LogCode {
  kLogCodeInvalid = 0, kLogCodeFiller = 0x3001, kLogCodeEpochMarker = 0x3002, kLogCodeDropLogType = 0x1011,
  kLogCodeArrayCreate = 0x1021, kLogCodeArrayOverwrite = 0x0022, kLogCodeArrayIncrement = 0x0023, kLogCodeSequentialTruncate = 0x1024,
  kLogCodeSequentialCreate = 0x1025, kLogCodeSequentialAppend = 0x0026, kLogCodeHashCreate = 0x1027, kLogCodeHashOverwrite = 0x0028,
  kLogCodeHashInsert = 0x0029, kLogCodeHashDelete = 0x002A, kLogCodeHashUpdate = 0x002B, kLogCodeMasstreeCreate = 0x1031,
  kLogCodeMasstreeOverwrite = 0x0032, kLogCodeMasstreeInsert = 0x0033, kLogCodeMasstreeDelete = 0x0034, kLogCodeMasstreeUpdate = 0x0035
}
 A unique identifier of all log types. More...
 
enum  LogCodeKind { kRecordLogs = 0, kStorageLogs = 1, kEngineLogs = 2, kMarkerLogs = 3 }
 Represents the kind of log types. More...
 

Functions

void _dummy_static_size_check__COUNTER__ ()
 
LogCodeKind get_log_code_kind (LogCode code)
 Returns the kind of the given log code. More...
 
bool is_valid_log_type (LogCode code)
 Returns if the LogCode value exists. More...
 
const char * get_log_type_name (LogCode code)
 Returns the names of LogCode enum defined in log_type.xmacro. More...
 
template<typename LOG_TYPE >
constexpr LogCode get_log_code ()
 Returns LogCode for the log type defined in log_type.xmacro. More...
 
template<>
constexpr LogCode get_log_code< foedus::log::FillerLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::log::EpochMarkerLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::DropLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::array::ArrayCreateLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::array::ArrayOverwriteLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::array::ArrayIncrementLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::sequential::SequentialTruncateLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::sequential::SequentialCreateLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::sequential::SequentialAppendLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::hash::HashCreateLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::hash::HashOverwriteLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::hash::HashInsertLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::hash::HashDeleteLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::hash::HashUpdateLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::masstree::MasstreeCreateLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::masstree::MasstreeOverwriteLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::masstree::MasstreeInsertLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::masstree::MasstreeDeleteLogType > ()
 
template<>
constexpr LogCode get_log_code< foedus::storage::masstree::MasstreeUpdateLogType > ()
 
void invoke_apply_engine (void *log_buffer, thread::Thread *context)
 Invokes the apply logic for an engine-wide log type. More...
 
void invoke_apply_storage (void *log_buffer, Engine *engine, storage::StorageId id)
 Invokes the apply logic for a storage-wide log type. More...
 
void invoke_apply_record (void *log_buffer, thread::Thread *context, storage::StorageId storage_id, xct::RwLockableXctId *owner_id_address, char *payload_address)
 Invokes the apply logic for a record-wise log type. More...
 
void invoke_assert_valid (void *log_buffer)
 Invokes the assertion logic of each log type. More...
 
void invoke_ostream (const void *buffer, std::ostream *ptr)
 Invokes the ostream operator for the given log type defined in log_type.xmacro. More...
 
std::ostream & operator<< (std::ostream &o, const LogHeader &v)
 
std::ostream & operator<< (std::ostream &o, const FillerLogType &v)
 
std::ostream & operator<< (std::ostream &o, const EpochMarkerLogType &v)
 
std::ostream & operator<< (std::ostream &o, const EpochHistory &v)
 
bool is_log_aligned (uint64_t offset)
 
uint64_t align_log_ceil (uint64_t offset)
 
uint64_t align_log_floor (uint64_t offset)
 
std::ostream & operator<< (std::ostream &o, const Logger &v)
 
std::ostream & operator<< (std::ostream &o, const MetaLogBuffer &v)
 
void on_non_durable_meta_log_found (const log::LogHeader *entry, Epoch durable_epoch, uint64_t offset)
 
std::ostream & operator<< (std::ostream &o, const MetaLogger &v)
 
std::ostream & operator<< (std::ostream &o, const ThreadLogBuffer &v)
 
std::ostream & operator<< (std::ostream &o, const ThreadLogBufferMeta &v)
 
std::ostream & operator<< (std::ostream &o, const ThreadEpockMark &v)
 

Function Documentation

void foedus::log::_dummy_static_size_check__COUNTER__ ( )
inline

Definition at line 115 of file common_log_types.hpp.

121 : public BaseLogType {
uint64_t foedus::log::align_log_ceil ( uint64_t  offset)
inline

Definition at line 62 of file logger_impl.cpp.

62  {
63  return assorted::align< uint64_t, FillerLogType::kLogWriteUnitSize >(offset);
64 }
uint64_t foedus::log::align_log_floor ( uint64_t  offset)
inline

Definition at line 65 of file logger_impl.cpp.

References foedus::log::FillerLogType::kLogWriteUnitSize.

65  {
66  if (offset % FillerLogType::kLogWriteUnitSize == 0) {
67  return offset;
68  } else {
69  return assorted::align< uint64_t, FillerLogType::kLogWriteUnitSize >(offset)
70  - FillerLogType::kLogWriteUnitSize;
71  }
72 }

Definition at line 23 of file log_type.hpp.

31 {
template<>
constexpr LogCode foedus::log::get_log_code< foedus::log::FillerLogType > ( )
inline

Definition at line 22 of file log_type.hpp.

31 {

Definition at line 25 of file log_type.hpp.

31 {

Definition at line 27 of file log_type.hpp.

31 {

Definition at line 26 of file log_type.hpp.

31 {

Definition at line 24 of file log_type.hpp.

31 {

Definition at line 31 of file log_type.hpp.

31 {

Definition at line 34 of file log_type.hpp.

38 :
39  *

Definition at line 33 of file log_type.hpp.

87 {

Definition at line 32 of file log_type.hpp.

32 {

Definition at line 35 of file log_type.hpp.

38 :
39  *

Definition at line 36 of file log_type.hpp.

38 :
39  *

Definition at line 39 of file log_type.hpp.

44 : For engine-wide operation.

Definition at line 38 of file log_type.hpp.

38 :
39  *

Definition at line 37 of file log_type.hpp.

38 :
39  *

Definition at line 40 of file log_type.hpp.

44 : For engine-wide operation.

Definition at line 30 of file log_type.hpp.

31 {

Definition at line 29 of file log_type.hpp.

31 {

Definition at line 28 of file log_type.hpp.

31 {
bool foedus::log::is_log_aligned ( uint64_t  offset)
inline

Definition at line 59 of file logger_impl.cpp.

References foedus::log::FillerLogType::kLogWriteUnitSize.

59  {
60  return offset % FillerLogType::kLogWriteUnitSize == 0;
61 }
void foedus::log::on_non_durable_meta_log_found ( const log::LogHeader entry,
Epoch  durable_epoch,
uint64_t  offset 
)

Definition at line 64 of file meta_logger_impl.cpp.

References foedus::xct::XctId::get_epoch_int(), foedus::xct::XctId::get_ordinal(), invoke_ostream(), foedus::log::LogHeader::log_length_, foedus::log::LogHeader::log_type_code_, foedus::log::LogHeader::storage_id_, and foedus::log::LogHeader::xct_id_.

67  {
68  std::stringstream ss;
69  ss << " <Log offset=\"" << assorted::Hex(offset) << "\""
70  << " len=\"" << assorted::Hex(entry->log_length_) << "\""
71  << " type=\"" << assorted::Hex(entry->log_type_code_) << "\""
72  << " storage_id=\"" << assorted::Hex(entry->storage_id_) << "\"";
73  if (entry->log_length_ >= 8U) {
74  ss << " xct_id_epoch=\"" << entry->xct_id_.get_epoch_int() << "\"";
75  ss << " xct_id_ordinal=\"" << entry->xct_id_.get_ordinal() << "\"";
76  }
77  ss << ">";
78  log::invoke_ostream(entry, &ss);
79  ss << "</Log>";
80  LOG(WARNING) << "Found a meta log that is not in durable epoch (" << durable_epoch
81  << "). Probably the last run didn't invoke wait_for_commit(). The operation is discarded."
82  << " Log content:" << std::endl << ss.str();
83 }
void invoke_ostream(const void *buffer, std::ostream *ptr)
Invokes the ostream operator for the given log type defined in log_type.xmacro.

Here is the call graph for this function:

std::ostream& foedus::log::operator<< ( std::ostream &  o,
const EpochHistory v 
)

Definition at line 26 of file epoch_history.cpp.

References foedus::log::EpochHistory::log_file_offset_, foedus::log::EpochHistory::log_file_ordinal_, foedus::log::EpochHistory::new_epoch_, and foedus::log::EpochHistory::old_epoch_.

26  {
27  o << "<EpochHistory old_epoch=\"" << v.old_epoch_ << "\" new_epoch=\"" << v.new_epoch_
28  << "\" log_file_ordinal=\"" << v.log_file_ordinal_
29  << "\" log_file_offset_=\"" << assorted::Hex(v.log_file_offset_) << "\" />";
30  return o;
31 }
std::ostream& foedus::log::operator<< ( std::ostream &  o,
const LogHeader v 
)

Definition at line 31 of file common_log_types.cpp.

References get_log_type_name(), foedus::log::LogHeader::get_type(), foedus::log::LogHeader::log_length_, foedus::log::LogHeader::log_type_code_, and foedus::log::LogHeader::storage_id_.

31  {
32  o << "<Header type=\"" << assorted::Hex(v.log_type_code_) << "\" type_name=\""
33  << get_log_type_name(v.get_type()) << "\" length=\"" << assorted::Hex(v.log_length_)
34  << "\" storage_id=\"" << v.storage_id_ << "\" />";
35  return o;
36 }
const char * get_log_type_name(LogCode code)
Returns the names of LogCode enum defined in log_type.xmacro.
Definition: log_type.cpp:28

Here is the call graph for this function:

std::ostream& foedus::log::operator<< ( std::ostream &  o,
const FillerLogType v 
)

Definition at line 37 of file common_log_types.cpp.

References foedus::log::BaseLogType::header_.

37  {
38  o << "<FillerLog>" << v.header_ << "</FillerLog>";
39  return o;
40 }
std::ostream& foedus::log::operator<< ( std::ostream &  o,
const EpochMarkerLogType v 
)

Definition at line 42 of file common_log_types.cpp.

References foedus::log::BaseLogType::header_, foedus::log::EpochMarkerLogType::log_file_offset_, foedus::log::EpochMarkerLogType::log_file_ordinal_, foedus::log::EpochMarkerLogType::logger_id_, foedus::log::EpochMarkerLogType::logger_in_node_ordinal_, foedus::log::EpochMarkerLogType::logger_numa_node_, foedus::log::EpochMarkerLogType::new_epoch_, and foedus::log::EpochMarkerLogType::old_epoch_.

42  {
43  o << "<EpochMarker>" << v.header_
44  << "<old_epoch_>" << v.old_epoch_ << "</old_epoch_>"
45  << "<new_epoch_>" << v.new_epoch_ << "</new_epoch_>"
46  << "<logger_numa_node_>" << static_cast<int>(v.logger_numa_node_) << "</logger_numa_node_>"
47  << "<logger_in_node_ordinal_>"
48  << static_cast<int>(v.logger_in_node_ordinal_) << "</logger_in_node_ordinal_>"
49  << "<logger_id_>" << v.logger_id_ << "</logger_id_>"
50  << "<log_file_ordinal_>" << v.log_file_ordinal_ << "</log_file_ordinal_>"
51  << "<log_file_offset_>" << assorted::Hex(v.log_file_offset_) << "</log_file_offset_>"
52  << "</EpochMarker>";
53  return o;
54 }
std::ostream& foedus::log::operator<< ( std::ostream &  o,
const MetaLogBuffer v 
)

Definition at line 70 of file meta_log_buffer.cpp.

References foedus::log::MetaLogControlBlock::buffer_used_, foedus::Attachable< CONTROL_BLOCK >::control_block_, foedus::log::MetaLogControlBlock::durable_offset_, and foedus::log::MetaLogControlBlock::oldest_offset_.

70  {
71  o << "<MetaLogBuffer>"
72  << "<buffer_used_>" << v.control_block_->buffer_used_ << "</buffer_used_>"
73  << "<oldest_offset_>" << v.control_block_->oldest_offset_ << "</oldest_offset_>"
74  << "<durable_offset_>" << v.control_block_->durable_offset_ << "</durable_offset_>"
75  << "</MetaLogBuffer>";
76  return o;
77 }
std::ostream& foedus::log::operator<< ( std::ostream &  o,
const MetaLogger v 
)

Definition at line 228 of file meta_logger_impl.cpp.

228  {
229  o << "<MetaLogger>"
230  << "<current_file_>";
231  if (v.current_file_) {
232  o << *v.current_file_;
233  } else {
234  o << "nullptr";
235  }
236  o << "</current_file_>"
237  << "</MetaLogger>";
238  return o;
239 }
std::ostream& foedus::log::operator<< ( std::ostream &  o,
const ThreadLogBuffer v 
)

Definition at line 330 of file thread_log_buffer.cpp.

330  {
331  o << v.meta_;
332  return o;
333 }
std::ostream& foedus::log::operator<< ( std::ostream &  o,
const ThreadLogBufferMeta v 
)

Definition at line 335 of file thread_log_buffer.cpp.

References foedus::log::ThreadLogBufferMeta::buffer_size_, foedus::log::ThreadLogBufferMeta::current_mark_index_, foedus::log::ThreadLogBufferMeta::increment_mark_index(), foedus::log::ThreadLogBufferMeta::offset_committed_, foedus::log::ThreadLogBufferMeta::offset_durable_, foedus::log::ThreadLogBufferMeta::offset_head_, foedus::log::ThreadLogBufferMeta::offset_tail_, foedus::log::ThreadLogBufferMeta::oldest_mark_index_, foedus::log::ThreadLogBufferMeta::thread_epoch_marks_, and foedus::log::ThreadLogBufferMeta::thread_id_.

335  {
336  o << "<ThreadLogBuffer>";
337  o << "<thread_id_>" << v.thread_id_ << "</thread_id_>";
338  o << "<buffer_size_>" << v.buffer_size_ << "</buffer_size_>";
339  o << "<offset_head_>" << v.offset_head_ << "</offset_head_>";
340  o << "<offset_durable_>" << v.offset_durable_ << "</offset_durable_>";
341  o << "<offset_committed_>" << v.offset_committed_ << "</offset_committed_>";
342  o << "<offset_tail_>" << v.offset_tail_ << "</offset_tail_>";
343  o << "<thread_epoch_marks"
344  << " oldest=\"" << v.oldest_mark_index_ << "\""
345  << " current=\"" << v.current_mark_index_ << "\""
346  << ">";
347 
348  // from oldest (inclusive) to current (exclusive).
349  for (uint32_t i = v.oldest_mark_index_;
350  i != v.current_mark_index_;
351  i = ThreadLogBufferMeta::increment_mark_index(i)) {
352  o << "<Entry index=\"" << i << "\">" << v.thread_epoch_marks_[i] << "</Entry>";
353  }
354  // also write out current entry
355  o << "<Entry index=\"" << v.current_mark_index_ << "\">"
356  << v.thread_epoch_marks_[v.current_mark_index_]
357  << "</Entry>";
358  o << "</thread_epoch_marks>";
359  o << "</ThreadLogBuffer>";
360  return o;
361 }

Here is the call graph for this function:

std::ostream& foedus::log::operator<< ( std::ostream &  o,
const ThreadEpockMark v 
)

Definition at line 363 of file thread_log_buffer.cpp.

References foedus::log::ThreadEpockMark::new_epoch_, foedus::log::ThreadEpockMark::offset_begin_, foedus::log::ThreadEpockMark::offset_end_, and foedus::log::ThreadEpockMark::old_epoch_.

363  {
364  o << "<ThreadEpockMark "
365  << " old=\"" << v.old_epoch_ << "\""
366  << " new=\"" << v.new_epoch_ << "\""
367  << " offset_begin=\"" << assorted::Hex(v.offset_begin_) << "\""
368  << " offset_end=\"" << assorted::Hex(v.offset_end_) << "\""
369  << " />";
370  return o;
371 }
std::ostream& foedus::log::operator<< ( std::ostream &  o,
const Logger v 
)

Definition at line 533 of file logger_impl.cpp.

References foedus::Attachable< CONTROL_BLOCK >::control_block_, foedus::log::LoggerControlBlock::current_ordinal_, foedus::log::LoggerControlBlock::epoch_history_count_, foedus::log::LoggerControlBlock::epoch_history_head_, foedus::fs::DirectIoFile::get_current_offset(), foedus::log::LoggerRef::get_durable_epoch(), foedus::log::LoggerRef::id_, foedus::log::LoggerRef::in_node_ordinal_, foedus::log::LoggerControlBlock::marked_epoch_, foedus::log::LoggerRef::numa_node_, foedus::log::LoggerControlBlock::oldest_file_offset_begin_, and foedus::log::LoggerControlBlock::oldest_ordinal_.

533  {
534  o << "<Logger>"
535  << "<id_>" << v.id_ << "</id_>"
536  << "<numa_node_>" << static_cast<int>(v.numa_node_) << "</numa_node_>"
537  << "<in_node_ordinal_>" << static_cast<int>(v.in_node_ordinal_) << "</in_node_ordinal_>"
538  << "<log_folder_>" << v.log_folder_ << "</log_folder_>";
539  o << "<assigned_thread_ids_>";
540  for (auto thread_id : v.assigned_thread_ids_) {
541  o << "<thread_id>" << thread_id << "</thread_id>";
542  }
543  o << "</assigned_thread_ids_>";
544  o << "<durable_epoch_>" << v.get_durable_epoch() << "</durable_epoch_>"
545  << "<marked_epoch_>" << v.control_block_->marked_epoch_ << "</marked_epoch_>"
546  << "<oldest_ordinal_>" << v.control_block_->oldest_ordinal_ << "</oldest_ordinal_>"
547  << "<oldest_file_offset_begin_>" << v.control_block_->oldest_file_offset_begin_
548  << "</oldest_file_offset_begin_>"
549  << "<current_ordinal_>" << v.control_block_->current_ordinal_ << "</current_ordinal_>";
550 
551  o << "<current_file_>";
552  if (v.current_file_) {
553  o << *v.current_file_;
554  } else {
555  o << "nullptr";
556  }
557  o << "</current_file_>";
558 
559  o << "<current_file_path_>" << v.current_file_path_ << "</current_file_path_>";
560 
561  o << "<current_file_length_>";
562  if (v.current_file_) {
563  o << v.current_file_->get_current_offset();
564  } else {
565  o << "nullptr";
566  }
567  o << "</current_file_length_>";
568 
569  o << "<epoch_history_head>"
570  << v.control_block_->epoch_history_head_ << "</epoch_history_head>";
571  o << "<epoch_history_count>"
572  << v.control_block_->epoch_history_count_ << "</epoch_history_count>";
573  o << "</Logger>";
574  return o;
575 }

Here is the call graph for this function: