Filesystem wrapper

Filesystem wrapper, an analogue of boost::filesystem. More...

Detailed Description

Filesystem wrapper, an analogue of boost::filesystem.

These methods abstract accesses to filesystems like boost::filesystem and std::filesystem in C++1z(?). We should not directly call POSIX or Windows filesystem APIs in other modules. Instead, all of them should go through this package.

Why this package exists

The best case scenario for us is the standard C++ library provides filesystem abstraction. However, the spec (based on Boost filesystem ver2) didn't get into even C++14, let alone its implmentations. Thus, we can't rely on it at all. Boost filesystem has a few issues, too. First, it is changing and has some issue when C++11 is enabled. Furthermore, the filesystem package is NOT a header-only module in boost. We do not want to introduce additional dependencies to the gigantic boost shared library, which might not be available or has versioning issues in some environment. Rather, we just need only a small subset of functionalities in those libraries because our usage of filesystem is quite minimal and simplistic. Hence, we do it ourselves.

Direct File I/O

Another goal to have this package is to abstract Direct I/O (O_DIRECT in linux and FILE_FLAG_NO_BUFFERING in Windows). We need them to bypass filesystem-level caching, but these are not in C++ standards at all and won't be (if it happens, Linus would get a heart attack: https://lkml.org/lkml/2007/1/10/233 ).

Class/method designs

Basically, we clone Boost filesystem's class/method into this package when we find a need for some filesystem API access in our code. Due to our specialized needs, we might simplify the API. Our goal is not to duplicate all Boost filesystem classes. Just add what we need.

Collaboration diagram for Filesystem wrapper:

Files
file	fwd.hpp
	Forward declarations of classes in filesystem package.

Classes
struct	foedus::fs::DeviceEmulationOptions
	Set of configurations to emulate slower devices for some experiments. More...
class	foedus::fs::DirectIoFile
	Represents an I/O stream on one file without filesystem caching. More...
struct	foedus::fs::FileStatus
	Analogue of boost::filesystem::file_status. More...
struct	foedus::fs::SpaceInfo
	Analogue of boost::filesystem::space_info. More...
class	foedus::fs::Path
	Analogue of boost::filesystem::path. More...

Typedefs
typedef assorted::FixedString< 508 >	foedus::fs::FixedPath
	Represents a fixed (thus can be placed in shared memory) path string. More...

Enumerations
enum	foedus::fs::FileType {   foedus::fs::kStatusError = 0, foedus::fs::kFileNotFound, foedus::fs::kRegularFile, foedus::fs::kDirectoryFile,   foedus::fs::kTypeUnknown }
	Analogue of boost::filesystem::file_type. More...
enum	foedus::fs::FilePermission {   foedus::fs::kNoPerms = 0, foedus::fs::kOwnerRead = 0400, foedus::fs::kOwnerWrite = 0200, foedus::fs::kOwnerExe = 0100,   foedus::fs::kOwnerAll = 0700, foedus::fs::kGroupRead = 040, foedus::fs::kGroupWrite = 020, foedus::fs::kGroupExe = 010,   foedus::fs::kGroupAll = 070, foedus::fs::kOthersRead = 04, foedus::fs::kOthersWrite = 02, foedus::fs::kOthersExe = 01,   foedus::fs::kOthersAll = 07, foedus::fs::kAllAll = kOwnerAll|kGroupAll|kOthersAll, foedus::fs::kPermsNotKnown = 0xFFFF }
	Analogue of boost::filesystem::perm. More...

Functions
FileStatus	foedus::fs::status (const Path &p)
	Returns the status of the file. More...
bool	foedus::fs::exists (const Path &p)
	Returns if the file exists. More...
bool	foedus::fs::is_directory (const Path &p)
	Returns if the file is a directory. More...
bool	foedus::fs::is_regular_file (const Path &p)
	Returns if the file is a regular file. More...
Path	foedus::fs::current_path ()
	Returns the current working directory. More...
Path	foedus::fs::home_path ()
	Returns the absolute path of the home directory of the user running this process. More...
Path	foedus::fs::absolute (const std::string &p)
	Returns the absolue path of the specified path. More...
bool	foedus::fs::create_directories (const Path &p, bool sync=false)
	Recursive mkdir (mkdirs). More...
bool	foedus::fs::create_directory (const Path &p, bool sync=false)
	mkdir. More...
uint64_t	foedus::fs::file_size (const Path &p)
	Returns size of the file. More...
bool	foedus::fs::remove (const Path &p)
	Deletes a regular file or an empty directory. More...
uint64_t	foedus::fs::remove_all (const Path &p)
	Recursively deletes a directory. More...
SpaceInfo	foedus::fs::space (const Path &p)
	Returns free space information for the device the file is on. More...
std::string	foedus::fs::unique_name (uint64_t differentiator=0)
	Equivalent to unique_path("%%%%-%%%%-%%%%-%%%%"). More...
std::string	foedus::fs::unique_name (const std::string &model, uint64_t differentiator=0)
	Returns a randomly generated file name with the given template. More...
bool	foedus::fs::fsync (const Path &path, bool sync_parent_directory=false)
	Makes the content and metadata of the file durable all the way up to devices. More...
bool	foedus::fs::atomic_rename (const Path &old_path, const Path &new_path)
	Renames the old file to the new file with the POSIX atomic-rename semantics. More...
bool	foedus::fs::durable_atomic_rename (const Path &old_path, const Path &new_path)
	fsync() on source file before rename, then fsync() on the parent folder after rename. More...
bool	foedus::fs::rename (const Path &old_path, const Path &new_path)
	Just a synonym of atomic_rename() to avoid confusion. More...

Typedef Documentation

typedef assorted::FixedString<508> foedus::fs::FixedPath

Represents a fixed (thus can be placed in shared memory) path string.

Definition at line 83 of file filesystem.hpp.

Enumeration Type Documentation

enum foedus::fs::FilePermission

Analogue of boost::filesystem::perm.

Enumerator
kNoPerms
kOwnerRead
kOwnerWrite
kOwnerExe
kOwnerAll
kGroupRead
kGroupWrite
kGroupExe
kGroupAll
kOthersRead
kOthersWrite
kOthersExe
kOthersAll
kAllAll
kPermsNotKnown

Definition at line 51 of file filesystem.hpp.

                    {
  kNoPerms = 0,       // kFileNotFound is kNoPerms rather than kPermsNotKnown

  // POSIX equivalent macros given in comments.
  // Values are from POSIX and are given in octal per the POSIX standard.

  // permission bits

  kOwnerRead = 0400,      // S_IRUSR, Read permission, owner
  kOwnerWrite = 0200,     // S_IWUSR, Write permission, owner
  kOwnerExe = 0100,       // S_IXUSR, Execute/search permission, owner
  kOwnerAll = 0700,       // S_IRWXU, Read, write, execute/search by owner

  kGroupRead = 040,       // S_IRGRP, Read permission, group
  kGroupWrite = 020,      // S_IWGRP, Write permission, group
  kGroupExe = 010,        // S_IXGRP, Execute/search permission, group
  kGroupAll = 070,        // S_IRWXG, Read, write, execute/search by group

  kOthersRead = 04,       // S_IROTH, Read permission, others
  kOthersWrite = 02,      // S_IWOTH, Write permission, others
  kOthersExe = 01,        // S_IXOTH, Execute/search permission, others
  kOthersAll = 07,        // S_IRWXO, Read, write, execute/search by others

  kAllAll = kOwnerAll|kGroupAll|kOthersAll,  // 0777

  kPermsNotKnown = 0xFFFF,   // present when directory_entry cache not loaded
};

enum foedus::fs::FileType

Analogue of boost::filesystem::file_type.

Enumerator
kStatusError
kFileNotFound
kRegularFile
kDirectoryFile
kTypeUnknown

Definition at line 38 of file filesystem.hpp.

              {
  kStatusError = 0,
  kFileNotFound,
  kRegularFile,
  kDirectoryFile,
  kTypeUnknown,
};

Function Documentation

Path foedus::fs::absolute

(

const std::string &

p

)

Returns the absolue path of the specified path.

Definition at line 62 of file filesystem.cpp.

                                  {
  return Path(p);
}

bool foedus::fs::atomic_rename	(	const Path &	old_path,
		const Path &	new_path
	)

Renames the old file to the new file with the POSIX atomic-rename semantics.

Parameters

[in]	old_path	path of the file to rename
[in]	new_path	path after rename

Returns

whether the rename succeeded

Precondition

exists(old_path)

!exists(new_path) is NOT a pre-condition. See below.

This is analogus to boost::filesystem::rename(), but we named this atomic_rename to clarify that this implementation guarantees the POSIX atomic-rename semantics. When new_path already exists, this method atomically swaps the file on filesystem with the old_path file, appropriately deleting the old file. This is an essential semantics to achieve safe and fault-tolerant file writes. And, for that usecase, do NOT forget to also call fsync before/after rename, too. Use durable_atomic_rename() to make sure.

See also

http://pubs.opengroup.org/onlinepubs/009695399/functions/rename.html

Eat My Data: How Everybody Gets File IO Wrong: https://www.flamingspork.com/talks/2007/06/eat_my_data.odp

Definition at line 199 of file filesystem.cpp.

References foedus::fs::Path::c_str(), and foedus::fs::rename().

Referenced by foedus::fs::durable_atomic_rename(), and foedus::fs::rename().

                                                               {
  return ::rename(old_path.c_str(), new_path.c_str()) == 0;
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool foedus::fs::create_directories	(	const Path &	p,
		bool	sync = false
	)

Recursive mkdir (mkdirs).

Parameters

[in]	p	path of the directory to create
[in]	sync	(optional, default false) wheter to call fsync() on the created directories and their parents. This is required to make sure the new directory entries become durable.

Returns

whether the directory already exists or creation succeeded

Definition at line 89 of file filesystem.cpp.

References foedus::fs::create_directory(), foedus::fs::Path::empty(), foedus::fs::exists(), and foedus::fs::Path::parent_path().

Referenced by foedus::log::MetaLogger::initialize_once(), foedus::log::LogManagerPimpl::initialize_once(), foedus::fs::DirectIoFile::open(), foedus::externalize::Externalizable::save_to_file(), and foedus::snapshot::SnapshotManagerPimpl::snapshot_metadata().

                                                  {
  if (exists(p)) {
    return true;
  }
  if (create_directory(p, sync)) {
    return true;
  }
  // if failed, create parent then try again
  Path parent = p.parent_path();
  if (parent.empty()) {
    return false;
  }
  if (!create_directories(parent, sync) && !exists(parent)) {
    return false;
  }
  // now ancestors exist.
  return create_directory(p, sync);
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool foedus::fs::create_directory	(	const Path &	p,
		bool	sync = false
	)

mkdir.

Parameters

[in]	p	path of the directory to create
[in]	sync	(optional, default false) wheter to call fsync() on the created directory and its parent. This is required to make sure the new directory entry becomes durable.

Returns

whether the directory already exists or creation whether succeeded

Definition at line 108 of file filesystem.cpp.

References foedus::fs::Path::c_str(), and foedus::fs::fsync().

Referenced by foedus::fs::create_directories().

                                                {
  int ret = ::mkdir(p.c_str(), S_IRWXU);
  if (ret != 0) {
    return false;
  }
  if (sync) {
    return fsync(p, true);
  } else {
    return true;
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

Path foedus::fs::current_path

(

)

Returns the current working directory.

Definition at line 66 of file filesystem.cpp.

References ASSERT_ND.

Referenced by foedus::fs::Path::Path().

                    {
  Path cur;
  for (size_t path_max = 128;; path_max *=2) {  // loop 'til buffer large enough
    std::vector<char> buf(path_max);
    if (::getcwd(&buf[0], path_max) == 0) {
      ASSERT_ND(errno == ERANGE);
    } else {
      cur = std::string(&buf[0]);
      break;
    }
  }
  return cur;
}

Here is the caller graph for this function:

bool foedus::fs::durable_atomic_rename	(	const Path &	old_path,
		const Path &	new_path
	)

fsync() on source file before rename, then fsync() on the parent folder after rename.

This method makes 2 fsync calls, one on old file before rename and another on the parent directory after rename.

Note that we don't need fsync on parent directory before rename assuming old_path and new_path is in the same folder (if not, you have to call fsync yourself before calling this method). Even if a crash happens right after rename, we still see the old content of new_path.

Also, we don't need fsync on new_path after rename because POSIX rename doesn't change the inode of renamed file. It's already there as soon as parent folder's fsync is done.

Quite complex and expensive, but this is required to make it durable regardless of filesystems. Fortunately, we have to call this method only once per epoch-advance.

Definition at line 236 of file filesystem.cpp.

References foedus::fs::atomic_rename(), foedus::fs::fsync(), and foedus::fs::Path::parent_path().

Referenced by foedus::externalize::Externalizable::save_to_file().

                                                                       {
  if (!fsync(old_path, false)) {
    return false;
  }
  if (!atomic_rename(old_path, new_path)) {
    return false;
  }
  return fsync(new_path.parent_path(), false);
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool foedus::fs::exists ( const Path &  p )

inline

Returns if the file exists.

Definition at line 128 of file filesystem.hpp.

References foedus::fs::FileStatus::exists(), and foedus::fs::status().

Referenced by foedus::memory::SharedMemory::alloc(), foedus::memory::SharedMemory::attach(), foedus::fs::create_directories(), foedus::log::MetaLogger::initialize_once(), foedus::log::LogManagerPimpl::initialize_once(), foedus::savepoint::SavepointManagerPimpl::initialize_once(), foedus::externalize::Externalizable::load_from_file(), foedus::fs::DirectIoFile::open(), foedus::storage::MetadataSerializer::save_all_storages_to_xml(), foedus::externalize::Externalizable::save_to_file(), and foedus::snapshot::SnapshotManagerPimpl::snapshot_metadata().

{return status(p).exists(); }

Here is the call graph for this function:

Here is the caller graph for this function:

uint64_t foedus::fs::file_size

(

const Path &

p

)

Returns size of the file.

Definition at line 120 of file filesystem.cpp.

References foedus::fs::Path::c_str().

Referenced by foedus::snapshot::LogMapper::handle_process(), foedus::snapshot::SnapshotWriter::open(), foedus::fs::DirectIoFile::open(), foedus::snapshot::SnapshotManagerPimpl::read_snapshot_metadata(), foedus::restart::RestartManagerPimpl::redo_meta_logs(), and foedus::snapshot::SnapshotManagerPimpl::snapshot_metadata().

                                  {
  struct stat path_stat;
  int ret = ::stat(p.c_str(), &path_stat);
  if (ret != 0) {
    return static_cast<uint64_t>(-1);
  }
  if (!S_ISREG(path_stat.st_mode)) {
    return static_cast<uint64_t>(-1);
  }
  return static_cast<uint64_t>(path_stat.st_size);
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool foedus::fs::fsync	(	const Path &	path,
		bool	sync_parent_directory = false
	)

Makes the content and metadata of the file durable all the way up to devices.

Parameters

[in]	path	path of the file to make durable
[in]	sync_parent_directory	(optional, default false) whether to also call fsync on the parent directory to make sure the directory entry is written to device. This is required when you create a new file, rename, etc.

Returns

whether the sync succeeded or not. If failed, check the errno global variable (set by lower-level library).

Surprisingly, there is no analogus method in boost::filesystem. This method provides the fundamental building block of fault-tolerant systems; fsync. We so far don't provide fdatasync (no metadata sync), but this should suffice.

Definition at line 203 of file filesystem.cpp.

References foedus::fs::Path::c_str(), foedus::fs::is_directory(), and foedus::fs::Path::parent_path().

Referenced by foedus::snapshot::SnapshotWriter::close(), foedus::fs::create_directory(), foedus::fs::durable_atomic_rename(), foedus::externalize::Externalizable::save_to_file(), foedus::snapshot::SnapshotManagerPimpl::snapshot_metadata(), foedus::fs::DirectIoFile::sync(), and foedus::fs::DirectIoFile::truncate().

                                                         {
  int sync_ret;
  if (!is_directory(path)) {
    int descriptor = ::open(path.c_str(), O_RDONLY);
    if (descriptor < 0) {
      return false;
    }
    sync_ret = ::fsync(descriptor);
    ::close(descriptor);
  } else {
    DIR *dir = ::opendir(path.c_str());
    if (!dir) {
      return false;
    }
    int descriptor = ::dirfd(dir);
    if (descriptor < 0) {
      return false;
    }

    sync_ret = ::fsync(descriptor);
    ::closedir(dir);
  }

  if (sync_ret != 0) {
    return false;
  }

  if (sync_parent_directory) {
    return fsync(path.parent_path(), false);
  } else {
    return true;
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

Path foedus::fs::home_path

(

)

Returns the absolute path of the home directory of the user running this process.

So far this checks only HOME environment variable, which might not be set in some environment. In that case, this returns an empty path. A truly crossplatform home_path is not in standard C++, unfortunately.

See also

http://stackoverflow.com/questions/2552416/how-can-i-find-the-users-home-dir-in-a-cross-platform-manner-using-c

Definition at line 80 of file filesystem.cpp.

Referenced by foedus::fs::Path::Path().

                 {
  const char *home = ::getenv("HOME");
  if (home) {
    return Path(home);
  } else {
    return Path();
  }
}

Here is the caller graph for this function:

bool foedus::fs::is_directory ( const Path &  p )

inline

Returns if the file is a directory.

Definition at line 133 of file filesystem.hpp.

References foedus::fs::FileStatus::is_directory(), and foedus::fs::status().

Referenced by foedus::fs::Path::child_paths(), and foedus::fs::fsync().

{return status(p).is_directory(); }

Here is the call graph for this function:

Here is the caller graph for this function:

bool foedus::fs::is_regular_file ( const Path &  p )

inline

Returns if the file is a regular file.

Definition at line 138 of file filesystem.hpp.

References foedus::fs::FileStatus::is_regular_file(), and foedus::fs::status().

{return status(p).is_regular_file(); }

Here is the call graph for this function:

bool foedus::fs::remove

(

const Path &

p

)

Deletes a regular file or an empty directory.

Returns

whether succeeded

Definition at line 132 of file filesystem.cpp.

References foedus::fs::FileStatus::exists(), foedus::fs::FileStatus::is_regular_file(), and foedus::fs::status().

Referenced by foedus::memory::SharedMemory::release_block().

                           {
  FileStatus s = status(p);
  if (!s.exists()) {
    return 0;
  } else if (s.is_regular_file()) {
    return std::remove(p.c_str()) == 0;
  } else {
    int ret = ::rmdir(p.c_str());
    if (ret == 0) {
      return true;
    } else {
      return false;
    }
  }
}

Here is the call graph for this function:

Here is the caller graph for this function:

uint64_t foedus::fs::remove_all

(

const Path &

p

)

Recursively deletes a directory.

Returns

number of files/directories deleted.

Definition at line 148 of file filesystem.cpp.

References foedus::fs::Path::child_paths().

                                   {
  uint64_t count = 1;
  std::vector< Path > child_paths = p.child_paths();
  for (Path child : child_paths) {
    count += remove_all(child);
  }
  remove(p);
  return count;
}

Here is the call graph for this function:

bool foedus::fs::rename ( const Path &  old_path, const Path &  new_path  )

inline

Just a synonym of atomic_rename() to avoid confusion.

Definition at line 286 of file filesystem.hpp.

References foedus::fs::atomic_rename().

Referenced by foedus::fs::atomic_rename().

                                                               {
  return atomic_rename(old_path, new_path);
}

Here is the call graph for this function:

Here is the caller graph for this function:

SpaceInfo foedus::fs::space

(

const Path &

p

)

Returns free space information for the device the file is on.

Definition at line 158 of file filesystem.cpp.

References foedus::fs::Path::c_str(), and foedus::fs::SpaceInfo::capacity_.

                               {
  struct statfs vfs;
  SpaceInfo info;
  int ret = ::statfs(p.c_str(), &vfs);
  if (ret == 0) {
    info.capacity_    = static_cast<uint64_t>(vfs.f_blocks) * vfs.f_bsize;
    info.free_        = static_cast<uint64_t>(vfs.f_bfree)  * vfs.f_bsize;
    info.available_   = static_cast<uint64_t>(vfs.f_bavail) * vfs.f_bsize;
  } else {
    info.available_ = 0;
    info.capacity_ = 0;
    info.free_ = 0;
  }
  return info;
}

Here is the call graph for this function:

FileStatus foedus::fs::status

(

const Path &

p

)

Returns the status of the file.

Definition at line 45 of file filesystem.cpp.

References foedus::fs::Path::c_str(), foedus::fs::kDirectoryFile, foedus::fs::kFileNotFound, foedus::fs::kRegularFile, foedus::fs::kStatusError, and foedus::fs::kTypeUnknown.

Referenced by foedus::storage::sequential::SequentialComposer::compose(), foedus::assorted::demangle_type_name(), foedus::fs::exists(), foedus::snapshot::LogMapper::handle_process(), foedus::soc::SocManagerPimpl::initialize_master(), foedus::EnginePimpl::initialize_once(), foedus::snapshot::MergeSort::initialize_once(), foedus::fs::is_directory(), foedus::fs::is_regular_file(), foedus::fs::remove(), foedus::soc::SocManagerPimpl::report_engine_fatal_error(), foedus::soc::SocManagerPimpl::wait_for_child_attach(), foedus::soc::SocManagerPimpl::wait_for_child_terminate(), foedus::soc::SocManagerPimpl::wait_for_children_module(), and foedus::soc::SocManagerPimpl::wait_for_master_module().

                                 {
  struct stat path_stat;
  int ret = ::stat(p.c_str(), &path_stat);
  if (ret != 0) {
    // This is quite normal as we use this to check if a file exists. No message for it.
    if (errno == ENOENT || errno == ENOTDIR) {
      return FileStatus(kFileNotFound);
    }
    return FileStatus(kStatusError);
  } else if (S_ISDIR(path_stat.st_mode)) {
    return FileStatus(kDirectoryFile);
  } else if (S_ISREG(path_stat.st_mode)) {
    return FileStatus(kRegularFile);
  }
  return FileStatus(kTypeUnknown);
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::string foedus::fs::unique_name

(

uint64_t

differentiator = 0

)

Equivalent to unique_path("%%%%-%%%%-%%%%-%%%%").

Definition at line 174 of file filesystem.cpp.

Referenced by foedus::externalize::Externalizable::save_to_file().

                                               {
  return unique_name("%%%%-%%%%-%%%%-%%%%", differentiator);
}

Here is the caller graph for this function:

std::string foedus::fs::unique_name	(	const std::string &	model,
		uint64_t	differentiator = 0
	)

Returns a randomly generated file name with the given template.

Parameters

[in]	model	file name template where % will be replaced with random hex numbers.
[in]	differentiator	optional parameter to further randomize this method.

We use std::chrono::high_resolution_clock::now() to get a random seed. However, even high_resolution_clock sometimes has low precision depending on environment. When you are concerned with a conflict (eg running many concurrent testcases), also give a differentiator.

Definition at line 177 of file filesystem.cpp.

                                                                       {
  const char* kHexChars = "0123456789abcdef";
  uint64_t seed64 = std::chrono::high_resolution_clock::now().time_since_epoch().count();
  seed64 += ::getpid();  // further use process ID to randomize. may help.
  seed64 ^= differentiator;
  uint32_t seed32 = (seed64 >> 32) ^ seed64;
  std::string s(model);
  for (size_t i = 0; i < s.size(); ++i) {
    if (s[i] == '%') {                 // digit request
      seed32 = ::rand_r(&seed32);
      s[i] = kHexChars[seed32 & 0xf];  // convert to hex digit and replace
    }
  }
  return s;
}