Asuran Internals

Chunks are a data structure containing a encrypted and compressed data slice. They also contain tags defining the encryption type and IV, the type of compression used, as well as the HMAC and ID tags.

Delta chunks are a data structure that contain encrypted, compressed data intended to be used as a patch to another chunk, to be used for delta-compression with the rsync algorithm. Like regular chunks, they contain data, an HMAC tag, an encryption type/IV tag, and have their own chunk ID. Unlike regular chunks, they can not be directly unpacked in any meaningful sense, and thus also contain a reference to another chunk (by its ChunkID) whose data it patches.

The majority of all interactions with the repository will occur through the normal transactions.

Some operations on a repository are non transactions, mainly maintenance operations.

Each flatfile repository has a header which is a compact message-pack representation of the following struct:

/// The header used by flatfile repositories
#[derive(Serialize, Deserialize, Debug)]
pub struct Header {
    /// The magic number, ASCII 'ASURAN_F'
    pub magic_number: [u8; 8],
    /// The UUID of the implemenation that wrote this flatfile
    pub implementation_uuid: [u8; 16],
    pub semver_major: u16,
    pub semver_minor: u16,
    pub semver_patch: u16,
    pub configuration: Configuration,
}

The transactions are encoded with the following enum. Chunks are included in the transaction that define them.

/// Represents the various types of transaction entries in a flatfile repository
#[derive(Serialize, Deserialize)]
pub enum FlatFileTransaction {
    /// An insertion of a data chunk into the repository
    Insert { id: ChunkID, chunk: Chunk },
    /// A deletion of a data chunk from the repository
    Delete { id: ChunkID },
    /// An insertion of an `Archive` into the repository
    ///
    /// The Serialized `Archive` is stored as a `Chunk` containing the serialized Arhive
    ManifestInsert {
	id: ChunkID,
	name: String,
	timestamp: DateTime<FixedOffset>,
    },
}

Manifest insertion transactions indicate that the following chunk contains an archive entry within the manifest, see the section on the flatfile manifest.

Not having seperate files to store things in means the flatfile repository does not have a manifest as such. Instead the manifest is reconstructed during the scan through phase of repository loading based on the Manifest Insertion transactions.

The timestamp of last modification is reconstructed by using the timestamp in the last archive.

Chunk Settings are reconstructed by using the settings of the first chunk inserted in the repository.

Segments are stored in numbered subfolders of the main repository folder. The maximum number of segments per folder, as well as the maximum size of a segment, are user configurable.

Folder names are the number of the lowest segment they contain.

Segment numbers increment from 0, with the lowest unassigned number being used when a new one needs to be created.

Segment files contain a binary format header, followed by the concatenation of the encoded chunks.

The index is a collection of numbered log files, stored in a sub folder of the repository called “index”, logging insertions, but only keeping track of the location of the chunks⁴. Deletions are not processed in the transaction, and the only way to remove an object from the index is when the index is recreated during a

On start up, Asuran loads all the available index files and parses through them, replaying history to build the in-memory index.

A manifest is simply a list of pointers to archive objects within the repository, accompanied by a set of default chunk settings and a last modified date. The archive pointers are stored in a collection of numbered files in the manifest subfolder of the repository, in a much similar manner to the index. Reads are not synchronized, but writes are, in a manner identical to that used by the index.

The key material for the repository is stored in a file called “key” in the root folder of the repository. It is encrypted using a key derived from a user supplied passphrase via argon2id. This is stored in a compact format messagepacked Asuran::repository::EncryptedKey struct, which has the following fields:

/// Stores the key, encrypted with another key dervied from the user specified
/// password/passphrase
///
/// Uses argon2 to derive the key encryption key from the user supplied key.
///
/// Uses a 32 byte salt that is randomly generated
///
/// Currently uses semi-arbitrary defaults for some values. TODO: allow configuration of this
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct EncryptedKey {
    encrypted_bytes: Vec<u8>,
    salt: [u8; 32],
    mem_cost: u32,
    time_cost: u32,
    encryption: Encryption,
}

Namespaces take the form of colon delimited lists of strings, sorted in order of least specific to most specific. The strings may take any value, and the meaning is prescribed by the target implementation, but they may only contain alphanumeric characters.

A terminating double colon is append to the end of the namespace specifier. This marks the end of the namespace portion of the specifier, and any data after it will be treated as part of the path.

The default/root namespace is simply “::”

Path specifiers are posix style paths, with no additional restrictions imposed.

ChunkLocation is a struct, serialized in compact messagepack format, with the following fields:

/// A pointer to a `Chunk`, annotated with information on what part of the object it
/// makes up
#[derive(Serialize, Deserialize, Copy, Clone, Eq, PartialEq, Debug)]
pub struct ChunkLocation {
    pub id: ChunkID,
    pub start: u64,
    pub length: u64,
}