From 122cbbea61564e477d84c9e53b35a0cd63248811 Mon Sep 17 00:00:00 2001 From: Arlo Godfrey Date: Sun, 28 Apr 2024 03:05:19 -0500 Subject: [PATCH 1/3] Release 0.16.0. See release notes. [Breaking] Inherited breaking changes from IpfsShipyard.Net.Http.Client 0.2.0. Inherited breaking changes from OwlCore.Storage 0.10.0 and 0.11.0. Inherited breaking changes from OwlCore.ComponentModel 0.7.0 and 0.8.0. Inherited breaking changes from OwlCore.Extensions 0.8.0. [Fixes] The changes in OwlCore.Storage 0.10.0 allowed the CommonTests to uncover previously unknown issues with the Kubo-based storage implementation. These issues are now fixed and the storage implementations are compliant with the latest CommonTests. [New] Added static SwarmKeyGen extension methods, which allow you to generate and write a private Kubo swarm key to a file for immediate use. Added a new PrivateKuboBootstrapper. This custom bootstrapper allows you to start a Kubo node with a private swarm key, automatically removing the default bootstrap nodes, applying LIBP2P_FORCE_PNET as needed, and setting up the provided BootstrapPeerMultiAddresses as your bootstrap peers. Added a new OwlCore.Kubo.Cache namespace. This is a limited set of API wrappers for the Ipfs core interfaces that enable caching functionality throughout your entire application domain, currently covering IKeyApi and INameApi. Note that you'll need to use ICoreApi instead of IpfsClient, and likewise for other Core interfaces (and their implementations) throughout your codebase to use this cache layer. Added TransformIpnsDagAsync extension method, which allows you to mutate and update a DAG object published to IPNS all in one go, with progress reporting. Added ResolveDagCidAsync extension method to Cid and IEnumerable{Cid}. Resolves the provided cid if it is an Ipns address and retrieves the content from the DAG. Added CreateKeyWithNameOfIdAsync extension method to IKeyApi. Creates an ipns key using a temp name, then renames it to the name of the key. Enables pushing to ipns without additional API calls to convert between ipns cid and name. Added GetOrCreateKeyAsync extension method to IKeyApi. Gets a key by name, or creates it if it does not exist. Added various helper methods to KuboBootstrapper for getting repo lock state, gateway and api uri, and converting between MultiAddress and Uri. Added all missing DhtRoutingMode values that have been added to Kubo as of 0.26.0. Added a LaunchConfigMode to KuboBootstrapper. Defines the behavior when a node is already running (when the repo is locked): Throw, Attach, or Relaunch. Added an ApiUriMode to KuboBootstrapper. Gets or sets an enum that determines how to use the supplied ApiUri: UseExisting, or OverwriteExisting. Added a GatewayUriMode to KuboBootstrapper. Gets or sets an enum that determines how to use the supplied GatewayUri: UseExisting, or OverwriteExisting. Added a UseAcceleratedDHTClient property to KuboBootstrapper. If set to true, the client DHT will be used on startup. [Improvements] As part of the move to ICoreApi, all internal calls to DoCommandAsync have been removed wherever IMfsApi is used, thanks to the implementation contributed in https://github.com/ipfs-shipyard/net-ipfs-core/pull/13/. Updated dependencies. --- src/AesPasswordEncryptedPubSub.cs | 12 +- src/BootstrapLaunchConflictMode.cs | 23 + src/Cache/CachedCoreApi.cs | 114 +++++ src/Cache/CachedKeyApi.cs | 85 ++++ src/Cache/CachedNameApi.cs | 205 ++++++++ src/Cache/KuboCacheSerializer.cs | 80 ++++ src/Cache/KuboCacheSerializerContext.cs | 16 + src/ConfigMode.cs | 17 + src/DhtRoutingMode.cs | 35 +- src/Downloader/KuboDownloader.cs | 32 +- src/Extensions/GenericKuboExtensions.cs | 157 +++++- src/Extensions/IGetCid.cs | 1 - src/Extensions/IpnsDagExtensions.cs | 92 ++++ .../ContentAddressedSystemFile.cs | 12 +- .../ContentAddressedSystemFolder.cs | 12 +- src/Extensions/StorableKuboExtensions.cs | 7 +- src/FolderWatchers/TimerBasedFolderWatcher.cs | 10 +- src/FolderWatchers/TimerBasedIpnsWatcher.cs | 12 +- src/FolderWatchers/TimerBasedMfsWatcher.cs | 15 +- src/IpfsFile.cs | 18 +- src/IpfsFolder.cs | 132 +++--- src/IpnsFile.cs | 14 +- src/IpnsFolder.cs | 37 +- src/KuboBootstrapper.cs | 446 +++++++++++++++--- src/MfsFile.cs | 33 +- src/MfsFolder.Modifiable.cs | 164 ++++--- src/MfsFolder.cs | 211 ++++----- src/MfsStream.cs | 21 +- src/Models/FilesFlushResponse.cs | 11 +- src/Models/MfsFileContentsBody.cs | 11 +- src/Models/MfsFileData.cs | 17 +- src/Models/MfsFileStatData.cs | 27 +- src/Models/ModelSerializer.cs | 19 +- src/Models/PublishedMessage.cs | 4 +- src/OwlCore.Kubo.csproj | 53 ++- src/PathHelpers.cs | 67 ++- src/PeerRoom.cs | 10 +- src/Polyfills/DoesNotReturnAttribute.cs | 17 - src/Polyfills/Index.cs | 155 ------ src/Polyfills/IsExternalInit.cs | 18 - src/Polyfills/NotNullWhenAttribute.cs | 28 -- src/PrivateKuboBootstrapper.cs | 82 ++++ src/SwarmKeyGen.cs | 55 +++ tests/OwlCore.Kubo.Tests/IpfsFileTests.cs | 8 +- tests/OwlCore.Kubo.Tests/IpnsFolderTests.cs | 5 +- .../KuboBootstrapperTests.cs | 4 +- tests/OwlCore.Kubo.Tests/LoopbackPubSubApi.cs | 35 +- tests/OwlCore.Kubo.Tests/MfsFolderTests.cs | 8 +- tests/OwlCore.Kubo.Tests/MfsStreamTests.cs | 8 +- .../OwlCore.Kubo.Tests.csproj | 8 +- tests/OwlCore.Kubo.Tests/TestFixture.cs | 5 +- 51 files changed, 1891 insertions(+), 777 deletions(-) create mode 100644 src/BootstrapLaunchConflictMode.cs create mode 100644 src/Cache/CachedCoreApi.cs create mode 100644 src/Cache/CachedKeyApi.cs create mode 100644 src/Cache/CachedNameApi.cs create mode 100644 src/Cache/KuboCacheSerializer.cs create mode 100644 src/Cache/KuboCacheSerializerContext.cs create mode 100644 src/ConfigMode.cs create mode 100644 src/Extensions/IpnsDagExtensions.cs rename src/Extensions/{OwlCore.Storage.SystemIO => OwlCore.Storage.System.IO}/ContentAddressedSystemFile.cs (67%) rename src/Extensions/{OwlCore.Storage.SystemIO => OwlCore.Storage.System.IO}/ContentAddressedSystemFolder.cs (67%) delete mode 100644 src/Polyfills/DoesNotReturnAttribute.cs delete mode 100644 src/Polyfills/Index.cs delete mode 100644 src/Polyfills/IsExternalInit.cs delete mode 100644 src/Polyfills/NotNullWhenAttribute.cs create mode 100644 src/PrivateKuboBootstrapper.cs create mode 100644 src/SwarmKeyGen.cs diff --git a/src/AesPasswordEncryptedPubSub.cs b/src/AesPasswordEncryptedPubSub.cs index 184bce9..228cc30 100644 --- a/src/AesPasswordEncryptedPubSub.cs +++ b/src/AesPasswordEncryptedPubSub.cs @@ -4,6 +4,7 @@ using OwlCore.Extensions; using System.Security.Cryptography; using System.Text; +using MemoryStream = System.IO.MemoryStream; using PublishedMessage = OwlCore.Kubo.Models.PublishedMessage; namespace OwlCore.Kubo; @@ -48,7 +49,7 @@ public async Task PublishAsync(string topic, Stream message, CancellationToken c message.Seek(0, SeekOrigin.Begin); var aes = Aes.Create(); - var passBytes = new Rfc2898DeriveBytes(password: _password, salt: Encoding.UTF8.GetBytes(_salt ?? string.Empty)); + var passBytes = new Rfc2898DeriveBytes(password: _password, salt: Encoding.UTF8.GetBytes(_salt ?? string.Empty), iterations: 2000); aes.Key = passBytes.GetBytes(aes.KeySize / 8); aes.IV = passBytes.GetBytes(aes.BlockSize / 8); @@ -56,8 +57,8 @@ public async Task PublishAsync(string topic, Stream message, CancellationToken c using var encryptedOutputStream = new MemoryStream(); using var streamEncryptor = new CryptoStream(encryptedOutputStream, aes.CreateEncryptor(), CryptoStreamMode.Write); - var unencryptedBytes = await message.ToBytesAsync(); - streamEncryptor.Write(unencryptedBytes, 0, unencryptedBytes.Length); + var unencryptedBytes = await message.ToBytesAsync(cancellationToken: cancel); + await streamEncryptor.WriteAsync(unencryptedBytes, 0, unencryptedBytes.Length, cancel); streamEncryptor.FlushFinalBlock(); encryptedOutputStream.Position = 0; @@ -70,7 +71,7 @@ public Task SubscribeAsync(string topic, Action handler, Canc { return Inner.SubscribeAsync(topic, msg => { - if (TryTransformPublishedMessage(msg) is IPublishedMessage transformedMsg) + if (TryTransformPublishedMessage(msg) is { } transformedMsg) { handler(transformedMsg); } @@ -94,7 +95,8 @@ public Task> SubscribedTopicsAsync(CancellationToken cancel try { using var outputStream = new MemoryStream(); - using var streamDecryptor = new CryptoStream(publishedMessage.DataStream, aes.CreateDecryptor(), CryptoStreamMode.Read); + using var inputStream = new MemoryStream(publishedMessage.DataBytes); + using var streamDecryptor = new CryptoStream(inputStream, aes.CreateDecryptor(), CryptoStreamMode.Read); streamDecryptor.CopyTo(outputStream); var outputBytes = outputStream.ToBytes(); diff --git a/src/BootstrapLaunchConflictMode.cs b/src/BootstrapLaunchConflictMode.cs new file mode 100644 index 0000000..12a75e2 --- /dev/null +++ b/src/BootstrapLaunchConflictMode.cs @@ -0,0 +1,23 @@ +namespace OwlCore.Kubo; + +/// +/// The behavior when the repo is locked and a node is already running on the requested port. +/// +public enum BootstrapLaunchConflictMode +{ + /// + /// Throw when the requested port is already in use. + /// + Throw, + + /// + /// If the repo is locked, load the Api and Gateway port, attach to it and shut it down, then bootstrap a new process on the requested ports. + /// + Relaunch, + + /// + /// If the repo is locked, load the configured (currently running) Api and Gateway port and attach to it without shutting it down or starting a new process. May discard the requested ports. + /// + /// Note that attaching does not allow you to take control of the running process. See instead if you need this functionality. + Attach, +} \ No newline at end of file diff --git a/src/Cache/CachedCoreApi.cs b/src/Cache/CachedCoreApi.cs new file mode 100644 index 0000000..c6d9487 --- /dev/null +++ b/src/Cache/CachedCoreApi.cs @@ -0,0 +1,114 @@ +using Ipfs.CoreApi; +using OwlCore.ComponentModel; +using OwlCore.Storage; + +namespace OwlCore.Kubo.Cache; + +/// +/// A cached api layer for . +/// +public class CachedCoreApi : ICoreApi, IDelegable, IFlushable, IAsyncInit +{ + /// + /// Creates a new instance of . + /// + /// The folder to store cached data in. + /// The inner to wrap around. + public CachedCoreApi(IModifiableFolder cacheFolder, ICoreApi inner) + { + Name = new CachedNameApi(cacheFolder) { Inner = inner.Name, KeyApi = inner.Key }; + Key = new CachedKeyApi(cacheFolder) { Inner = inner.Key }; + Inner = inner; + } + + /// + /// The inner, unwrapped core api to delegate to. + /// + public ICoreApi Inner { get; } + + /// + public IBitswapApi Bitswap => Inner.Bitswap; + + /// + public IBlockApi Block => Inner.Block; + + /// + public IBlockRepositoryApi BlockRepository => Inner.BlockRepository; + + /// + public IBootstrapApi Bootstrap => Inner.Bootstrap; + + /// + public IConfigApi Config => Inner.Config; + + /// + public IDagApi Dag => Inner.Dag; + + /// + public IDhtApi Dht => Inner.Dht; + + /// + public IDnsApi Dns => Inner.Dns; + + /// + public IFileSystemApi FileSystem => Inner.FileSystem; + + /// + public IMfsApi Mfs => Inner.Mfs; + + /// + public IGenericApi Generic => Inner.Generic; + + /// + public IKeyApi Key { get; } + + /// + public INameApi Name { get; } + + /// + public IObjectApi Object => Inner.Object; + + /// + public IPinApi Pin => Inner.Pin; + + /// + public IPubSubApi PubSub => Inner.PubSub; + + /// + public IStatsApi Stats => Inner.Stats; + + /// + public ISwarmApi Swarm => Inner.Swarm; + + /// + public bool IsInitialized { get; set; } + + /// + public async Task FlushAsync(CancellationToken cancellationToken) + { + await ((CachedNameApi)Name).FlushAsync(cancellationToken); + + await ((CachedNameApi)Name).SaveAsync(cancellationToken); + await ((CachedKeyApi)Key).SaveAsync(cancellationToken); + } + + /// + public async Task InitAsync(CancellationToken cancellationToken = default) + { + try + { + // Try loading data from API + await ((CachedKeyApi)Key).InitAsync(cancellationToken); + } + catch + { + // Load data from disk as fallback + await ((CachedKeyApi)Key).LoadAsync(cancellationToken); + } + + await ((CachedNameApi)Name).LoadAsync(cancellationToken); + + // Allow multiple initialization + IsInitialized = true; + } +} diff --git a/src/Cache/CachedKeyApi.cs b/src/Cache/CachedKeyApi.cs new file mode 100644 index 0000000..3738220 --- /dev/null +++ b/src/Cache/CachedKeyApi.cs @@ -0,0 +1,85 @@ +using Ipfs; +using Ipfs.CoreApi; +using OwlCore.ComponentModel; +using OwlCore.Storage; + +namespace OwlCore.Kubo.Cache; + +/// +/// A cached api layer for . +/// +public class CachedKeyApi : SettingsBase, IKeyApi, IDelegable, IAsyncInit +{ + /// + /// The cached record type for created or resolved s. + /// + public record KeyInfo(string Name, Cid Id) : IKey; + + /// + /// Creates a new instance of . + /// + /// The folder to store cached name resolutions. + public CachedKeyApi(IModifiableFolder folder) + : base(folder, KuboCacheSerializer.Singleton) + { + FlushDefaultValues = false; + } + + /// + /// The resolved ipns keys. + /// + public List Keys + { + get => GetSetting(() => new List()); + set => SetSetting(value); + } + + /// + public required IKeyApi Inner { get; init; } + + /// + public async Task CreateAsync(string name, string keyType, int size, CancellationToken cancel = default) + { + var res = await Inner.CreateAsync(name, keyType, size, cancel); + + var existing = Keys.FirstOrDefault(x => x.Name == res.Name); + if (existing is not null) + Keys.Remove(existing); + + Keys.Add(new KeyInfo(Name: res.Name, Id: res.Id)); + return res; + } + + /// + public Task> ListAsync(CancellationToken cancel = default) => Task.FromResult>(Keys); + + /// + public Task RemoveAsync(string name, CancellationToken cancel = default) => Inner.RemoveAsync(name, cancel); + + /// + public Task RenameAsync(string oldName, string newName, CancellationToken cancel = default) => Inner.RenameAsync(oldName, newName, cancel); + + /// + public Task ExportAsync(string name, char[] password, CancellationToken cancel = default) => Inner.ExportAsync(name, password, cancel); + + /// + public Task ImportAsync(string name, string pem, char[]? password = null, CancellationToken cancel = default) => Inner.ImportAsync(name, pem, password, cancel); + + /// + /// Initializes local values with fresh data from the API. + /// + /// A token that can be used to cancel the ongoing operation. + public async Task InitAsync(CancellationToken cancellationToken = default) + { + var res = await Inner.ListAsync(cancellationToken); + Keys = res.Select(x => new KeyInfo(x.Name, x.Id)).ToList(); + + await SaveAsync(cancellationToken); + + // Allow multiple initialization + IsInitialized = true; + } + + /// + public bool IsInitialized { get; private set; } +} \ No newline at end of file diff --git a/src/Cache/CachedNameApi.cs b/src/Cache/CachedNameApi.cs new file mode 100644 index 0000000..1efee81 --- /dev/null +++ b/src/Cache/CachedNameApi.cs @@ -0,0 +1,205 @@ +using CommunityToolkit.Diagnostics; +using Ipfs; +using Ipfs.CoreApi; +using OwlCore.ComponentModel; +using OwlCore.Storage; + +namespace OwlCore.Kubo.Cache; + +/// +/// A cache layer for . No API calls will be made to Kubo until is called. +/// +/// +/// Recommended for code that pushes to ipns in bursts of updates, allowing you to defer the publication of the final ipns value. +/// +public class CachedNameApi : SettingsBase, INameApi, IDelegable, IFlushable +{ + /// + /// The cached record for a published path name in a . + /// + public record PublishedPathName(string path, bool resolve, string key, TimeSpan? lifetime, NamedContent returnValue); + + /// + /// The cached record for a published cid name in a . + /// + public record PublishedCidName(Cid id, string key, TimeSpan? lifetime, NamedContent returnValue); + + /// + /// The cached record for a resolved name in a . + /// + public record ResolvedName(string name, bool recursive, string returnValue); + + /// + /// Creates a new instance of . + /// + /// The folder to store cached name resolutions. + public CachedNameApi(IModifiableFolder folder) + : base(folder, KuboCacheSerializer.Singleton) + { + FlushDefaultValues = false; + } + + /// + /// The names that have been resolved. + /// + public List ResolvedNames + { + get => GetSetting(() => new List()); + set => SetSetting(value); + } + + /// + /// The latest named content that has been published via . + /// + public List PublishedCidNamedContent + { + get => GetSetting(() => new List()); + set => SetSetting(value); + } + + /// + /// The latest named content that has been published via . + /// + public List PublishedStringNamedContent + { + get => GetSetting(() => new List()); + set => SetSetting(value); + } + + /// + public required INameApi Inner { get; init; } + + /// + /// The Key API to use for getting existing keys that can be published to. + /// + public required IKeyApi KeyApi { get; init; } + + /// + /// Flushes the cached requests to the underlying resource. + /// + /// A token that can be used to cancel the ongoing operation. + public async Task FlushAsync(CancellationToken cancellationToken = default) + { + foreach (var item in PublishedCidNamedContent.ToArray()) + { + cancellationToken.ThrowIfCancellationRequested(); + + Console.WriteLine($"Flushing key {item.key} with value {item.id}"); + + // Publish to ipfs + var result = await Inner.PublishAsync(item.id, item.key, item.lifetime, cancellationToken); + + // Verify result matches original returned data + _ = Guard.Equals(result.ContentPath, item.returnValue.ContentPath); + _ = Guard.Equals(result.NamePath, item.returnValue.NamePath); + + // Update cache + PublishedCidNamedContent.Remove(item); + PublishedCidNamedContent.Add(item with { returnValue = result }); + } + + foreach (var item in PublishedStringNamedContent) + { + cancellationToken.ThrowIfCancellationRequested(); + + Console.WriteLine($"Flushing key {item.key} with value {item.path}"); + + // Publish to ipfs + var result = await Inner.PublishAsync(item.path, item.resolve, item.key, item.lifetime, cancellationToken); + + // Verify result matches original returned data + _ = Guard.Equals(result.ContentPath, item.returnValue.ContentPath); + _ = Guard.Equals(result.NamePath, item.returnValue.NamePath); + + // Update cache + PublishedStringNamedContent.Remove(item); + PublishedStringNamedContent.Add(item with { returnValue = result }); + } + } + + /// + public async Task PublishAsync(string path, bool resolve = true, string key = "self", TimeSpan? lifetime = null, CancellationToken cancel = default) + { + if (PublishedStringNamedContent.FirstOrDefault(x => x.key == key) is { } existing) + PublishedStringNamedContent.Remove(existing); + + var keys = await KeyApi.ListAsync(cancel); + var existingKey = keys.FirstOrDefault(x => x.Name == key); + var keyId = existingKey?.Id; + + NamedContent published = new() { ContentPath = path, NamePath = $"/ipns/{keyId}" }; + + PublishedStringNamedContent.Add(new(path, resolve, key, lifetime, published)); + return published; + } + + /// + public async Task PublishAsync(Cid id, string key = "self", TimeSpan? lifetime = null, CancellationToken cancel = default) + { + if (PublishedCidNamedContent.FirstOrDefault(x => x.key == key) is { } existing) + PublishedCidNamedContent.Remove(existing); + + var keys = await KeyApi.ListAsync(cancel); + var existingKey = keys.FirstOrDefault(x => x.Name == key); + var keyId = existingKey?.Id; + + NamedContent published = new() { ContentPath = $"/ipfs/{id}", NamePath = $"/ipns/{keyId}" }; + PublishedCidNamedContent.Add(new(id, key, lifetime, published)); + + return published; + } + + /// + /// + /// Using nocache = true here forces immediate name resolution via API, falling back to cache on failure. + /// + /// Using nocache = false here checks the cache first, and falls back to the API if not found. + /// + public async Task ResolveAsync(string name, bool recursive = false, bool nocache = false, CancellationToken cancel = default) + { + if (nocache) + { + try + { + // Don't resolve with cache, but still save resolved data to cache. + var resToCache = await Inner.ResolveAsync(name, recursive, nocache, cancel); + + var existing = ResolvedNames.FirstOrDefault(x => x.name == name); + if (existing is not null) + ResolvedNames.Remove(existing); + + ResolvedNames.Add(new(name, recursive, resToCache)); + + return resToCache; + } + catch + { + // request failed, continue with cache anyway + } + } + + // Check if name is in published cache + if (PublishedCidNamedContent.FirstOrDefault(x => x.returnValue.NamePath is not null && (name.Contains(x.returnValue.NamePath) || x.returnValue.NamePath.Contains(name))) is { } publishedCidNamedContent) + { + if (publishedCidNamedContent.returnValue.ContentPath is not null) + return publishedCidNamedContent.returnValue.ContentPath; + } + + // Check in other published cache + if (PublishedStringNamedContent.FirstOrDefault(x => x.returnValue.NamePath is not null && (name.Contains(x.returnValue.NamePath) || x.returnValue.NamePath.Contains(name))) is { } publishedStringNamedContent) + { + if (publishedStringNamedContent.returnValue.ContentPath is not null) + return publishedStringNamedContent.returnValue.ContentPath; + } + + // Check if previously resolved. + if (ResolvedNames.FirstOrDefault(x => x.name == name) is { } resolvedName) + return resolvedName.returnValue; + + // If not, resolve the name and cache. + var result = await Inner.ResolveAsync(name, recursive, nocache, cancel); + ResolvedNames.Add(new(name, recursive, result)); + + return result; + } +} diff --git a/src/Cache/KuboCacheSerializer.cs b/src/Cache/KuboCacheSerializer.cs new file mode 100644 index 0000000..0354f5a --- /dev/null +++ b/src/Cache/KuboCacheSerializer.cs @@ -0,0 +1,80 @@ +using CommunityToolkit.Diagnostics; +using OwlCore.ComponentModel; +using System.Text.Json; + +namespace OwlCore.Kubo.Cache; + +/// +/// An and implementation for serializing and deserializing streams using System.Text.Json. +/// +public class KuboCacheSerializer : IAsyncSerializer, ISerializer +{ + /// + /// A singleton instance for . + /// + public static KuboCacheSerializer Singleton { get; } = new(); + + /// + public async Task SerializeAsync(T data, CancellationToken? cancellationToken = null) + { + var stream = new MemoryStream(); + await JsonSerializer.SerializeAsync(stream, data, typeof(T), context: KuboCacheSerializerContext.Default, cancellationToken: cancellationToken ?? CancellationToken.None); + return stream; + } + + /// + public async Task SerializeAsync(Type inputType, object data, CancellationToken? cancellationToken = null) + { + var stream = new MemoryStream(); + await JsonSerializer.SerializeAsync(stream, data, inputType, context: KuboCacheSerializerContext.Default, cancellationToken: cancellationToken ?? CancellationToken.None); + return stream; + } + + /// + public async Task DeserializeAsync(Stream serialized, CancellationToken? cancellationToken = null) + { + var result = await JsonSerializer.DeserializeAsync(serialized, typeof(TResult), KuboCacheSerializerContext.Default); + Guard.IsNotNull(result); + return (TResult)result; + } + + /// + public async Task DeserializeAsync(Type returnType, Stream serialized, CancellationToken? cancellationToken = null) + { + var result = await JsonSerializer.DeserializeAsync(serialized, returnType, KuboCacheSerializerContext.Default); + Guard.IsNotNull(result); + return result; + } + + /// + public Stream Serialize(T data) + { + var stream = new MemoryStream(); + JsonSerializer.SerializeAsync(stream, data, typeof(T), context: KuboCacheSerializerContext.Default, cancellationToken: CancellationToken.None); + return stream; + } + + /// + public Stream Serialize(Type type, object data) + { + var stream = new MemoryStream(); + JsonSerializer.SerializeAsync(stream, data, type, context: KuboCacheSerializerContext.Default, cancellationToken: CancellationToken.None); + return stream; + } + + /// + public TResult Deserialize(Stream serialized) + { + var result = JsonSerializer.Deserialize(serialized, typeof(TResult), KuboCacheSerializerContext.Default); + Guard.IsNotNull(result); + return (TResult)result; + } + + /// + public object Deserialize(Type type, Stream serialized) + { + var result = JsonSerializer.Deserialize(serialized, type, KuboCacheSerializerContext.Default); + Guard.IsNotNull(result); + return result; + } +} diff --git a/src/Cache/KuboCacheSerializerContext.cs b/src/Cache/KuboCacheSerializerContext.cs new file mode 100644 index 0000000..777db2b --- /dev/null +++ b/src/Cache/KuboCacheSerializerContext.cs @@ -0,0 +1,16 @@ +using System.Text.Json.Serialization; + +namespace OwlCore.Kubo.Cache; + +/// +/// Supplies type information for settings values in . +/// +[JsonSourceGenerationOptions(WriteIndented = true)] +[JsonSerializable(typeof(bool))] +[JsonSerializable(typeof(string))] +[JsonSerializable(typeof(int))] +[JsonSerializable(typeof(List))] +[JsonSerializable(typeof(List))] +public partial class KuboCacheSerializerContext : JsonSerializerContext +{ +} \ No newline at end of file diff --git a/src/ConfigMode.cs b/src/ConfigMode.cs new file mode 100644 index 0000000..7f444d4 --- /dev/null +++ b/src/ConfigMode.cs @@ -0,0 +1,17 @@ +namespace OwlCore.Kubo; + +/// +/// Different ways of handling config settings. +/// +public enum ConfigMode +{ + /// + /// If the config value is already set, it will be used instead. + /// + UseExisting, + + /// + /// If the config value is already set, it will be overwritten. + /// + OverwriteExisting, +} \ No newline at end of file diff --git a/src/DhtRoutingMode.cs b/src/DhtRoutingMode.cs index 8742327..e5ed8dc 100644 --- a/src/DhtRoutingMode.cs +++ b/src/DhtRoutingMode.cs @@ -6,20 +6,49 @@ public enum DhtRoutingMode { /// - /// This is the default routing mode. In the normal, DHT mode IPFS can retrieve content from any peer and seed content to other peers outside your local network. + /// Your node will use NO routing system. You'll have to explicitly connect to peers that have the content you're looking for. + /// + None, + + /// + /// Your node will use the public IPFS DHT (aka "Amino") and parallel IPNI routers. + /// + /// Will accelerate some types of routing with Delegated Routing V1 HTTP API introduced in IPIP-337 in addition to the Amino DHT. By default, an instance of IPNI at https://cid.contact is used. + Auto, + + /// + /// Your node will behave as in "auto" but without running a DHT server. + /// + /// Will accelerate some types of routing with Delegated Routing V1 HTTP API introduced in IPIP-337 in addition to the Amino DHT. By default, an instance of IPNI at https://cid.contact is used. + AutoClient, + + /// + /// In the normal DHT mode IPFS can retrieve content from any peer and seed content to other peers outside your local network. Your node will ONLY use the Amino DHT (no IPNI routers). /// /// - /// This is the recommended choice for most devices. /// If you're working with constrained resources, use . /// The accelerated DHT is available for devices with extra resources (such as a desktop), trading these for faster and more reliable content routing. /// Dht, /// - /// Ideal for devices with constrained resources. In the "dhtclient" mode, IPFS can ask other peers for content, but it will not seed content to peers outside of your local network. + /// In server mode, your node will query other peers for DHT records, and will respond to requests from other peers (both requests to store records and requests to retrieve records). + /// + /// + /// Please do not set this unless you're sure your node is reachable from the public network. + /// + DhtServer, + + /// + /// In client mode, your node will query the DHT as a client but will not respond to requests from other peers. This mode is less resource-intensive than server mode. /// /// /// To further optimize IPFS for devices with constrained resources, try setting the 'lowpower' config profile. /// DhtClient, + + /// + /// All default routers are disabled, and only ones defined in Kubo's 'Routing.Routers' config will be used. + /// + Custom, } \ No newline at end of file diff --git a/src/Downloader/KuboDownloader.cs b/src/Downloader/KuboDownloader.cs index fd14012..963562c 100644 --- a/src/Downloader/KuboDownloader.cs +++ b/src/Downloader/KuboDownloader.cs @@ -3,10 +3,10 @@ using OwlCore.Storage; using System.Runtime.InteropServices; using System.Text.Json; +using OwlCore.Storage.System.Net.Http; namespace OwlCore.Kubo; - /// /// Automatically downloads and extracts the correct Kubo binary for the running operating system and architecture. /// @@ -41,12 +41,12 @@ public static async Task GetLatestBinaryAsync(HttpClient client, Cancella var versionsFile = new HttpFile($"{httpKuboSourcePath}/versions", client); // Scan the versions file and return the latest - var latestVersion = await GetLatestKuboVersionAsync(versionsFile); + var latestVersion = await GetLatestKuboVersionAsync(versionsFile, cancellationToken); var rawVersion = $"v{latestVersion.Major}.{latestVersion.Minor}.{latestVersion.Build}"; // Scan the latest dist.json and get the relative path to the binary archive. var distJson = new HttpFile($"{httpKuboSourcePath}/{rawVersion}/dist.json", client); - var binaryArchiveRelativeDownloadLink = await GetDownloadLink(distJson); + var binaryArchiveRelativeDownloadLink = await GetDownloadLink(distJson, cancellationToken); // Set up the archive file and use ArchiveFolder to crawl the contents of the archive for the Kubo binary. var binaryArchive = new HttpFile($"{httpKuboSourcePath}/{rawVersion}/{binaryArchiveRelativeDownloadLink}", client); @@ -72,12 +72,12 @@ public static async Task GetLatestBinaryAsync(IpfsClient client, Cancella var versionsFile = new IpnsFile($"{ipfsKuboSourcePath}/versions", client); // Scan the versions file and return the latest - var latestVersion = await GetLatestKuboVersionAsync(versionsFile); + var latestVersion = await GetLatestKuboVersionAsync(versionsFile, cancellationToken); var rawVersion = $"v{latestVersion.Major}.{latestVersion.Minor}.{latestVersion.Build}"; // Scan the latest dist.json and get the relative path to the binary archive. var distJson = new IpnsFile($"{ipfsKuboSourcePath}/{rawVersion}/dist.json", client); - var binaryArchiveRelativeDownloadLink = await GetDownloadLink(distJson); + var binaryArchiveRelativeDownloadLink = await GetDownloadLink(distJson, cancellationToken); // Set up the archive file and use ArchiveFolder to crawl the contents of the archive for the Kubo binary. var binaryArchive = new IpnsFile($"{ipfsKuboSourcePath}/{rawVersion}/{binaryArchiveRelativeDownloadLink}", client); @@ -118,7 +118,7 @@ public static async Task GetBinaryVersionAsync(HttpClient client, Version // Scan the dist.json for this version and get the relative path to the binary archive. var distJson = new HttpFile($"{httpKuboSourcePath}/{rawVersion}/dist.json", client); - var binaryArchiveRelativeDownloadLink = await GetDownloadLink(distJson); + var binaryArchiveRelativeDownloadLink = await GetDownloadLink(distJson, cancellationToken); // Set up the archive file and use ArchiveFolder to crawl the contents of the archive for the Kubo binary. var binaryArchive = new HttpFile($"{httpKuboSourcePath}/{rawVersion}/{binaryArchiveRelativeDownloadLink}", client); @@ -148,7 +148,7 @@ public static async Task GetBinaryVersionAsync(IpfsClient client, Version // Scan the dist.json for this version and get the relative path to the binary archive. var distJson = new IpnsFile($"{ipfsKuboSourcePath}/{rawVersion}/dist.json", client); - var binaryArchiveRelativeDownloadLink = await GetDownloadLink(distJson); + var binaryArchiveRelativeDownloadLink = await GetDownloadLink(distJson, cancellationToken); // Set up the archive file and use ArchiveFolder to crawl the contents of the archive for the Kubo binary. var binaryArchive = new IpnsFile($"{ipfsKuboSourcePath}/{rawVersion}/{binaryArchiveRelativeDownloadLink}", client); @@ -203,15 +203,16 @@ private static async IAsyncEnumerable DepthFirstSearch(IFolder folder) /// Reads the provided dist.json for a specific Kubo version and returns the appropriate download link for your current architecture and platform. /// /// The dist.json file to scan. + /// A token that can be used to cancel the ongoing operation. /// The relative file path where the binary archive can be found. - private static async Task GetDownloadLink(IFile kuboVersionDistJson) + private static async Task GetDownloadLink(IFile kuboVersionDistJson, CancellationToken cancellationToken) { Guard.IsTrue(kuboVersionDistJson.Name == "dist.json", nameof(kuboVersionDistJson.Name), "versions file must be named dist.json"); - using var distJsonStream = await kuboVersionDistJson.OpenStreamAsync(); + using var distJsonStream = await kuboVersionDistJson.OpenReadAsync(cancellationToken: cancellationToken); Guard.IsNotNull(distJsonStream); - var distData = await JsonSerializer.DeserializeAsync(distJsonStream, KuboDistributionJsonContext.Default.KuboDistributionData); + var distData = await JsonSerializer.DeserializeAsync(distJsonStream, KuboDistributionJsonContext.Default.KuboDistributionData, cancellationToken); Guard.IsNotNull(distData); Guard.IsNotNull(distData.Platforms); @@ -229,16 +230,21 @@ private static async Task GetDownloadLink(IFile kuboVersionDistJson) } /// - /// Retrieves and parses the the latest version of Kubo from the provided url. + /// Retrieves and parses the latest version of Kubo from the provided url. /// /// /// A token that can be used to cancel the ongoing operation. public static async Task GetLatestKuboVersionAsync(IFile versionsFile, CancellationToken cancellationToken = default) { - using var versionsFileStream = await versionsFile.OpenStreamAsync(FileAccess.Read, cancellationToken); + using var versionsFileStream = await versionsFile.OpenReadAsync(cancellationToken); using var reader = new StreamReader(versionsFileStream); - var versionInformationFromServer = reader.ReadToEnd(); + +#if NET7_0_OR_GREATER + var versionInformationFromServer = await reader.ReadToEndAsync(cancellationToken); +#else + var versionInformationFromServer = await reader.ReadToEndAsync(); +#endif Guard.IsNotNullOrWhiteSpace(versionInformationFromServer); diff --git a/src/Extensions/GenericKuboExtensions.cs b/src/Extensions/GenericKuboExtensions.cs index 15e0088..971b0fb 100644 --- a/src/Extensions/GenericKuboExtensions.cs +++ b/src/Extensions/GenericKuboExtensions.cs @@ -1,5 +1,6 @@ -using Ipfs; -using Ipfs.Http; +using CommunityToolkit.Diagnostics; +using Ipfs; +using Ipfs.CoreApi; namespace OwlCore.Kubo; @@ -12,11 +13,157 @@ public static partial class GenericKuboExtensions /// Gets the CID of the provided object. /// /// The object to serialize into the Dag. - /// The Ipfs client to use. + /// A client that can be used to communicate with Ipfs. + /// Whether to pin the provided data to the node, keeping it retrievable until unpinned. Defaults to false. /// A token that can be used to cancel the ongoing operation. /// - public static Task GetCidAsync(this object serializable, IpfsClient client, CancellationToken cancellationToken) + public static Task GetDagCidAsync(this object serializable, ICoreApi client, bool pin = false, CancellationToken cancellationToken = default) { - return client.Dag.PutAsync(serializable, cancel: cancellationToken, pin: false); + return client.Dag.PutAsync(serializable, cancel: cancellationToken, pin: pin); + } + + /// + /// Resolves the provided if it is an Ipns address and retrieves the content from the DAG. + /// + /// The cid of the DAG object to retrieve. + /// A client that can be used to communicate with Ipfs. + /// Whether to use cached entries if Ipns is resolved. + /// A token that can be used to cancel the ongoing task. + /// The deserialized DAG content, if any. + public static async Task<(TResult? Result, Cid ResultCid)> ResolveDagCidAsync(this Cid cid, ICoreApi client, bool nocache, CancellationToken cancellationToken = default) + { + if (cid.ContentType == "libp2p-key") + { + var ipnsResResult = await client.Name.ResolveAsync($"/ipns/{cid}", recursive: true, nocache: nocache, cancel: cancellationToken); + + cid = Cid.Decode(ipnsResResult.Replace("/ipfs/", "")); + } + + var res = await client.Dag.GetAsync(cid, cancellationToken); + + Guard.IsNotNull(res); + return (res, cid); + } + + /// + /// Resolves the provided if it is an Ipns address and retrieves the content from the DAG. + /// + /// The cid of the DAG object to retrieve. + /// A client that can be used to communicate with Ipfs. + /// Whether to use cached entries if Ipns is resolved. + /// A token that can be used to cancel the ongoing task. + /// The deserialized DAG content, if any. + public static async Task<(TResult? Result, Cid ResultCid)> ResolveDagCidAsync(this ICoreApi client, Cid cid, bool nocache, CancellationToken cancellationToken = default) + { + if (cid.ContentType == "libp2p-key") + { + var ipnsResResult = await client.Name.ResolveAsync($"/ipns/{cid}", recursive: true, nocache: nocache, cancel: cancellationToken); + + cid = Cid.Decode(ipnsResResult.Replace("/ipfs/", "")); + } + + var res = await client.Dag.GetAsync(cid, cancellationToken); + + Guard.IsNotNull(res); + return (res, cid); + } + + /// + /// Resolves the provided as Ipns addresses and retrieves the content from the DAG. + /// + /// The type to deserialize to. + /// The IPNS CIDs of the Dag objects to retrieve. + /// A client that can be used to communicate with Ipfs. + /// Whether to use cached entries if Ipns is resolved. + /// A token that can be used to cancel the ongoing task. + /// An async enumerable that yields the requested data. + public static IAsyncEnumerable<(TResult? Result, Cid ResultCid)> ResolveDagCidAsync(this IEnumerable cids, ICoreApi client, bool nocache, CancellationToken cancellationToken = default) + => cids + .ToAsyncEnumerable() + .SelectAwaitWithCancellation(async (cid, cancel) => await cid.ResolveDagCidAsync(client, nocache, CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, cancel).Token)); + + /// + /// Resolves the provided as Ipns addresses and retrieves the content from the DAG. + /// + /// The type to deserialize to. + /// The IPNS CIDs of the Dag objects to retrieve. + /// A client that can be used to communicate with Ipfs. + /// Whether to use cached entries if Ipns is resolved. + /// A token that can be used to cancel the ongoing task. + /// An async enumerable that yields the requested data. + public static IAsyncEnumerable<(TResult? Result, Cid ResultCid)> ResolveDagCidAsync(this ICoreApi client, IEnumerable cids, bool nocache, CancellationToken cancellationToken = default) + => cids + .ToAsyncEnumerable() + .SelectAwaitWithCancellation(async (cid, cancel) => await cid.ResolveDagCidAsync(client, nocache, CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, cancel).Token)); + + /// + /// Creates an ipns key using a temporary name, then renames it to match the Id of the key. + /// + /// + /// Enables pushing to ipns without additional API calls to convert between ipns cid and name. + /// + /// The key api to use for accessing ipfs keys. + /// The size of the key to create. + /// A token that can be used to cancel the ongoing task. + /// A task containing the created key. + public static async Task CreateKeyWithNameOfIdAsync(this IKeyApi keyApi, int size = 4096, CancellationToken cancellationToken = default) + { + var key = await keyApi.CreateAsync(name: "temp", "ed25519", size, cancellationToken); + + // Rename key name to the key id + return await keyApi.RenameAsync("temp", $"{key.Id}", cancellationToken); + } + + /// + /// Gets a key by name, or creates it if it does not exist. + /// + /// The API to use for keys. + /// The name of the key to get or create. + /// The size of the key to create. + /// A token that can be used to cancel the ongoing task. + /// + public static async Task GetOrCreateKeyAsync(this IKeyApi keyApi, string keyName, int size = 4096, CancellationToken cancellationToken = default) + { + // Get or create ipns key + var keys = await keyApi.ListAsync(cancellationToken); + if (keys.FirstOrDefault(x => x.Name == keyName) is not { } key) + { + // Key does not exist, create it. + key = await keyApi.CreateAsync(keyName, "ed25519", size, cancellationToken); + } + + return key; + } + + /// + /// Gets a key by name, or creates it if it does not exist. + /// + /// The client to use for communicating with the ipfs network. + /// The name of the key to get or create. + /// The lifetime this ipns key should stay alive before needing to be rebroadcast by this node. + /// The size of the key to create. + /// A token that can be used to cancel the ongoing task. + /// Given the created ipns key, provides the default value to be published to it. + /// + public static async Task GetOrCreateKeyAsync(this ICoreApi client, string keyName, Func getDefaultValue, TimeSpan ipnsLifetime, int size = 4096, CancellationToken cancellationToken = default) + { + // Get or create ipns key + var keys = await client.Key.ListAsync(cancellationToken); + if (keys.FirstOrDefault(x => x.Name == keyName) is not { } key) + { + // Key does not exist, create it. + key = await client.Key.CreateAsync(keyName, "ed25519", size, cancellationToken); + Guard.IsNotNull(key); + + // Get default value and cid + var defaultValue = getDefaultValue(key); + Guard.IsNotNull(defaultValue); + + // Publish default value cid + var cid = await client.Dag.PutAsync(defaultValue, cancel: cancellationToken, pin: true); + await client.Name.PublishAsync(cid, key.Name, ipnsLifetime, cancellationToken); + } + + return key; } } diff --git a/src/Extensions/IGetCid.cs b/src/Extensions/IGetCid.cs index 809649c..56c0f33 100644 --- a/src/Extensions/IGetCid.cs +++ b/src/Extensions/IGetCid.cs @@ -1,5 +1,4 @@ using Ipfs; -using Ipfs.Http; using OwlCore.Storage; namespace OwlCore.Kubo; diff --git a/src/Extensions/IpnsDagExtensions.cs b/src/Extensions/IpnsDagExtensions.cs new file mode 100644 index 0000000..9e96964 --- /dev/null +++ b/src/Extensions/IpnsDagExtensions.cs @@ -0,0 +1,92 @@ +using Ipfs; +using Ipfs.CoreApi; + +namespace OwlCore.Kubo.Extensions; + +/// +/// Extensions for using the Dag with Ipns. +/// +public static partial class IpnsDagExtensions +{ + /// + /// Retrieve and transform the data from the provided IPNS CID, then update the IPNS record. + /// + /// The type of data to deserialize, transform and re-serialize. + /// The ipns key to use when resolving the data + /// The name of the key to publish the transformed data to. + /// The transformation to apply over the data before publishing. + /// Whether to use the cache when resolving ipns links. + /// The lifetime of the published ipns entry. Other nodes will drop the record after this amount of time. Your node should be online to rebroadcast ipns at least once every iteration of this lifetime. + /// The client to use for calls to Kubo. + /// Defines a provider for reporting progress. + /// A token that can be used to cancel the ongoing operation. + /// A Task that represents the asynchronous operation. + /// Raised when the provided CID fails to resolve and execution cannot continue. + public static async Task TransformIpnsDagAsync(this ICoreApi client, Cid sourceDagCid, string destinationKeyName, Action transform, bool nocache, TimeSpan ipnsLifetime, IProgress? progress = null, CancellationToken cancellationToken = default) + { + Cid cid = sourceDagCid; + + // Resolve ipns + if (cid.ContentType == "libp2p-key") + { + progress?.Report(IpnsUpdateState.ResolvingIpns); + var ipnsResResult = await client.Name.ResolveAsync($"/ipns/{cid}", recursive: true, nocache: nocache, cancel: cancellationToken); + + cid = Cid.Decode(ipnsResResult.Replace("/ipfs/", "")); + } + + // Resolve data + progress?.Report(IpnsUpdateState.ResolvingDag); + var data = await client.Dag.GetAsync(cid, cancellationToken); + if (data is null) + throw new InvalidOperationException("Failed to resolve data from the provided CID."); + + // Update data + progress?.Report(IpnsUpdateState.TransformingData); + transform(data); + + // Save data + progress?.Report(IpnsUpdateState.AddingToDag); + var newCid = await client.Dag.PutAsync(data, cancel: cancellationToken); + + // Publish new cid to ipns + progress?.Report(IpnsUpdateState.PublishingToIpns); + await client.Name.PublishAsync(newCid, destinationKeyName, ipnsLifetime, cancel: cancellationToken); + } + + /// + /// Describes an update state for the IPNS update process. + /// + public enum IpnsUpdateState + { + /// + /// No update state or not started. + /// + None, + + /// + /// Data is being resolved by the DAG Api. + /// + ResolvingIpns, + + /// + /// Data is being resolved by the DAG Api. + /// + ResolvingDag, + + /// + /// Data is being transformed by the consumer. + /// + TransformingData, + + /// + /// Data is being stored in the Ipfs DAG. + /// + AddingToDag, + + /// + /// The new data is being published to IPNS. + /// + PublishingToIpns, + } +} \ No newline at end of file diff --git a/src/Extensions/OwlCore.Storage.SystemIO/ContentAddressedSystemFile.cs b/src/Extensions/OwlCore.Storage.System.IO/ContentAddressedSystemFile.cs similarity index 67% rename from src/Extensions/OwlCore.Storage.SystemIO/ContentAddressedSystemFile.cs rename to src/Extensions/OwlCore.Storage.System.IO/ContentAddressedSystemFile.cs index 613b92a..12d20e4 100644 --- a/src/Extensions/OwlCore.Storage.SystemIO/ContentAddressedSystemFile.cs +++ b/src/Extensions/OwlCore.Storage.System.IO/ContentAddressedSystemFile.cs @@ -1,21 +1,21 @@ using CommunityToolkit.Diagnostics; using Ipfs; -using Ipfs.Http; +using Ipfs.CoreApi; using OwlCore.Kubo; -namespace OwlCore.Storage.SystemIO; +namespace OwlCore.Storage.System.IO; /// -/// An implementation of with added support for . +/// An implementation of with added support for . /// -public class ContentAddressedSystemFile : OwlCore.Storage.SystemIO.SystemFile, IGetCid +public class ContentAddressedSystemFile : SystemFile, IGetCid { /// /// Creates a new instance of . /// /// /// - public ContentAddressedSystemFile(string path, IpfsClient client) + public ContentAddressedSystemFile(string path, ICoreApi client) : base(path) { Client = client; @@ -24,7 +24,7 @@ public ContentAddressedSystemFile(string path, IpfsClient client) /// /// The IPFS Client to use for retrieving the content. /// - public IpfsClient Client { get; } + public ICoreApi Client { get; } /// public async Task GetCidAsync(CancellationToken cancellationToken) diff --git a/src/Extensions/OwlCore.Storage.SystemIO/ContentAddressedSystemFolder.cs b/src/Extensions/OwlCore.Storage.System.IO/ContentAddressedSystemFolder.cs similarity index 67% rename from src/Extensions/OwlCore.Storage.SystemIO/ContentAddressedSystemFolder.cs rename to src/Extensions/OwlCore.Storage.System.IO/ContentAddressedSystemFolder.cs index cd2fcf6..21d1d0c 100644 --- a/src/Extensions/OwlCore.Storage.SystemIO/ContentAddressedSystemFolder.cs +++ b/src/Extensions/OwlCore.Storage.System.IO/ContentAddressedSystemFolder.cs @@ -1,21 +1,21 @@ using CommunityToolkit.Diagnostics; using Ipfs; -using Ipfs.Http; +using Ipfs.CoreApi; using OwlCore.Kubo; -namespace OwlCore.Storage.SystemIO; +namespace OwlCore.Storage.System.IO; /// -/// An implementation of with added support for . +/// An implementation of with added support for . /// -public class ContentAddressedSystemFolder : OwlCore.Storage.SystemIO.SystemFolder, IGetCid +public class ContentAddressedSystemFolder : SystemFolder, IGetCid { /// /// Creates a new instance of . /// /// /// - public ContentAddressedSystemFolder(string path, IpfsClient client) + public ContentAddressedSystemFolder(string path, ICoreApi client) : base(path) { Client = client; @@ -24,7 +24,7 @@ public ContentAddressedSystemFolder(string path, IpfsClient client) /// /// The IPFS Client to use for retrieving the content. /// - public IpfsClient Client { get; } + public ICoreApi Client { get; } /// public async Task GetCidAsync(CancellationToken cancellationToken) diff --git a/src/Extensions/StorableKuboExtensions.cs b/src/Extensions/StorableKuboExtensions.cs index 302c482..f9fa842 100644 --- a/src/Extensions/StorableKuboExtensions.cs +++ b/src/Extensions/StorableKuboExtensions.cs @@ -1,8 +1,9 @@ using CommunityToolkit.Diagnostics; using Ipfs; +using Ipfs.CoreApi; using Ipfs.Http; using OwlCore.Storage; -using OwlCore.Storage.SystemIO; +using OwlCore.Storage.System.IO; namespace OwlCore.Kubo; @@ -12,7 +13,7 @@ namespace OwlCore.Kubo; public static partial class StorableKuboExtensions { /// - public static async Task GetCidAsync(this IStorable item, IpfsClient client, CancellationToken cancellationToken) + public static async Task GetCidAsync(this IStorable item, ICoreApi client, CancellationToken cancellationToken) { cancellationToken.ThrowIfCancellationRequested(); @@ -43,7 +44,7 @@ public static async Task GetCidAsync(this IStorable item, IpfsClient client var res = await client.FileSystem.AddAsync(stream, file.Name, new() { OnlyHash = true, - Pin = false + Pin = false, }, cancellationToken); Guard.IsFalse(res.IsDirectory); diff --git a/src/FolderWatchers/TimerBasedFolderWatcher.cs b/src/FolderWatchers/TimerBasedFolderWatcher.cs index 0aff90e..fc76e9b 100644 --- a/src/FolderWatchers/TimerBasedFolderWatcher.cs +++ b/src/FolderWatchers/TimerBasedFolderWatcher.cs @@ -1,6 +1,6 @@ -using System.Collections.Specialized; -using OwlCore.Extensions; +using OwlCore.Extensions; using OwlCore.Storage; +using System.Collections.Specialized; namespace OwlCore.Kubo.FolderWatchers; @@ -14,11 +14,11 @@ public abstract class TimerBasedFolderWatcher : IFolderWatcher /// /// Creates a new instance of . /// - /// The folder being watched for changes. + /// The folder being watched for changes. /// How often checks for updates should be made. - public TimerBasedFolderWatcher(IMutableFolder folder, TimeSpan interval) + public TimerBasedFolderWatcher(IMutableFolder kuboNomadFolder, TimeSpan interval) { - Folder = folder; + Folder = kuboNomadFolder; _timer = new Timer(_ => ExecuteAsync().Forget()); _timer.Change(TimeSpan.Zero, interval); diff --git a/src/FolderWatchers/TimerBasedIpnsWatcher.cs b/src/FolderWatchers/TimerBasedIpnsWatcher.cs index c1f2fa0..5263def 100644 --- a/src/FolderWatchers/TimerBasedIpnsWatcher.cs +++ b/src/FolderWatchers/TimerBasedIpnsWatcher.cs @@ -1,9 +1,9 @@ -using System.Collections.Specialized; -using CommunityToolkit.Diagnostics; +using CommunityToolkit.Diagnostics; using Ipfs; +using Ipfs.CoreApi; using Ipfs.Http; -using OwlCore.Extensions; using OwlCore.Storage; +using System.Collections.Specialized; namespace OwlCore.Kubo.FolderWatchers; @@ -12,7 +12,7 @@ namespace OwlCore.Kubo.FolderWatchers; /// public class TimerBasedIpnsWatcher : TimerBasedFolderWatcher { - private readonly IpfsClient _ipfsClient; + private readonly ICoreApi _ipfsClient; private Cid? _lastKnownRootCid; private List _knownItems = new(); @@ -23,7 +23,7 @@ public class TimerBasedIpnsWatcher : TimerBasedFolderWatcher /// The IpfsClient used to check for changes to the IPNS address. /// The folder being watched for changes. /// The interval that IPNS should be checked for updates. - public TimerBasedIpnsWatcher(IpfsClient ipfsClient, IpnsFolder folder, TimeSpan interval) + public TimerBasedIpnsWatcher(ICoreApi ipfsClient, IpnsFolder folder, TimeSpan interval) : base(folder, interval) { _ipfsClient = ipfsClient; @@ -40,7 +40,7 @@ public override async Task ExecuteAsync() { var ipnsPath = Folder.Id; - var resolvedIpnsValue = await _ipfsClient.ResolveAsync(ipnsPath, recursive: true); + var resolvedIpnsValue = await _ipfsClient.Name.ResolveAsync(ipnsPath, recursive: true); Guard.IsNotNullOrWhiteSpace(resolvedIpnsValue); var cid = resolvedIpnsValue.Split(new[] { "/ipfs/" }, StringSplitOptions.None)[1]; diff --git a/src/FolderWatchers/TimerBasedMfsWatcher.cs b/src/FolderWatchers/TimerBasedMfsWatcher.cs index 48ba32f..890a2ed 100644 --- a/src/FolderWatchers/TimerBasedMfsWatcher.cs +++ b/src/FolderWatchers/TimerBasedMfsWatcher.cs @@ -1,11 +1,12 @@ using CommunityToolkit.Diagnostics; using Ipfs; +using Ipfs.CoreApi; using Ipfs.Http; +using OwlCore.Kubo.Models; using OwlCore.Storage; using System.Collections.Specialized; using System.Text; using System.Text.Json; -using OwlCore.Kubo.Models; namespace OwlCore.Kubo.FolderWatchers; @@ -14,7 +15,7 @@ namespace OwlCore.Kubo.FolderWatchers; /// public class TimerBasedMfsWatcher : TimerBasedFolderWatcher { - private readonly IpfsClient _client; + private readonly ICoreApi _client; private bool _running; private Cid? _lastKnownRootCid; @@ -26,7 +27,7 @@ public class TimerBasedMfsWatcher : TimerBasedFolderWatcher /// The folder to watch for changes. /// The client used to make requests to Kubo. /// How often checks for updates should be made. - public TimerBasedMfsWatcher(IpfsClient client, MfsFolder folder, TimeSpan interval) + public TimerBasedMfsWatcher(ICoreApi client, MfsFolder folder, TimeSpan interval) : base(folder, interval) { _client = client; @@ -46,13 +47,7 @@ public override async Task ExecuteAsync() var folder = (MfsFolder)Folder; // This can be a long running operation, so reruns should be prevented for the duration to avoid concurrent requests. - var serialized = await _client.DoCommandAsync("files/stat", CancellationToken.None, folder.Path, "long=true"); - var result = await JsonSerializer.DeserializeAsync(new MemoryStream(Encoding.UTF8.GetBytes(serialized)), typeof(MfsFileStatData), ModelSerializer.Default); - - Guard.IsNotNull(result); - - var data = (MfsFileStatData)result; - + var data = await _client.Mfs.StatAsync(folder.Path); if (data.Hash is not null && _lastKnownRootCid != data.Hash) { var items = await folder.GetItemsAsync().ToListAsync(); diff --git a/src/IpfsFile.cs b/src/IpfsFile.cs index 84a1332..b267c4c 100644 --- a/src/IpfsFile.cs +++ b/src/IpfsFile.cs @@ -1,5 +1,5 @@ using Ipfs; -using Ipfs.Http; +using Ipfs.CoreApi; using OwlCore.ComponentModel; using OwlCore.Storage; @@ -15,7 +15,7 @@ public class IpfsFile : IFile, IChildFile, IGetCid /// /// The CID of the file, such as "QmXg9Pp2ytZ14xgmQjYEiHjVjMFXzCVVEcRTWJBmLgR39V". /// The IPFS Client to use for retrieving the content. - public IpfsFile(Cid cid, IpfsClient client) + public IpfsFile(Cid cid, ICoreApi client) { Name = cid; Id = cid; @@ -28,7 +28,7 @@ public IpfsFile(Cid cid, IpfsClient client) /// The CID of the file, such as "QmXg9Pp2ytZ14xgmQjYEiHjVjMFXzCVVEcRTWJBmLgR39V". /// The name of the file. /// The IPFS Client to use for retrieving the content. - public IpfsFile(Cid cid, string name, IpfsClient client) + public IpfsFile(Cid cid, string name, ICoreApi client) { Name = !string.IsNullOrWhiteSpace(name) ? name : cid; Id = cid; @@ -38,29 +38,29 @@ public IpfsFile(Cid cid, string name, IpfsClient client) /// /// The IPFS Client to use for retrieving the content. /// - public IpfsClient Client { get; } + public ICoreApi Client { get; } /// - public string Id { get; } + public virtual string Id { get; } /// - public string Name { get; } + public virtual string Name { get; } /// /// The parent directory, if any. /// - internal IpfsFolder? Parent { get; init; } = null; + public virtual IpfsFolder? Parent { get; init; } = null; /// public Task GetParentAsync(CancellationToken cancellationToken = default) => Task.FromResult(Parent); /// - public async Task OpenStreamAsync(FileAccess accessMode = FileAccess.Read, CancellationToken cancellationToken = default) + public virtual async Task OpenStreamAsync(FileAccess accessMode = FileAccess.Read, CancellationToken cancellationToken = default) { if (accessMode.HasFlag(FileAccess.Write)) throw new NotSupportedException("Attempted to write data to an immutable file on IPFS."); - var fileData = await Client.FileSystem.ListFileAsync(Id, cancellationToken); + var fileData = await Client.Mfs.StatAsync($"/ipfs/{Id}", cancellationToken); var stream = await Client.FileSystem.ReadFileAsync(Id, cancellationToken); var streamWithLength = new LengthOverrideStream(stream, fileData.Size); diff --git a/src/IpfsFolder.cs b/src/IpfsFolder.cs index 339a9cf..365f474 100644 --- a/src/IpfsFolder.cs +++ b/src/IpfsFolder.cs @@ -1,93 +1,91 @@ using CommunityToolkit.Diagnostics; using Ipfs; -using Ipfs.Http; +using Ipfs.CoreApi; using OwlCore.Storage; using System.Runtime.CompilerServices; -namespace OwlCore.Kubo +namespace OwlCore.Kubo; + +/// +/// A folder that resides on IPFS. +/// +public class IpfsFolder : IFolder, IChildFolder, IGetCid { + /// - /// A folder that resides on IPFS. + /// Creates a new instance of . /// - public class IpfsFolder : IFolder, IChildFolder, IGetCid + /// The CID of the folder, such as "QmXg9Pp2ytZ14xgmQjYEiHjVjMFXzCVVEcRTWJBmLgR39V". + /// The IPFS Client to use for retrieving the content. + public IpfsFolder(Cid cid, ICoreApi client) { + Id = cid; + Name = cid; + Client = client; + } - /// - /// Creates a new instance of . - /// - /// The CID of the folder, such as "QmXg9Pp2ytZ14xgmQjYEiHjVjMFXzCVVEcRTWJBmLgR39V". - /// The IPFS Client to use for retrieving the content. - public IpfsFolder(Cid cid, IpfsClient client) - { - Id = cid; - Name = cid; - Client = client; - } + /// + /// Creates a new instance of . + /// + /// The CID of the folder, such as "QmXg9Pp2ytZ14xgmQjYEiHjVjMFXzCVVEcRTWJBmLgR39V". + /// The name of the folder. + /// The IPFS Client to use for retrieving the content. + public IpfsFolder(Cid cid, string name, ICoreApi client) + { + Id = cid; + Name = !string.IsNullOrWhiteSpace(name) ? name : cid; + Client = client; + } - /// - /// Creates a new instance of . - /// - /// The CID of the folder, such as "QmXg9Pp2ytZ14xgmQjYEiHjVjMFXzCVVEcRTWJBmLgR39V". - /// The name of the folder. - /// The IPFS Client to use for retrieving the content. - public IpfsFolder(Cid cid, string name, IpfsClient client) - { - Id = cid; - Name = !string.IsNullOrWhiteSpace(name) ? name : cid; - Client = client; - } + /// + /// The IPFS Client to use for retrieving the content. + /// + public ICoreApi Client { get; } - /// - /// The IPFS Client to use for retrieving the content. - /// - public IpfsClient Client { get; } + /// + public string Id { get; } - /// - public string Id { get; } + /// + public string Name { get; } - /// - public string Name { get; } + /// + /// The parent directory, if any. + /// + internal IpfsFolder? Parent { get; init; } = null; - /// - /// The parent directory, if any. - /// - internal IpfsFolder? Parent { get; init; } = null; + /// + public Task GetParentAsync(CancellationToken cancellationToken = default) => Task.FromResult(Parent); - /// - public Task GetParentAsync(CancellationToken cancellationToken = default) => Task.FromResult(Parent); + /// + public virtual async IAsyncEnumerable GetItemsAsync(StorableType type = StorableType.All, [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + var itemInfo = await Client.FileSystem.ListAsync(Id, cancellationToken); + Guard.IsTrue(itemInfo.IsDirectory); - /// - public virtual async IAsyncEnumerable GetItemsAsync(StorableType type = StorableType.All, [EnumeratorCancellation] CancellationToken cancellationToken = default) + foreach (var link in itemInfo.Links) { - var itemInfo = await Client.FileSystem.ListFileAsync(Id, cancellationToken); - Guard.IsTrue(itemInfo.IsDirectory); + Guard.IsNotNullOrWhiteSpace(link.Id); + Guard.IsNotNull(link.Name); - foreach (var link in itemInfo.Links) + var linkedItemInfo = await Client.Mfs.StatAsync($"/ipfs/{link.Id}", cancellationToken); + if (linkedItemInfo.IsDirectory && type.HasFlag(StorableType.Folder)) { - Guard.IsNotNullOrWhiteSpace(link.Id); - Guard.IsNotNull(link.Name); - - var linkedItemInfo = await Client.FileSystem.ListFileAsync(link.Id, cancellationToken); - - if (linkedItemInfo.IsDirectory && type.HasFlag(StorableType.Folder)) + yield return new IpfsFolder(link.Id, link.Name, Client) { - yield return new IpfsFolder(linkedItemInfo.Id, link.Name, Client) - { - Parent = this, - }; - } - else if (type.HasFlag(StorableType.File)) + Parent = this, + }; + } + else if (type.HasFlag(StorableType.File)) + { + yield return new IpfsFile(link.Id, link.Name, Client) { - yield return new IpfsFile(linkedItemInfo.Id, link.Name, Client) - { - Parent = this, - }; - } + Parent = this, + }; } } - - /// - public Task GetCidAsync(CancellationToken cancellationToken) => Task.FromResult(Id); } -} + + /// + public Task GetCidAsync(CancellationToken cancellationToken) => Task.FromResult(Id); +} \ No newline at end of file diff --git a/src/IpnsFile.cs b/src/IpnsFile.cs index c499dce..58cf416 100644 --- a/src/IpnsFile.cs +++ b/src/IpnsFile.cs @@ -1,6 +1,6 @@ using CommunityToolkit.Diagnostics; using Ipfs; -using Ipfs.Http; +using Ipfs.CoreApi; using OwlCore.Storage; namespace OwlCore.Kubo; @@ -15,7 +15,7 @@ public class IpnsFile : IFile, IChildFile, IGetCid /// /// A resolvable IPNS address, such as "ipfs.tech" or "k51qzi5uqu5dip7dqovvkldk0lz03wjkc2cndoskxpyh742gvcd5fw4mudsorj". /// The IPFS Client to use for retrieving the content. - public IpnsFile(string ipnsAddress, IpfsClient client) + public IpnsFile(string ipnsAddress, ICoreApi client) { Id = ipnsAddress; Name = PathHelpers.GetFolderItemName(ipnsAddress); @@ -31,7 +31,7 @@ public IpnsFile(string ipnsAddress, IpfsClient client) /// /// The IPFS Client to use for retrieving the content. /// - public IpfsClient Client { get; } + public ICoreApi Client { get; } /// /// The parent directory, if any. @@ -57,10 +57,12 @@ public virtual async Task OpenStreamAsync(FileAccess accessMode = FileAc /// The resolved CID. public async Task GetCidAsync(CancellationToken cancellationToken) { - var resolvedIpns = await Client.ResolveAsync(Id, recursive: true, cancel: cancellationToken); + var resolvedIpns = await Client.Name.ResolveAsync(Id, recursive: true, cancel: cancellationToken); Guard.IsNotNull(resolvedIpns); - Cid cid = resolvedIpns.Split(new[] { "/ipfs/" }, StringSplitOptions.None)[1]; - return cid; + var cidOfResolvedIpfsPath = await Client.Mfs.StatAsync(resolvedIpns, cancel: cancellationToken); + + Guard.IsNotNull(cidOfResolvedIpfsPath.Hash); + return cidOfResolvedIpfsPath.Hash; } } \ No newline at end of file diff --git a/src/IpnsFolder.cs b/src/IpnsFolder.cs index 0c168e1..31c0f88 100644 --- a/src/IpnsFolder.cs +++ b/src/IpnsFolder.cs @@ -1,6 +1,6 @@ using CommunityToolkit.Diagnostics; using Ipfs; -using Ipfs.Http; +using Ipfs.CoreApi; using OwlCore.Kubo.FolderWatchers; using OwlCore.Storage; using System.Runtime.CompilerServices; @@ -10,14 +10,14 @@ namespace OwlCore.Kubo; /// /// A folder that resides on IPFS behind an IPNS Address. /// -public class IpnsFolder : IMutableFolder, IChildFolder, IFastGetRoot, IFastGetItem, IFastGetItemRecursive, IGetCid +public class IpnsFolder : IMutableFolder, IChildFolder, IGetRoot, IGetItem, IGetItemRecursive, IGetCid { /// /// Creates a new instance of . /// /// A resolvable IPNS address, such as "/ipns/ipfs.tech" or "/ipns/k51qzi5uqu5dip7dqovvkldk0lz03wjkc2cndoskxpyh742gvcd5fw4mudsorj". /// The IPFS Client to use for retrieving the content. - public IpnsFolder(string ipnsAddress, IpfsClient client) + public IpnsFolder(string ipnsAddress, ICoreApi client) { Guard.IsTrue(ipnsAddress.StartsWith("/ipns/"), nameof(ipnsAddress), "Value must start with /ipns/"); @@ -40,7 +40,7 @@ public IpnsFolder(string ipnsAddress, IpfsClient client) /// /// The IPFS Client to use for retrieving the content. /// - protected IpfsClient Client { get; } + protected ICoreApi Client { get; } /// public virtual Task GetParentAsync(CancellationToken cancellationToken = default) @@ -52,7 +52,7 @@ public IpnsFolder(string ipnsAddress, IpfsClient client) } /// - public virtual Task GetRootAsync() + public virtual Task GetRootAsync(CancellationToken cancellationToken = default) { if (Id == "/") return Task.FromResult(null); @@ -71,13 +71,15 @@ public IpnsFolder(string ipnsAddress, IpfsClient client) /// public virtual async IAsyncEnumerable GetItemsAsync(StorableType type = StorableType.All, [EnumeratorCancellation] CancellationToken cancellationToken = default) { - var itemInfo = await Client.FileSystem.ListFileAsync(Id, cancellationToken); + var cid = await GetCidAsync(cancellationToken); + var itemInfo = await Client.FileSystem.ListAsync(cid, cancellationToken); Guard.IsTrue(itemInfo.IsDirectory); foreach (var link in itemInfo.Links) { Guard.IsNotNullOrWhiteSpace(link.Id); - var item = await GetFileOrFolderFromId($"{Id}/{link.Name}", cancellationToken); + var path = $"{Id}/{link.Name}"; + var item = await GetFileOrFolderFromId(path, cancellationToken); if (item is IFolder && type.HasFlag(StorableType.Folder)) yield return item; @@ -101,7 +103,8 @@ public virtual Task GetFolderWatcherAsync(CancellationToken canc private async Task GetFileOrFolderFromId(string path, CancellationToken cancellationToken = default) { - var linkedItemInfo = await Client.FileSystem.ListFileAsync(path, cancellationToken); + var cid = await GetCidAsync(path, cancellationToken); + var linkedItemInfo = await Client.Mfs.StatAsync($"/ipfs/{cid}", cancellationToken); if (linkedItemInfo.IsDirectory) return new IpnsFolder(path, Client) { Parent = this, }; @@ -114,12 +117,20 @@ private async Task GetFileOrFolderFromId(string path, Cancellati /// /// Used to cancel the ongoing operation. /// The resolved CID. - public async Task GetCidAsync(CancellationToken cancellationToken) + public Task GetCidAsync(CancellationToken cancellationToken) => GetCidAsync(Id, cancellationToken); + + /// + /// Retrieves the current CID of this item from IPNS. + /// + /// The ID to resolve. + /// Used to cancel the ongoing operation. + /// The resolved CID. + private async Task GetCidAsync(string id, CancellationToken cancellationToken) { - var resolvedIpns = await Client.ResolveAsync(Id, recursive: true, cancel: cancellationToken); - Guard.IsNotNull(resolvedIpns); + var resolvedIpns = await Client.Name.ResolveAsync(id, recursive: true, cancel: cancellationToken); + var cidOfResolvedPath = await Client.Mfs.StatAsync(resolvedIpns, cancel: cancellationToken); - Cid cid = resolvedIpns.Split(new[] { "/ipfs/" }, StringSplitOptions.None)[1]; - return cid; + Guard.IsNotNull(cidOfResolvedPath.Hash); + return cidOfResolvedPath.Hash; } } \ No newline at end of file diff --git a/src/KuboBootstrapper.cs b/src/KuboBootstrapper.cs index e2dd298..881bba1 100644 --- a/src/KuboBootstrapper.cs +++ b/src/KuboBootstrapper.cs @@ -1,9 +1,14 @@ -using System.Diagnostics; -using System.Runtime.InteropServices; -using CommunityToolkit.Diagnostics; +using CommunityToolkit.Diagnostics; +using Ipfs; using Ipfs.Http; +using Newtonsoft.Json.Linq; +using OwlCore.Extensions; using OwlCore.Storage; -using OwlCore.Storage.SystemIO; +using OwlCore.Storage.System.IO; +using System.Diagnostics; +using System.Linq; +using System.Runtime.InteropServices; +using System.Text; namespace OwlCore.Kubo; @@ -14,7 +19,8 @@ public class KuboBootstrapper : IDisposable { private IpfsClient? _client; private readonly Func> _getKuboBinaryFile; - private SystemFile? _executableBinary; + private SystemFile? _kuboBinaryFile; + private SystemFolder? _repoFolder; /// /// Create a new instance of . @@ -42,7 +48,7 @@ public KuboBootstrapper(string repoPath, Version kuboVersion) /// /// The path to the kubo repository folder. Provided to Kubo. public KuboBootstrapper(string repoPath) - : this(repoPath, canceltok => KuboDownloader.GetLatestBinaryAsync(canceltok)) + : this(repoPath, KuboDownloader.GetLatestBinaryAsync) { } @@ -51,11 +57,34 @@ public KuboBootstrapper(string repoPath) /// public Process? Process { get; private set; } + /// + /// The environment variables to set for the Kubo process. + /// + public Dictionary EnvironmentVariables { get; set; } = []; + /// /// The path to the kubo repository folder. Provided to Kubo. /// public string RepoPath { get; set; } + /// + /// The folder containing the kubo repository. + /// + public SystemFolder RepoFolder => _repoFolder ??= new SystemFolder(RepoPath); + + /// + /// The Kubo binary being bootstrapped, compatible with the currently running OS and Architecture. + /// + public SystemFile? KuboBinaryFile => _kuboBinaryFile; + + /// + /// Gets or sets the folder where the Kubo binary will be copied to and run from via a new . + /// + /// + /// This location must be one where the current environment can execute a binary. For both Linux and Windows, one common location for this is the Temp folder. + /// + public SystemFolder BinaryWorkingFolder { get; set; } = new(Path.GetTempPath()); + /// /// The address where the API should be hosted. /// @@ -66,6 +95,21 @@ public KuboBootstrapper(string repoPath) /// public Uri GatewayUri { get; set; } = new("http://127.0.0.1:8080"); + /// + /// Gets or sets an enum that determines how to use the supplied . + /// + public ConfigMode ApiUriMode { get; set; } = ConfigMode.OverwriteExisting; + + /// + /// Gets or sets an enum that determines how to use the supplied . + /// + public ConfigMode GatewayUriMode { get; set; } = ConfigMode.OverwriteExisting; + + /// + /// The behavior when a node is already running (when the repo is locked). + /// + public BootstrapLaunchConflictMode LaunchConflictMode { get; set; } = BootstrapLaunchConflictMode.Throw; + /// /// Gets or creates an to interact with the given . /// @@ -74,77 +118,157 @@ public KuboBootstrapper(string repoPath) /// /// The routing mode that should be used. /// - public DhtRoutingMode RoutingMode { get; init; } + public DhtRoutingMode RoutingMode { get; init; } = DhtRoutingMode.Auto; /// - /// The Kubo profiles that will be applied before starting the daemon. + /// + /// This alternative Amino DHT client with a Full-Routing-Table strategy will do a complete scan of the DHT every hour and record all nodes found. Then when a lookup is tried instead of having to go through multiple Kad hops it is able to find the 20 final nodes by looking up the in-memory recorded network table. + /// + /// + /// + /// This means sustained higher memory to store the routing table and extra CPU and network bandwidth for each network scan. However the latency of individual read/write operations should be ~10x faster and the provide throughput up to 6 million times faster on larger datasets! + /// /// - public List StartupProfiles { get; } = new(); + /// + /// When it is enabled: + /// Client DHT operations (reads and writes) should complete much faster. + /// The provider will now use a keyspace sweeping mode allowing to keep alive CID sets that are multiple orders of magnitude larger. + /// The standard Bucket-Routing-Table DHT will still run for the DHT server (if the DHT server is enabled). This means the classical routing table will still be used to answer other nodes. This is critical to maintain to not harm the network. + /// + /// The operations 'ipfs stats dht' will default to showing information about the accelerated DHT client. + /// + /// Caveats: + /// + /// Running the accelerated client likely will result in more resource consumption (connections, RAM, CPU, bandwidth) + /// Users that are limited in the number of parallel connections their machines/networks can perform will likely suffer. + /// The resource usage is not smooth as the client crawls the network in rounds and reproviding is similarly done in rounds. + /// Users who previously had a lot of content but were unable to advertise it on the network will see an increase in egress bandwidth as their nodes start to advertise all of their CIDs into the network. If you have lots of data entering your node that you don't want to advertise, then consider using Reprovider Strategies to reduce the number of CIDs that you are reproviding. Similarly, if you are running a node that deals mostly with short-lived temporary data (e.g. you use a separate node for ingesting data then for storing and serving it) then you may benefit from using Strategic Providing to prevent advertising of data that you ultimately will not have. + /// + /// Currently, the DHT is not usable for queries for the first 5-10 minutes of operation as the routing table is being prepared. This means operations like searching the DHT for particular peers or content will not work initially. + /// You can see if the DHT has been initially populated by running 'ipfs stats dht'. + /// Currently, the accelerated DHT client is not compatible with LAN-based DHTs and will not perform operations against them. + /// + /// + public bool UseAcceleratedDHTClient { get; set; } /// - /// Gets or sets the folder where the Kubo binary will be copied to and run from via a new . + /// The Kubo profiles that will be applied before starting the daemon. /// - /// - /// This location must be one where the current environment can execute a binary. For both Linux and Windows, one common location for this is the Temp folder. - /// - public SystemFolder BinaryWorkingFolder { get; set; } = new(Path.GetTempPath()); + public List StartupProfiles { get; } = new(); /// /// Loads the binary and starts it in a new process. /// /// Cancels the startup process. /// - public async Task StartAsync(CancellationToken cancellationToken = default) + public virtual async Task StartAsync(CancellationToken cancellationToken = default) { - IFile? kuboBinary = await BinaryWorkingFolder - .GetFilesAsync(cancellationToken) - .FirstOrDefaultAsync(x => x.Name.StartsWith("ipfs") || x.Name.StartsWith("kubo")); + // Get or download the Kubo binary, if needed. + _kuboBinaryFile ??= await GetOrDownloadExecutableKuboBinary(cancellationToken); + + if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux)) + SetExecutablePermissionsForBinary(_kuboBinaryFile); - // Retrieve the kubo binary if we don't have it - kuboBinary ??= await _getKuboBinaryFile(cancellationToken); + var repoLocked = await IsRepoLockedAsync(new SystemFolder(RepoPath), cancellationToken); - _executableBinary ??= (SystemFile)await BinaryWorkingFolder.CreateCopyOfAsync(kuboBinary, overwrite: false, cancellationToken); + // If the repo is locked, check the launch conflict mode + if (repoLocked) + { + switch (LaunchConflictMode) + { + case BootstrapLaunchConflictMode.Throw: + throw new InvalidOperationException("The repository is locked and the launch conflict mode is set to throw."); + + case BootstrapLaunchConflictMode.Relaunch: + // Attach to the existing node and shut it down + var apiMultiAddr = await GetApiAsync(cancellationToken); + Guard.IsNotNull(apiMultiAddr); + await new IpfsClient { ApiUri = TcpIpv4MultiAddressToUri(apiMultiAddr) }.Generic.ShutdownAsync(); + break; + + case BootstrapLaunchConflictMode.Attach: + ApiUriMode = ConfigMode.UseExisting; + GatewayUriMode = ConfigMode.UseExisting; + break; + } + } - if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux)) - SetExecutablePermissionsForBinary(_executableBinary); + if (ApiUriMode == ConfigMode.UseExisting) + { + var apiMultiAddr = await GetApiAsync(cancellationToken); + if (apiMultiAddr is not null) + ApiUri = TcpIpv4MultiAddressToUri(apiMultiAddr); + } - ApplySettings(); + if (GatewayUriMode == ConfigMode.UseExisting) + { + var gatewayMultiAddr = await GetGatewayAsync(cancellationToken); + if (gatewayMultiAddr is not null) + GatewayUri = TcpIpv4MultiAddressToUri(gatewayMultiAddr); + } - var processStartInfo = new ProcessStartInfo(_executableBinary.Path, $"daemon --routing={RoutingMode.ToString().ToLowerInvariant()} --enable-pubsub-experiment --enable-namesys-pubsub --repo-dir \"{RepoPath}\"") + if (LaunchConflictMode == BootstrapLaunchConflictMode.Attach && repoLocked) { - CreateNoWindow = true + // In attach mode, we don't bootstrap the process. + // Accessing the Client property will connect to the running Daemon using the Kubo RPC API port we have set. + return; + } + + // Settings must be applied before bootstrapping. + await ApplySettingsAsync(cancellationToken); + + // Setup process info + var processStartInfo = new ProcessStartInfo(_kuboBinaryFile.Path, $"daemon --routing={RoutingMode.ToString().ToLowerInvariant()} --enable-pubsub-experiment --enable-namesys-pubsub --repo-dir \"{RepoPath}\"") + { + CreateNoWindow = true, }; + // Setup environment variables + foreach (var item in EnvironmentVariables) + processStartInfo.EnvironmentVariables.Add(item.Key, item.Value); + + await StartAsync(processStartInfo, cancellationToken); + } + + private async Task StartAsync(ProcessStartInfo processStartInfo, CancellationToken cancellationToken) + { + // Setup process Process = new Process { StartInfo = processStartInfo, - EnableRaisingEvents = true + EnableRaisingEvents = true, }; - Process.StartInfo.RedirectStandardOutput = true; - Process.StartInfo.RedirectStandardInput = true; - Process.StartInfo.RedirectStandardError = true; - Process.StartInfo.UseShellExecute = false; + var process = Process; - Process.Start(); + process.StartInfo.RedirectStandardOutput = true; + process.StartInfo.RedirectStandardInput = true; + process.StartInfo.RedirectStandardError = true; + process.StartInfo.UseShellExecute = false; - var cancellationCleanup = cancellationToken.Register(() => Process.Dispose()); + // Start + process.Start(); - Process.BeginOutputReadLine(); - Process.BeginErrorReadLine(); + // Close the process if cancellation is called early. + var cancellationCleanup = cancellationToken.Register(() => process.Dispose()); + process.BeginOutputReadLine(); + process.BeginErrorReadLine(); + + // Create a task completion source to wait for the daemon to be ready. var startupCompletion = new TaskCompletionSource(); - Process.OutputDataReceived += Process_OutputDataReceived; - Process.ErrorDataReceived += ProcessOnErrorDataReceived; + process.OutputDataReceived += Process_OutputDataReceived; + process.ErrorDataReceived += ProcessOnErrorDataReceived; cancellationToken.ThrowIfCancellationRequested(); + // Wait for daemon to be ready await startupCompletion.Task; cancellationToken.ThrowIfCancellationRequested(); - Process.OutputDataReceived -= Process_OutputDataReceived; - Process.ErrorDataReceived -= ProcessOnErrorDataReceived; + process.OutputDataReceived -= Process_OutputDataReceived; + process.ErrorDataReceived -= ProcessOnErrorDataReceived; cancellationCleanup.Dispose(); @@ -157,10 +281,23 @@ void Process_OutputDataReceived(object? sender, DataReceivedEventArgs e) startupCompletion.SetResult(null); } - void ProcessOnErrorDataReceived(object sender, DataReceivedEventArgs e) + async void ProcessOnErrorDataReceived(object sender, DataReceivedEventArgs e) { if (!string.IsNullOrWhiteSpace(e.Data) && e.Data.Contains("Error: ")) { + if (e.Data.Contains($"/ip4/{ApiUri.Host}/tcp/{ApiUri.Port}") && e.Data.Contains("bind")) + { + process.OutputDataReceived -= Process_OutputDataReceived; + process.ErrorDataReceived -= ProcessOnErrorDataReceived; + + // Failed to bind to port, process may be orphaned. + // Connect and shutdown via api instead. + await new IpfsClient { ApiUri = ApiUri }.Generic.ShutdownAsync(); + await StartAsync(processStartInfo, cancellationToken); + startupCompletion.SetResult(null); + return; + } + throw new InvalidOperationException($"Error received while starting daemon: {e.Data}"); } } @@ -169,46 +306,240 @@ void ProcessOnErrorDataReceived(object sender, DataReceivedEventArgs e) /// /// Stops the bootstrapped process. /// - public void Stop() + public virtual void Stop() { - Guard.IsNotNullOrWhiteSpace(_executableBinary?.Path); - - if (Process is not null && !Process.HasExited) + if (_kuboBinaryFile is not null && Process is not null && !Process.HasExited) { // Gracefully shutdown the running Kubo Daemon - RunExecutable(_executableBinary, $"shutdown --repo-dir \"{RepoPath}\"", throwOnError: false); + RunExecutable(_kuboBinaryFile, $"shutdown --repo-dir \"{RepoPath}\"", throwOnError: false); Process.Close(); } Process = null; } + /// + /// Gets or downloads the Kubo binary and sets it up in the . + /// + /// A token that can be used to cancel the ongoing operation. + protected virtual async Task GetOrDownloadExecutableKuboBinary(CancellationToken cancellationToken) + { + IFile? existingBinary = await BinaryWorkingFolder + .GetFilesAsync(cancellationToken) + .FirstOrDefaultAsync(x => x.Name.StartsWith("ipfs") || x.Name.StartsWith("kubo"), cancellationToken: cancellationToken); + + if (existingBinary is null) + { + // Retrieve the kubo binary if we don't have it + existingBinary ??= await _getKuboBinaryFile(cancellationToken); + + // Copy it into the binary working folder, store the new file for use. + return (SystemFile)await BinaryWorkingFolder.CreateCopyOfAsync(existingBinary, overwrite: false, cancellationToken); + } + + return (SystemFile)existingBinary; + } + + /// + /// Checks if the Kubo repository at the provided has an active repo.lock. + /// + /// A token that can be used to cancel the ongoing operation. + /// A Task containing a boolean value. If true, the repo is locked, otherwise false. + public Task IsRepoLockedAsync(CancellationToken cancellationToken) => IsRepoLockedAsync(new SystemFolder(RepoPath), cancellationToken); + + /// + /// Gets the gateway address from the provided . + /// + /// A token that can be used to cancel the ongoing operation. + /// The formatted gateway address, if found. + public Task GetGatewayAsync(CancellationToken cancellationToken) => GetGatewayAsync(new SystemFolder(RepoPath), cancellationToken); + + /// + /// Gets the gateway address from the provided . + /// + /// A token that can be used to cancel the ongoing operation. + /// The formatted api address, if found. + public Task GetApiAsync(CancellationToken cancellationToken) => GetApiAsync(new SystemFolder(RepoPath), cancellationToken); + + /// + /// Checks of the Kubo repository at the provided has an active repo.lock. + /// + /// The Kubo repository to check. + /// A token that can be used to cancel the ongoing operation. + /// + public static async Task IsRepoLockedAsync(IFolder kuboRepo, CancellationToken cancellationToken) + { + try + { + var target = await kuboRepo.GetFirstByNameAsync("repo.lock", cancellationToken); + return true; + } + catch (FileNotFoundException) + { + return false; + } + } + + /// + /// Converts a TcpIpv4 to a with the location and port. + /// + /// The to transform. + /// A standard uri containing the location and port provided from the . + public static Uri TcpIpv4MultiAddressToUri(MultiAddress multiAddress) + { + return new Uri($"http://{multiAddress.Protocols[0].Value}:{multiAddress.Protocols[1].Value}"); + } + + /// + /// Gets the gateway address from the provided . + /// + /// The Kubo repository to check. + /// A token that can be used to cancel the ongoing operation. + /// The formatted gateway address, if found. + public static async Task GetGatewayAsync(IFolder kuboRepo, CancellationToken cancellationToken) + { + IFile? file = null; + try + { + file = (IFile)await kuboRepo.GetFirstByNameAsync("config", cancellationToken); + } + catch (FileNotFoundException) + { + return null; + } + + using var stream = await file.OpenStreamAsync(FileAccess.Read, cancellationToken); + var bytes = await stream.ToBytesAsync(cancellationToken: cancellationToken); + + var strng = Encoding.UTF8.GetString(bytes).Trim(); + var config = JObject.Parse(strng); + + var addresses = config["Addresses"]; + Guard.IsNotNull(addresses); + + var gateway = addresses["Gateway"]?.Value(); + Guard.IsNotNullOrWhiteSpace(gateway); + + return new MultiAddress(gateway); + } + + /// + /// Gets the API address from the provided . + /// + /// The Kubo repository to check. + /// A token that can be used to cancel the ongoing operation. + /// The formatted api address, if found. + public static async Task GetApiAsync(IFolder kuboRepo, CancellationToken cancellationToken) + { + IFile? file = null; + try + { + file = (IFile)await kuboRepo.GetFirstByNameAsync("config", cancellationToken); + } + catch (FileNotFoundException) + { + return null; + } + + using var stream = await file.OpenStreamAsync(FileAccess.Read, cancellationToken); + var bytes = await stream.ToBytesAsync(cancellationToken: cancellationToken); + + var strng = Encoding.UTF8.GetString(bytes).Trim(); + var config = JObject.Parse(strng); + + var addresses = config["Addresses"]; + Guard.IsNotNull(addresses); + + var api = addresses["API"]?.Value(); + Guard.IsNotNullOrWhiteSpace(api); + + return new MultiAddress(api); + } + /// /// Initializes the local node with the provided settings. /// /// A that represents the asynchronous operation. - public void ApplySettings() + public virtual async Task ApplySettingsAsync(CancellationToken cancellationToken) { - Guard.IsNotNullOrWhiteSpace(_executableBinary?.Path); + Guard.IsNotNullOrWhiteSpace(_kuboBinaryFile?.Path); + // Init if needed try { - RunExecutable(_executableBinary, $"init --repo-dir \"{RepoPath}\"", throwOnError: false); + RunExecutable(_kuboBinaryFile, $"init --repo-dir \"{RepoPath}\"", throwOnError: false); } catch { // ignored } - RunExecutable(_executableBinary, $"config --repo-dir \"{RepoPath}\" Routing.Type {RoutingMode.ToString().ToLowerInvariant()}", throwOnError: true); - RunExecutable(_executableBinary, $"config --repo-dir \"{RepoPath}\" Addresses.API /ip4/{ApiUri.Host}/tcp/{ApiUri.Port}", throwOnError: true); - RunExecutable(_executableBinary, $"config --repo-dir \"{RepoPath}\" Addresses.Gateway /ip4/{GatewayUri.Host}/tcp/{GatewayUri.Port}", throwOnError: true); + await ApplyRoutingSettingsAsync(cancellationToken); + await ApplyPortSettingsAsync(cancellationToken); + await ApplyStartupProfileSettingsAsync(cancellationToken); + } + + /// + /// Initializes the local node with the provided startup profile settings. + /// + /// A that represents the asynchronous operation. + protected virtual async Task ApplyStartupProfileSettingsAsync(CancellationToken cancellationToken) + { + Guard.IsNotNullOrWhiteSpace(_kuboBinaryFile?.Path); + // Startup profiles foreach (var profile in StartupProfiles) - RunExecutable(_executableBinary, $"config --repo-dir {RepoPath} profile apply {profile}", throwOnError: true); + RunExecutable(_kuboBinaryFile, $"config --repo-dir {RepoPath} profile apply {profile}", throwOnError: true); + } + + /// + /// Initializes the local node with the provided port settings. + /// + /// A that represents the asynchronous operation. + protected virtual async Task ApplyPortSettingsAsync(CancellationToken cancellationToken) + { + Guard.IsNotNullOrWhiteSpace(_kuboBinaryFile?.Path); + + // Port options + if (ApiUriMode == ConfigMode.OverwriteExisting) + RunExecutable(_kuboBinaryFile, $"config --repo-dir \"{RepoPath}\" Addresses.API /ip4/{ApiUri.Host}/tcp/{ApiUri.Port}", throwOnError: true); + + if (GatewayUriMode == ConfigMode.OverwriteExisting) + RunExecutable(_kuboBinaryFile, $"config --repo-dir \"{RepoPath}\" Addresses.Gateway /ip4/{GatewayUri.Host}/tcp/{GatewayUri.Port}", throwOnError: true); + + if (GatewayUriMode == ConfigMode.UseExisting) + { + var existingGatewayUri = await GetGatewayAsync(cancellationToken); + if (existingGatewayUri is null) + RunExecutable(_kuboBinaryFile, $"config --repo-dir \"{RepoPath}\" Addresses.Gateway /ip4/{GatewayUri.Host}/tcp/{GatewayUri.Port}", throwOnError: true); + } + + if (ApiUriMode == ConfigMode.UseExisting) + { + var existingApiUri = await GetApiAsync(cancellationToken); + if (existingApiUri is null) + RunExecutable(_kuboBinaryFile, $"config --repo-dir \"{RepoPath}\" Addresses.API /ip4/{ApiUri.Host}/tcp/{ApiUri.Port}", throwOnError: true); + } + } + + /// + /// Initializes the local node with the provided routing settings. + /// + /// A that represents the asynchronous operation. + protected virtual async Task ApplyRoutingSettingsAsync(CancellationToken cancellationToken) + { + Guard.IsNotNullOrWhiteSpace(_kuboBinaryFile?.Path); + + RunExecutable(_kuboBinaryFile, $"config --repo-dir \"{RepoPath}\" Routing.Type {RoutingMode.ToString().ToLowerInvariant()}", throwOnError: true); + + RunExecutable(_kuboBinaryFile, $"config --repo-dir \"{RepoPath}\" Routing.AcceleratedDHTClient \"{UseAcceleratedDHTClient.ToString().ToLower()}\" --json", throwOnError: true); } - private void SetExecutablePermissionsForBinary(SystemFile file) + /// + /// Sets executable permissions for the given file. + /// + /// The file to adjust execute permissions for. + protected virtual void SetExecutablePermissionsForBinary(SystemFile file) { // This should only be used on linux. if (!RuntimeInformation.IsOSPlatform(OSPlatform.Linux)) @@ -217,7 +548,14 @@ private void SetExecutablePermissionsForBinary(SystemFile file) RunExecutable(new SystemFile("/bin/bash"), $"-c \"chmod 777 '{file.Path}'\"", throwOnError: true); } - private void RunExecutable(SystemFile file, string arguments, bool throwOnError) + /// + /// Runs the provided executable with the given arguments. + /// + /// The binary file to execute. + /// The execution arguments to provide. + /// Whether to throw when stderr is emitted. + /// + protected void RunExecutable(SystemFile file, string arguments, bool throwOnError) { var processStartInfo = new ProcessStartInfo(file.Path, arguments) { diff --git a/src/MfsFile.cs b/src/MfsFile.cs index 064d0c5..00540be 100644 --- a/src/MfsFile.cs +++ b/src/MfsFile.cs @@ -1,10 +1,7 @@ using CommunityToolkit.Diagnostics; using Ipfs; -using Ipfs.Http; -using OwlCore.Kubo.Models; +using Ipfs.CoreApi; using OwlCore.Storage; -using System.Text; -using System.Text.Json; namespace OwlCore.Kubo; @@ -17,8 +14,8 @@ public class MfsFile : IFile, IChildFile, IGetCid /// Creates a new instance of . /// /// - /// The IPFS Client to use for retrieving the content. - public MfsFile(string path, IpfsClient client) + /// The client to use for interacting with ipfs. + public MfsFile(string path, ICoreApi client) { Guard.IsNotNullOrWhiteSpace(path); Path = path; @@ -28,7 +25,7 @@ public MfsFile(string path, IpfsClient client) static string GetFolderItemName(string path) { var parts = path.Trim('/').Split('/').ToArray(); - return parts[parts.Length - 1]; + return parts[^1]; } } @@ -52,18 +49,15 @@ static string GetFolderItemName(string path) /// /// The IPFS Client to use for retrieving the content. /// - protected IpfsClient Client { get; } + protected ICoreApi Client { get; } /// public async Task OpenStreamAsync(FileAccess accessMode = FileAccess.Read, CancellationToken cancellationToken = default) { - var serialized = await Client.DoCommandAsync("files/stat", cancellationToken, Path, "long=true"); - var result = await JsonSerializer.DeserializeAsync(new MemoryStream(Encoding.UTF8.GetBytes(serialized)), typeof(MfsFileStatData), ModelSerializer.Default, cancellationToken); + var data = await Client.Mfs.StatAsync(Path, cancellationToken); - Guard.IsNotNull(result); - - var data = (MfsFileStatData)result; - Guard.IsNotNullOrWhiteSpace(data.Hash); + Guard.IsNotNull(data); + Guard.IsNotNullOrWhiteSpace(data.Hash?.ToString()); Guard.IsNotNull(data.Size); return new MfsStream(Path, (long)data.Size, Client) { InternalCanWrite = accessMode.HasFlag(FileAccess.Write) }; @@ -75,15 +69,10 @@ public async Task OpenStreamAsync(FileAccess accessMode = FileAccess.Rea /// A Task that represents the asynchronous operation. Value is the CID of the file that was flushed to disk. public async Task FlushAsync(CancellationToken cancellationToken = default) { - var serialized = await Client.DoCommandAsync("files/flush", cancellationToken, Path); - Guard.IsNotNullOrWhiteSpace(serialized); - - var result = (FilesFlushResponse?)await JsonSerializer.DeserializeAsync(new MemoryStream(Encoding.UTF8.GetBytes(serialized)), typeof(FilesFlushResponse), ModelSerializer.Default, cancellationToken); - - // This field is always present if the operation was successful. - Guard.IsNotNullOrWhiteSpace(result?.Cid); + var cid = await Client.Mfs.FlushAsync(Path, cancellationToken); + Guard.IsNotNullOrWhiteSpace(cid); - return result.Cid; + return cid; } /// diff --git a/src/MfsFolder.Modifiable.cs b/src/MfsFolder.Modifiable.cs index c70da3f..8dcf83a 100644 --- a/src/MfsFolder.Modifiable.cs +++ b/src/MfsFolder.Modifiable.cs @@ -2,94 +2,120 @@ using OwlCore.Kubo.FolderWatchers; using OwlCore.Storage; -namespace OwlCore.Kubo +namespace OwlCore.Kubo; + +public partial class MfsFolder : IModifiableFolder, IMoveFrom, ICreateCopyOf { - public partial class MfsFolder : IModifiableFolder, IFastFileMove, IFastFileCopy, IFastFileCopy, IFastFileCopy + /// + /// The interval that MFS should be checked for updates. + /// + public TimeSpan UpdateCheckInterval { get; } = TimeSpan.FromSeconds(10); + + /// + public virtual async Task DeleteAsync(IStorableChild item, CancellationToken cancellationToken = default) { - /// - /// The interval that MFS should be checked for updates. - /// - public TimeSpan UpdateCheckInterval { get; } = TimeSpan.FromSeconds(10); + cancellationToken.ThrowIfCancellationRequested(); + Guard.IsNotNullOrWhiteSpace(item.Name); - /// - public virtual async Task DeleteAsync(IStorableChild item, CancellationToken cancellationToken = default) - { - cancellationToken.ThrowIfCancellationRequested(); - Guard.IsNotNullOrWhiteSpace(item.Name); + await Client.Mfs.RemoveAsync($"{Path}{item.Name}", recursive: true, force: true, cancellationToken); + } - await Client.DoCommandAsync("files/rm", cancellationToken, $"{Path}{item.Name}", "recursive=true", "force=true"); - } + /// + public Task CreateCopyOfAsync(IFile fileToCopy, bool overwrite, CancellationToken cancellationToken, + CreateCopyOfDelegate fallback) + { + if (fileToCopy is MfsFile mfsFile) + return CreateCopyOfAsync(mfsFile, overwrite, cancellationToken); - /// - public virtual async Task CreateCopyOfAsync(MfsFile fileToCopy, bool overwrite = false, CancellationToken cancellationToken = default) - { - cancellationToken.ThrowIfCancellationRequested(); + if (fileToCopy is IpfsFile ipfsFile) + return CreateCopyOfAsync(ipfsFile, overwrite, cancellationToken); - await Client.DoCommandAsync("files/cp", cancellationToken, arg: fileToCopy.Path, $"arg={Path}"); - return new MfsFile($"{Path}{fileToCopy.Name}", Client); - } + if (fileToCopy is IpnsFile ipnsFile) + return CreateCopyOfAsync(ipnsFile, overwrite, cancellationToken); - /// - public virtual async Task CreateCopyOfAsync(IpfsFile fileToCopy, bool overwrite = false, CancellationToken cancellationToken = default) - { - cancellationToken.ThrowIfCancellationRequested(); + return fallback(this, fileToCopy, overwrite, cancellationToken); + } - await Client.DoCommandAsync("files/cp", cancellationToken, arg: $"/ipfs/{fileToCopy.Id}", $"arg={Path}"); - return new MfsFile($"{Path}{fileToCopy.Name}", Client); - } + /// + public Task MoveFromAsync(IChildFile fileToMove, IModifiableFolder source, bool overwrite, CancellationToken cancellationToken, + MoveFromDelegate fallback) + { + if (fileToMove is MfsFile mfsFile) + return MoveFromAsync(mfsFile, source, overwrite, cancellationToken); - /// - public virtual async Task CreateCopyOfAsync(IpnsFile fileToCopy, bool overwrite = false, CancellationToken cancellationToken = default) - { - cancellationToken.ThrowIfCancellationRequested(); + return fallback(this, fileToMove, source, overwrite, cancellationToken); + } - var cid = await fileToCopy.GetCidAsync(cancellationToken); - await Client.DoCommandAsync("files/cp", cancellationToken, arg: $"/ipfs/{cid}", $"arg={Path}"); + /// + public virtual async Task CreateCopyOfAsync(MfsFile fileToCopy, bool overwrite = false, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); - return new MfsFile($"{Path}{fileToCopy.Name}", Client); - } + await Client.Mfs.CopyAsync(fileToCopy.Path, Path, cancel: cancellationToken); + return new MfsFile($"{Path}{fileToCopy.Name}", Client); + } - /// - public virtual async Task MoveFromAsync(MfsFile fileToMove, IModifiableFolder source, bool overwrite = false, CancellationToken cancellationToken = default) - { - cancellationToken.ThrowIfCancellationRequested(); + /// + public virtual async Task CreateCopyOfAsync(IpfsFile fileToCopy, bool overwrite = false, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); - await Client.DoCommandAsync("files/mv", cancellationToken, arg: fileToMove.Path, $"arg={Path}{fileToMove.Name}"); - return new MfsFile($"{Path}{fileToMove.Name}", Client); - } + await Client.Mfs.CopyAsync($"/ipfs/{fileToCopy.Id}", Path, cancel: cancellationToken); + return new MfsFile($"{Path}{fileToCopy.Name}", Client); + } - /// - public virtual async Task CreateFolderAsync(string name, bool overwrite = false, CancellationToken cancellationToken = default) + /// + public virtual async Task CreateCopyOfAsync(IpnsFile fileToCopy, bool overwrite = false, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var cid = await fileToCopy.GetCidAsync(cancellationToken); + await Client.Mfs.CopyAsync($"/ipfs/{cid}", Path, cancel: cancellationToken); + + return new MfsFile($"{Path}{fileToCopy.Name}", Client); + } + + /// + public virtual async Task MoveFromAsync(MfsFile fileToMove, IModifiableFolder source, bool overwrite = false, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + await Client.Mfs.MoveAsync(fileToMove.Path, $"{Path}{fileToMove.Name}", cancellationToken); + return new MfsFile($"{Path}{fileToMove.Name}", Client); + } + + /// + public virtual async Task CreateFolderAsync(string name, bool overwrite = false, CancellationToken cancellationToken = default) + { + if (overwrite) { - if (overwrite) - { - await Client.DoCommandAsync("files/rm", cancellationToken, $"{Path}{name}", "recursive=true"); - } - - try - { - await Client.DoCommandAsync("files/mkdir", cancellationToken, arg: $"{Path}{name}"); - } - catch (Exception ex) when (ex.Message.ToLower().Contains("file already exists")) - { - // Ignored, return existing path if exists - } - - return new MfsFolder($"{Path}{name}", Client); + await Client.Mfs.RemoveAsync($"{Path}{name}", recursive: true, force: true, cancellationToken); } - /// - public virtual async Task CreateFileAsync(string name, bool overwrite = false, CancellationToken cancellationToken = default) + try { - await Client.UploadAsync("files/write", CancellationToken.None, new MemoryStream(), null, $"arg={Path}{name}", $"create=true", overwrite ? $"truncate=true" : string.Empty); - - return new MfsFile($"{Path}{name}", Client); + await Client.Mfs.MakeDirectoryAsync($"{Path}{name}", cancel: cancellationToken); } - - /// - public virtual Task GetFolderWatcherAsync(CancellationToken cancellationToken = default) + catch (Exception ex) when (ex.Message.ToLower().Contains("file already exists")) { - return Task.FromResult(new TimerBasedMfsWatcher(Client, this, UpdateCheckInterval)); + // Ignored, return existing path if exists } + + return new MfsFolder($"{Path}{name}", Client); + } + + /// + public virtual async Task CreateFileAsync(string name, bool overwrite = false, CancellationToken cancellationToken = default) + { + await Client.Mfs.WriteAsync($"{Path}{name}", new MemoryStream(), new() { Create = true, Truncate = overwrite }); + + return new MfsFile($"{Path}{name}", Client); + } + + + /// + public virtual Task GetFolderWatcherAsync(CancellationToken cancellationToken = default) + { + return Task.FromResult(new TimerBasedMfsWatcher(Client, this, UpdateCheckInterval)); } -} +} \ No newline at end of file diff --git a/src/MfsFolder.cs b/src/MfsFolder.cs index 09675a1..f030356 100644 --- a/src/MfsFolder.cs +++ b/src/MfsFolder.cs @@ -1,5 +1,6 @@ using CommunityToolkit.Diagnostics; using Ipfs; +using Ipfs.CoreApi; using Ipfs.Http; using OwlCore.Kubo.Models; using OwlCore.Storage; @@ -7,151 +8,133 @@ using System.Text; using System.Text.Json; -namespace OwlCore.Kubo +namespace OwlCore.Kubo; + +/// +/// A folder that resides in Kubo's Mutable Filesystem. +/// +public partial class MfsFolder : IFolder, IChildFolder, IGetItem, IGetItemRecursive, IGetFirstByName, IGetRoot, IGetCid { /// - /// A folder that resides in Kubo's Mutable Filesystem. + /// Creates a new instance of . /// - public partial class MfsFolder : IFolder, IChildFolder, IFastGetItem, IFastGetItemRecursive, IFastGetFirstByName, IFastGetRoot, IGetCid + /// The IPFS api to use for retrieving the content. + /// The MFS path to the folder. + public MfsFolder(string path, ICoreApi client) { - /// - /// Creates a new instance of . - /// - /// The IPFS Client to use for retrieving the content. - /// The MFS path to the folder. - public MfsFolder(string path, IpfsClient client) - { - Guard.IsNotNullOrWhiteSpace(path); - - // Add trailing slash if missing. - if (!path.EndsWith("/")) - path += "/"; - - Path = path; - Id = path; - Client = client; - Name = PathHelpers.GetFolderItemName(path); - } - - /// - public virtual string Id { get; } + Guard.IsNotNullOrWhiteSpace(path); - /// - public string Name { get; } + // Add trailing slash if missing. + if (!path.EndsWith("/")) + path += "/"; - /// - /// The MFS path to the file. Relative to the root of MFS. - /// - public string Path { get; } + Path = path; + Id = path; + Client = client; + Name = PathHelpers.GetFolderItemName(path); + } - /// - /// The IPFS Client to use for retrieving the content. - /// - protected IpfsClient Client { get; } + /// + public virtual string Id { get; } - /// - public virtual async IAsyncEnumerable GetItemsAsync(StorableType type = StorableType.All, [EnumeratorCancellation] CancellationToken cancellationToken = default) - { - var serialized = await Client.DoCommandAsync("files/ls", cancellationToken, Path, "long=true"); - var result = await JsonSerializer.DeserializeAsync(new MemoryStream(Encoding.UTF8.GetBytes(serialized)), typeof(MfsFileContentsBody), ModelSerializer.Default, cancellationToken); + /// + public string Name { get; } - Guard.IsNotNull(result); + /// + /// The MFS path to the file. Relative to the root of MFS. + /// + public string Path { get; } - var data = (MfsFileContentsBody)result; + /// + /// The IPFS Client to use for retrieving the content. + /// + protected ICoreApi Client { get; } - foreach (var link in data.Entries ?? Enumerable.Empty()) - { - Guard.IsNotNullOrWhiteSpace(link.Hash); - var linkedItemInfo = await Client.FileSystem.ListFileAsync(link.Hash, cancellationToken); - - if (linkedItemInfo.IsDirectory) - { - if (type.HasFlag(StorableType.Folder)) - yield return new MfsFolder($"{Path}{link.Name}", Client); - } - else - { - if (type.HasFlag(StorableType.File)) - yield return new MfsFile($"{Path}{link.Name}", Client); - } - } - } + /// + public virtual async IAsyncEnumerable GetItemsAsync(StorableType type = StorableType.All, [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + var result = await Client.Mfs.ListAsync(Path, cancel: cancellationToken); - /// - public virtual async Task GetFirstByNameAsync(string name, CancellationToken cancellationToken = new CancellationToken()) + foreach (var link in result ?? Enumerable.Empty()) { - var mfsPath = $"{Id}{name}"; + Guard.IsNotNullOrWhiteSpace(link.Id); + var linkedItemInfo = await Client.Mfs.StatAsync($"/ipfs/{link.Id}", cancellationToken); - try + if (linkedItemInfo.IsDirectory) { - Guard.IsNotNullOrWhiteSpace(name); - - var serialized = await Client.DoCommandAsync("files/stat", cancellationToken, mfsPath, "long=true"); - var result = await JsonSerializer.DeserializeAsync(new MemoryStream(Encoding.UTF8.GetBytes(serialized)), typeof(MfsFileStatData), ModelSerializer.Default, cancellationToken); - - Guard.IsNotNull(result); - - var data = (MfsFileStatData)result; - Guard.IsNotNullOrWhiteSpace(data.Type); - - return data.Type == "directory" ? new MfsFolder(mfsPath, Client) : new MfsFile(mfsPath, Client); + if (type.HasFlag(StorableType.Folder)) + yield return new MfsFolder($"{Path}{link.Name}", Client); } - catch (HttpRequestException httpRequestException) when (httpRequestException.Message.Contains("file does not exist")) + else { - throw new FileNotFoundException(); + if (type.HasFlag(StorableType.File)) + yield return new MfsFile($"{Path}{link.Name}", Client); } } + } - /// - public virtual Task GetParentAsync(CancellationToken cancellationToken = default) => Task.FromResult(new MfsFolder(PathHelpers.GetParentPath(Path), Client)); - - /// - public virtual Task GetRootAsync() => Task.FromResult(new MfsFolder("/", Client)); + /// + public virtual async Task GetFirstByNameAsync(string name, CancellationToken cancellationToken = new CancellationToken()) + { + var mfsPath = $"{Id}{name}"; - /// - public virtual async Task GetItemAsync(string id, CancellationToken cancellationToken = default) + try { - try - { - Guard.IsNotNullOrWhiteSpace(id); + Guard.IsNotNullOrWhiteSpace(name); - var serialized = await Client.DoCommandAsync("files/stat", cancellationToken, id, "long=true"); - var result = await JsonSerializer.DeserializeAsync(new MemoryStream(Encoding.UTF8.GetBytes(serialized)), typeof(MfsFileStatData), ModelSerializer.Default, cancellationToken); + var data = await Client.Mfs.StatAsync(mfsPath, cancellationToken); - Guard.IsNotNull(result); + return data.IsDirectory ? new MfsFolder(mfsPath, Client) : new MfsFile(mfsPath, Client); + } + catch (HttpRequestException httpRequestException) when (httpRequestException.Message.Contains("file does not exist")) + { + throw new FileNotFoundException(); + } + } - var data = (MfsFileStatData)result; - Guard.IsNotNullOrWhiteSpace(data.Type); + /// + public virtual Task GetParentAsync(CancellationToken cancellationToken = default) => Task.FromResult(new MfsFolder(PathHelpers.GetParentPath(Path), Client)); - return data.Type == "directory" ? new MfsFolder(id, Client) : new MfsFile(id, Client); - } - catch (HttpRequestException httpRequestException) when (httpRequestException.Message.Contains("file does not exist")) - { - throw new FileNotFoundException(); - } - } + /// + public virtual Task GetRootAsync(CancellationToken cancellationToken = default) => Task.FromResult(new MfsFolder("/", Client)); + + /// + public virtual async Task GetItemAsync(string id, CancellationToken cancellationToken = default) + { + try + { + Guard.IsNotNullOrWhiteSpace(id); - /// - public virtual Task GetItemRecursiveAsync(string id, CancellationToken cancellationToken = default) => GetItemAsync(id, cancellationToken); + var data = await Client.Mfs.StatAsync(id, cancellationToken); - /// - /// Flushes the file contents to disk and returns the CID of the folder contents. - /// - /// A Task that represents the asynchronous operation. Value is the CID of the file that was flushed to disk. - public virtual async Task FlushAsync(CancellationToken cancellationToken = default) + return data.IsDirectory ? new MfsFolder(id, Client) : new MfsFile(id, Client); + } + catch (HttpRequestException httpRequestException) when (httpRequestException.Message.Contains("file does not exist")) { - var serialized = await Client.DoCommandAsync("files/flush", cancellationToken, Path); - Guard.IsNotNullOrWhiteSpace(serialized); + throw new FileNotFoundException(); + } + } - var result = (FilesFlushResponse?)await JsonSerializer.DeserializeAsync(new MemoryStream(Encoding.UTF8.GetBytes(serialized)), typeof(FilesFlushResponse), ModelSerializer.Default, cancellationToken); + /// + public virtual Task GetItemRecursiveAsync(string id, CancellationToken cancellationToken = default) => GetItemAsync(id, cancellationToken); - // This field is always present if the operation was successful. - Guard.IsNotNullOrWhiteSpace(result?.Cid); + /// + /// Flushes the file contents to disk and returns the CID of the folder contents. + /// + /// A Task that represents the asynchronous operation. Value is the CID of the file that was flushed to disk. + public virtual async Task FlushAsync(CancellationToken cancellationToken = default) + { + var serialized = await Client.Mfs.FlushAsync(Path, cancellationToken); + Guard.IsNotNullOrWhiteSpace(serialized); - return result.Cid; - } + var result = (FilesFlushResponse?)await JsonSerializer.DeserializeAsync(new MemoryStream(Encoding.UTF8.GetBytes(serialized)), typeof(FilesFlushResponse), ModelSerializer.Default, cancellationToken); - /// - public Task GetCidAsync(CancellationToken cancellationToken) => FlushAsync(cancellationToken); + // This field is always present if the operation was successful. + Guard.IsNotNullOrWhiteSpace(result?.Cid); + + return result.Cid; } -} + + /// + public Task GetCidAsync(CancellationToken cancellationToken) => FlushAsync(cancellationToken); +} \ No newline at end of file diff --git a/src/MfsStream.cs b/src/MfsStream.cs index e725aa6..54a0a96 100644 --- a/src/MfsStream.cs +++ b/src/MfsStream.cs @@ -1,6 +1,8 @@ using CommunityToolkit.Common; using CommunityToolkit.Diagnostics; -using Ipfs.Http; +using Ipfs.CoreApi; +using System.Text; +using OwlCore.Extensions; namespace OwlCore.Kubo; @@ -18,7 +20,7 @@ public class MfsStream : Stream /// The MFS path of the file. /// The known length of the stream. /// The client to use for interacting with IPFS. - public MfsStream(string path, long length, IpfsClient client) + public MfsStream(string path, long length, ICoreApi client) { _path = path; _length = length; @@ -28,7 +30,7 @@ public MfsStream(string path, long length, IpfsClient client) /// /// The IPFS Client to use for retrieving the content. /// - public IpfsClient Client { get; } + public ICoreApi Client { get; } /// public override bool CanRead => true; @@ -51,13 +53,13 @@ public MfsStream(string path, long length, IpfsClient client) /// public override void Flush() { - _ = Client.DoCommandAsync("files/flush", CancellationToken.None, _path).Result; + _ = Client.Mfs.FlushAsync(_path).Result; } /// public override Task FlushAsync(CancellationToken cancellationToken) { - return Client.DoCommandAsync("files/flush", cancellationToken, _path); + return Client.Mfs.FlushAsync(_path, cancellationToken); } /// @@ -72,11 +74,8 @@ public override async Task ReadAsync(byte[] buffer, int offset, int count, Guard.IsLessThanOrEqualTo(offset + count, Length); Guard.IsGreaterThanOrEqualTo(offset, 0); - var result = await Client.PostDownloadAsync("files/read", cancellationToken, _path, $"offset={Position + offset}", $"count={count}"); - - using var memStream = new MemoryStream(); - await result.CopyToAsync(memStream); - var bytes = memStream.ToArray(); + var result = await Client.Mfs.ReadFileStreamAsync(_path, offset: Position + offset, count: count, cancellationToken); + var bytes = await result.ToBytesAsync(cancellationToken); for (var i = 0; i < bytes.Length; i++) buffer[i] = bytes[i]; @@ -151,7 +150,7 @@ public override async Task WriteAsync(byte[] buffer, int offset, int count, Canc SetLength(Position + count); } - await Client.Upload2Async("files/write", cancellationToken, new MemoryStream(buffer, offset, count), GetFileName(_path), $"arg={_path}", $"offset={Position}", $"count={count}", $"create=true"); + await Client.Mfs.WriteAsync(_path, buffer, new() { Offset = Position, Count = count, Create = true }, cancellationToken); Position += count; } diff --git a/src/Models/FilesFlushResponse.cs b/src/Models/FilesFlushResponse.cs index 00ab4ab..af5c419 100644 --- a/src/Models/FilesFlushResponse.cs +++ b/src/Models/FilesFlushResponse.cs @@ -1,7 +1,6 @@ -namespace OwlCore.Kubo.Models +namespace OwlCore.Kubo.Models; + +internal class FilesFlushResponse { - internal class FilesFlushResponse - { - public string? Cid { get; set; } - } -} + public string? Cid { get; set; } +} \ No newline at end of file diff --git a/src/Models/MfsFileContentsBody.cs b/src/Models/MfsFileContentsBody.cs index baf880b..8361ad0 100644 --- a/src/Models/MfsFileContentsBody.cs +++ b/src/Models/MfsFileContentsBody.cs @@ -1,7 +1,6 @@ -namespace OwlCore.Kubo.Models +namespace OwlCore.Kubo.Models; + +internal class MfsFileContentsBody { - internal class MfsFileContentsBody - { - public List? Entries { get; set; } - } -} + public List? Entries { get; set; } +} \ No newline at end of file diff --git a/src/Models/MfsFileData.cs b/src/Models/MfsFileData.cs index 4200141..df138a6 100644 --- a/src/Models/MfsFileData.cs +++ b/src/Models/MfsFileData.cs @@ -1,13 +1,12 @@ -namespace OwlCore.Kubo.Models +namespace OwlCore.Kubo.Models; + +internal class MfsFileData { - internal class MfsFileData - { - public string? Hash { get; set; } + public string? Hash { get; set; } - public string? Name { get; set; } + public string? Name { get; set; } - public int? Type { get; set; } + public int? Type { get; set; } - public long? Size { get; set; } - } -} + public long? Size { get; set; } +} \ No newline at end of file diff --git a/src/Models/MfsFileStatData.cs b/src/Models/MfsFileStatData.cs index 465cff8..3c7e956 100644 --- a/src/Models/MfsFileStatData.cs +++ b/src/Models/MfsFileStatData.cs @@ -1,15 +1,14 @@ -namespace OwlCore.Kubo.Models +namespace OwlCore.Kubo.Models; + +internal class MfsFileStatData { - internal class MfsFileStatData - { - public int? Blocks { get; set; } - public ulong? CumulativeSize { get; set; } - public bool? Local { get; set; } - public ulong? SizeLocal { get; set; } - public bool? WithLocality { get; set; } - public string? Hash { get; set; } - public string? Name { get; set; } - public string? Type { get; set; } - public ulong? Size { get; set; } - } -} + public int? Blocks { get; set; } + public ulong? CumulativeSize { get; set; } + public bool? Local { get; set; } + public ulong? SizeLocal { get; set; } + public bool? WithLocality { get; set; } + public string? Hash { get; set; } + public string? Name { get; set; } + public string? Type { get; set; } + public ulong? Size { get; set; } +} \ No newline at end of file diff --git a/src/Models/ModelSerializer.cs b/src/Models/ModelSerializer.cs index ea154ba..92aa75f 100644 --- a/src/Models/ModelSerializer.cs +++ b/src/Models/ModelSerializer.cs @@ -1,13 +1,12 @@ using System.Text.Json.Serialization; -namespace OwlCore.Kubo.Models +namespace OwlCore.Kubo.Models; + +[JsonSourceGenerationOptions(WriteIndented = true)] +[JsonSerializable(typeof(MfsFileData))] +[JsonSerializable(typeof(MfsFileContentsBody))] +[JsonSerializable(typeof(MfsFileStatData))] +[JsonSerializable(typeof(FilesFlushResponse))] +internal partial class ModelSerializer : JsonSerializerContext { - [JsonSourceGenerationOptions(WriteIndented = true)] - [JsonSerializable(typeof(MfsFileData))] - [JsonSerializable(typeof(MfsFileContentsBody))] - [JsonSerializable(typeof(MfsFileStatData))] - [JsonSerializable(typeof(FilesFlushResponse))] - internal partial class ModelSerializer : JsonSerializerContext - { - } -} +} \ No newline at end of file diff --git a/src/Models/PublishedMessage.cs b/src/Models/PublishedMessage.cs index 6028f51..10ef3f2 100644 --- a/src/Models/PublishedMessage.cs +++ b/src/Models/PublishedMessage.cs @@ -1,5 +1,5 @@ -using System.Text; -using Ipfs; +using Ipfs; +using System.Text; namespace OwlCore.Kubo.Models; diff --git a/src/OwlCore.Kubo.csproj b/src/OwlCore.Kubo.csproj index c41e166..0578340 100644 --- a/src/OwlCore.Kubo.csproj +++ b/src/OwlCore.Kubo.csproj @@ -2,7 +2,7 @@ netstandard2.0;net6.0;net7.0 enable - 10.0 + 12.0 nullable true false @@ -14,13 +14,44 @@ $(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb Arlo Godfrey - 0.15.1 + 0.16.0 OwlCore An essential toolkit for Kubo, IPFS and the distributed web. LICENSE.txt +--- 0.16.0 --- +[Breaking] +Inherited breaking changes from IpfsShipyard.Net.Http.Client 0.2.0. +Inherited breaking changes from OwlCore.Storage 0.10.0 and 0.11.0. +Inherited breaking changes from OwlCore.ComponentModel 0.7.0 and 0.8.0. +Inherited breaking changes from OwlCore.Extensions 0.8.0. + +[Fixes] +The changes in OwlCore.Storage 0.10.0 allowed the CommonTests to uncover previously unknown issues with the Kubo-based storage implementation. These issues are now fixed and the storage implementations are compliant with the latest CommonTests. + +[New] +Added static SwarmKeyGen extension methods, which allow you to generate and write a private Kubo swarm key to a file for immediate use. +Added a new PrivateKuboBootstrapper. This custom bootstrapper allows you to start a Kubo node with a private swarm key, automatically removing the default bootstrap nodes, applying LIBP2P_FORCE_PNET as needed, and setting up the provided BootstrapPeerMultiAddresses as your bootstrap peers. +Added a new OwlCore.Kubo.Cache namespace. This is a limited set of API wrappers for the Ipfs core interfaces that enable caching functionality throughout your entire application domain, currently covering IKeyApi and INameApi. Note that you'll need to use ICoreApi instead of IpfsClient, and likewise for other Core interfaces (and their implementations) throughout your codebase to use this cache layer. + +Added TransformIpnsDagAsync extension method, which allows you to mutate and update a DAG object published to IPNS all in one go, with progress reporting. +Added ResolveDagCidAsync extension method to Cid and IEnumerable{Cid}. Resolves the provided cid if it is an Ipns address and retrieves the content from the DAG. +Added CreateKeyWithNameOfIdAsync extension method to IKeyApi. Creates an ipns key using a temp name, then renames it to the name of the key. Enables pushing to ipns without additional API calls to convert between ipns cid and name. +Added GetOrCreateKeyAsync extension method to IKeyApi. Gets a key by name, or creates it if it does not exist. + +Added various helper methods to KuboBootstrapper for getting repo lock state, gateway and api uri, and converting between MultiAddress and Uri. +Added all missing DhtRoutingMode values that have been added to Kubo as of 0.26.0. +Added a LaunchConfigMode to KuboBootstrapper. Defines the behavior when a node is already running (when the repo is locked): Throw, Attach, or Relaunch. +Added an ApiUriMode to KuboBootstrapper. Gets or sets an enum that determines how to use the supplied ApiUri: UseExisting, or OverwriteExisting. +Added a GatewayUriMode to KuboBootstrapper. Gets or sets an enum that determines how to use the supplied GatewayUri: UseExisting, or OverwriteExisting. +Added a UseAcceleratedDHTClient property to KuboBootstrapper. If set to true, the client DHT will be used on startup. + +[Improvements] +As part of the move to ICoreApi, all internal calls to DoCommandAsync have been removed wherever IMfsApi is used, thanks to the implementation contributed in https://github.com/ipfs-shipyard/net-ipfs-core/pull/13/. +Updated dependencies. + --- 0.15.1 --- [Fixes] Fixed an issue where MfsFolder would throw StackOverflowException when calling GetParentAsync on a folder two or more subdirectories in. Added tests. @@ -351,14 +382,20 @@ Added unit tests. - - + + - - - + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + - + \ No newline at end of file diff --git a/src/PathHelpers.cs b/src/PathHelpers.cs index 8794a5e..137948c 100644 --- a/src/PathHelpers.cs +++ b/src/PathHelpers.cs @@ -1,46 +1,45 @@ -namespace OwlCore.Kubo +namespace OwlCore.Kubo; + +/// +/// A collection of helpers for working with paths. +/// +public static class PathHelpers { - /// - /// A collection of helpers for working with paths. - /// - public static class PathHelpers + internal static string GetFolderItemName(string path) { - internal static string GetFolderItemName(string path) - { - var parts = path.Trim('/').Split('/').ToArray(); - return parts[^1]; - } + var parts = path.Trim('/').Split('/').ToArray(); + return parts[^1]; + } - internal static string GetParentPath(string relativePath) - { - // If the provided path is the root. - if (relativePath.Trim('/').Split('/').Count() == 1) - return "/"; + internal static string GetParentPath(string relativePath) + { + // If the provided path is the root. + if (relativePath.Trim('/').Split('/').Count() == 1) + return "/"; - var directorySeparatorChar = '/'; + var directorySeparatorChar = '/'; - // Path.GetDirectoryName() treats strings that end with a directory separator as a directory. If there's no trailing slash, it's treated as a file. - var isFolder = relativePath.EndsWith(directorySeparatorChar.ToString()); + // Path.GetDirectoryName() treats strings that end with a directory separator as a directory. If there's no trailing slash, it's treated as a file. + var isFolder = relativePath.EndsWith(directorySeparatorChar.ToString()); - // Run it twice for folders. The first time only shaves off the trailing directory separator. - var parentDirectoryName = isFolder ? System.IO.Path.GetDirectoryName(System.IO.Path.GetDirectoryName(relativePath)) : System.IO.Path.GetDirectoryName(relativePath); + // Run it twice for folders. The first time only shaves off the trailing directory separator. + var parentDirectoryName = isFolder ? Path.GetDirectoryName(Path.GetDirectoryName(relativePath)) : Path.GetDirectoryName(relativePath); - // It also doesn't return a string that has a path separator at the end. - return parentDirectoryName?.Replace('\\', '/') + (isFolder ? directorySeparatorChar : string.Empty) ?? string.Empty; - } + // It also doesn't return a string that has a path separator at the end. + return parentDirectoryName?.Replace('\\', '/') + (isFolder ? directorySeparatorChar : string.Empty) ?? string.Empty; + } - internal static string GetParentDirectoryName(string relativePath) - { - // If the provided path is the root. - if (System.IO.Path.GetPathRoot(relativePath)?.Replace('\\', '/') == relativePath) - return relativePath; + internal static string GetParentDirectoryName(string relativePath) + { + // If the provided path is the root. + if (Path.GetPathRoot(relativePath)?.Replace('\\', '/') == relativePath) + return relativePath; - var directorySeparatorChar = System.IO.Path.DirectorySeparatorChar; + var directorySeparatorChar = Path.DirectorySeparatorChar; - var parentPath = GetParentPath(relativePath); - var parentParentPath = GetParentPath(parentPath); + var parentPath = GetParentPath(relativePath); + var parentParentPath = GetParentPath(parentPath); - return parentPath.Replace(parentParentPath, "").TrimEnd(directorySeparatorChar); - } + return parentPath.Replace(parentParentPath, "").TrimEnd(directorySeparatorChar); } -} +} \ No newline at end of file diff --git a/src/PeerRoom.cs b/src/PeerRoom.cs index 30831d4..284b2da 100644 --- a/src/PeerRoom.cs +++ b/src/PeerRoom.cs @@ -81,11 +81,17 @@ private async void Timer_Elapsed(object? sender, System.Timers.ElapsedEventArgs public Peer ThisPeer { get; } /// - /// The topic being used for communication. + /// The name of the topic being used for communication. /// public string TopicName { get; } - internal bool HeartbeatEnabled { get; set; } = true; + /// + /// Gets or sets a boolean that indicates whether the heartbeat for this peer is enabled. + /// + /// + /// If disabled, other peers will not see this peer in the peer room because the heartbeat will not be broadcast. This can be useful when building specialized peer rooms. + /// + public bool HeartbeatEnabled { get; set; } = true; /// /// Broadcasts a heartbeat to listeners on the topic. diff --git a/src/Polyfills/DoesNotReturnAttribute.cs b/src/Polyfills/DoesNotReturnAttribute.cs deleted file mode 100644 index acd83c7..0000000 --- a/src/Polyfills/DoesNotReturnAttribute.cs +++ /dev/null @@ -1,17 +0,0 @@ -// -#pragma warning disable -#nullable enable annotations - -// Licensed to the .NET Foundation under one or more agreements. -// The .NET Foundation licenses this file to you under the MIT license. - -namespace System.Diagnostics.CodeAnalysis -{ - /// - /// Applied to a method that will never return under any circumstance. - /// - [global::System.AttributeUsage(global::System.AttributeTargets.Method, Inherited = false)] - internal sealed class DoesNotReturnAttribute : global::System.Attribute - { - } -} \ No newline at end of file diff --git a/src/Polyfills/Index.cs b/src/Polyfills/Index.cs deleted file mode 100644 index d411e4f..0000000 --- a/src/Polyfills/Index.cs +++ /dev/null @@ -1,155 +0,0 @@ -// -#pragma warning disable -#nullable enable annotations - -// Licensed to the .NET Foundation under one or more agreements. -// The .NET Foundation licenses this file to you under the MIT license. - -namespace System -{ - /// Represent a type can be used to index a collection either from the start or the end. - /// - /// Index is used by the C# compiler to support the new index syntax - /// - /// int[] someArray = new int[5] { 1, 2, 3, 4, 5 } ; - /// int lastElement = someArray[^1]; // lastElement = 5 - /// - /// - internal readonly struct Index : global::System.IEquatable - { - private readonly int _value; - - /// Construct an Index using a value and indicating if the index is from the start or from the end. - /// The index value. it has to be zero or positive number. - /// Indicating if the index is from the start or from the end. - /// - /// If the Index constructed from the end, index value 1 means pointing at the last element and index value 0 means pointing at beyond last element. - /// - [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)] - public Index(int value, bool fromEnd = false) - { - if (value < 0) - { - global::System.Index.ThrowHelper.ThrowValueArgumentOutOfRange_NeedNonNegNumException(); - } - - if (fromEnd) - _value = ~value; - else - _value = value; - } - - // The following private constructors mainly created for perf reason to avoid the checks - private Index(int value) - { - _value = value; - } - - /// Create an Index pointing at first element. - public static global::System.Index Start => new global::System.Index(0); - - /// Create an Index pointing at beyond last element. - public static global::System.Index End => new global::System.Index(~0); - - /// Create an Index from the start at the position indicated by the value. - /// The index value from the start. - [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)] - public static global::System.Index FromStart(int value) - { - if (value < 0) - { - global::System.Index.ThrowHelper.ThrowValueArgumentOutOfRange_NeedNonNegNumException(); - } - - return new global::System.Index(value); - } - - /// Create an Index from the end at the position indicated by the value. - /// The index value from the end. - [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)] - public static global::System.Index FromEnd(int value) - { - if (value < 0) - { - global::System.Index.ThrowHelper.ThrowValueArgumentOutOfRange_NeedNonNegNumException(); - } - - return new global::System.Index(~value); - } - - /// Returns the index value. - public int Value - { - get - { - if (_value < 0) - return ~_value; - else - return _value; - } - } - - /// Indicates whether the index is from the start or the end. - public bool IsFromEnd => _value < 0; - - /// Calculate the offset from the start using the giving collection length. - /// The length of the collection that the Index will be used with. length has to be a positive value - /// - /// For performance reason, we don't validate the input length parameter and the returned offset value against negative values. - /// we don't validate either the returned offset is greater than the input length. - /// It is expected Index will be used with collections which always have non negative length/count. If the returned offset is negative and - /// then used to index a collection will get out of range exception which will be same affect as the validation. - /// - [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)] - public int GetOffset(int length) - { - int offset = _value; - if (IsFromEnd) - { - // offset = length - (~value) - // offset = length + (~(~value) + 1) - // offset = length + value + 1 - - offset += length + 1; - } - return offset; - } - - /// Indicates whether the current Index object is equal to another object of the same type. - /// An object to compare with this object - public override bool Equals([global::System.Diagnostics.CodeAnalysis.NotNullWhen(true)] object? value) => value is global::System.Index && _value == ((global::System.Index)value)._value; - - /// Indicates whether the current Index object is equal to another Index object. - /// An object to compare with this object - public bool Equals(global::System.Index other) => _value == other._value; - - /// Returns the hash code for this instance. - public override int GetHashCode() => _value; - - /// Converts integer number to an Index. - public static implicit operator global::System.Index(int value) => FromStart(value); - - /// Converts the value of the current Index object to its equivalent string representation. - public override string ToString() - { - if (IsFromEnd) - return ToStringFromEnd(); - - return ((uint)Value).ToString(); - } - - private string ToStringFromEnd() - { - return '^' + Value.ToString(); - } - - private static class ThrowHelper - { - [global::System.Diagnostics.CodeAnalysis.DoesNotReturn] - public static void ThrowValueArgumentOutOfRange_NeedNonNegNumException() - { - throw new global::System.ArgumentOutOfRangeException("value", "Non-negative number required."); - } - } - } -} \ No newline at end of file diff --git a/src/Polyfills/IsExternalInit.cs b/src/Polyfills/IsExternalInit.cs deleted file mode 100644 index d2c4c3d..0000000 --- a/src/Polyfills/IsExternalInit.cs +++ /dev/null @@ -1,18 +0,0 @@ -// -#pragma warning disable -#nullable enable annotations - -// Licensed to the .NET Foundation under one or more agreements. -// The .NET Foundation licenses this file to you under the MIT license. - -namespace System.Runtime.CompilerServices -{ - /// - /// Reserved to be used by the compiler for tracking metadata. - /// This class should not be used by developers in source code. - /// - [global::System.ComponentModel.EditorBrowsable(global::System.ComponentModel.EditorBrowsableState.Never)] - internal static class IsExternalInit - { - } -} \ No newline at end of file diff --git a/src/Polyfills/NotNullWhenAttribute.cs b/src/Polyfills/NotNullWhenAttribute.cs deleted file mode 100644 index 8e680ee..0000000 --- a/src/Polyfills/NotNullWhenAttribute.cs +++ /dev/null @@ -1,28 +0,0 @@ -// -#pragma warning disable -#nullable enable annotations - -// Licensed to the .NET Foundation under one or more agreements. -// The .NET Foundation licenses this file to you under the MIT license. - -namespace System.Diagnostics.CodeAnalysis -{ - /// - /// Specifies that when a method returns , the parameter will not be null even if the corresponding type allows it. - /// - [global::System.AttributeUsage(global::System.AttributeTargets.Parameter, Inherited = false)] - internal sealed class NotNullWhenAttribute : global::System.Attribute - { - /// - /// Initializes the attribute with the specified return value condition. - /// - /// The return value condition. If the method returns this value, the associated parameter will not be null. - public NotNullWhenAttribute(bool returnValue) - { - ReturnValue = returnValue; - } - - /// Gets the return value condition. - public bool ReturnValue { get; } - } -} \ No newline at end of file diff --git a/src/PrivateKuboBootstrapper.cs b/src/PrivateKuboBootstrapper.cs new file mode 100644 index 0000000..e989fa2 --- /dev/null +++ b/src/PrivateKuboBootstrapper.cs @@ -0,0 +1,82 @@ +using CommunityToolkit.Diagnostics; +using Ipfs; +using OwlCore.Storage; + +namespace OwlCore.Kubo; + +/// +/// An easy-to-use bootstrapper for private Kubo swarms. +/// +public class PrivateKuboBootstrapper : KuboBootstrapper +{ + /// + /// Creates a new instance of . + /// + public PrivateKuboBootstrapper(string repoPath, Func> getKuboBinaryFile) + : base(repoPath, getKuboBinaryFile) + { + } + + /// + /// Creates a new instance of . + /// + public PrivateKuboBootstrapper(string repoPath, Version kuboVersion) + : base(repoPath, kuboVersion) + { + } + + /// + /// Creates a new instance of . + /// + public PrivateKuboBootstrapper(string repoPath) + : base(repoPath) + { + } + + /// + /// Gets or sets a bool indicating whether Kubo should force the use of private networks, and fail if no swarm key is found. Default is true. + /// + public bool Libp2pForcePnet { get; set; } = true; + + /// + /// The list of peers that will be bootstrapped instead of the default ones. These are the *trusted peers* from which to learn about other peers in the network. + /// + public required IEnumerable BootstrapPeerMultiAddresses { get; set; } + + /// + /// The behavior to use if the swarm key already exists in the repo. + /// + public required ConfigMode SwarmKeyConfigMode { get; set; } + + /// + /// The swarm key to use. + /// + /// + /// Generate a swarm key file via . + /// + public required IFile SwarmKeyFile { get; set; } + + /// + protected override async Task ApplyRoutingSettingsAsync(CancellationToken cancellationToken) + { + Guard.IsNotNull(KuboBinaryFile); + + // Copy swarm key to repo + await RepoFolder.CreateCopyOfAsync(SwarmKeyFile, overwrite: SwarmKeyConfigMode == ConfigMode.OverwriteExisting, cancellationToken); + + if (Libp2pForcePnet) + EnvironmentVariables["LIBP2P_FORCE_PNET"] = "1"; + + // Clear out all existing bootstrap peers + RunExecutable(KuboBinaryFile, $"bootstrap rm --all --repo-dir \"{RepoFolder.Path}\"", throwOnError: false); + + // Setup bootstrap peers + foreach (var multiAddress in BootstrapPeerMultiAddresses) + { + RunExecutable(KuboBinaryFile, $"bootstrap add {multiAddress} --repo-dir \"{RepoFolder.Path}\"", throwOnError: false); + } + + // Always call base at the end for these overridden settings methods. + await base.ApplyRoutingSettingsAsync(cancellationToken); + } +} \ No newline at end of file diff --git a/src/SwarmKeyGen.cs b/src/SwarmKeyGen.cs new file mode 100644 index 0000000..4a45eba --- /dev/null +++ b/src/SwarmKeyGen.cs @@ -0,0 +1,55 @@ +using System.Security.Cryptography; +using OwlCore.Storage; +using OwlCore.Storage.Memory; + +namespace OwlCore.Kubo +{ + /// + /// A static helpers for generating swarm keys. + /// + public static class SwarmKeyGen + { + /// + /// Generates a new swarm key. + /// + /// Ported from . + /// An in-memory file containing the private key. + public static async Task CreateAsync(CancellationToken cancellationToken = default) + { + // Create in-memory file + var file = new MemoryFile(new MemoryStream()); + + // Write key to file + await CreateAsync(file, cancellationToken); + + // Return the written file + return file; + } + /// + /// Generates a new swarm key. + /// + /// Ported from . + /// An in-memory file containing the private key. + public static async Task CreateAsync(IFile file, CancellationToken cancellationToken = default) + { + byte[] key = new byte[32]; // 32 bytes for the key + + using (RandomNumberGenerator rng = RandomNumberGenerator.Create()) + { + rng.GetBytes(key); // Fill the array with cryptographically secure random bytes + } + + // Convert the byte array to a hexadecimal string + string hexString = BitConverter.ToString(key).Replace("-", "").ToLower(); + + var swarmKey = $""" + /key/swarm/psk/1.0.0/"); + /base16/ + {hexString} + """; + + // Write key to file + await file.WriteTextAsync(swarmKey, cancellationToken); + } + } +} diff --git a/tests/OwlCore.Kubo.Tests/IpfsFileTests.cs b/tests/OwlCore.Kubo.Tests/IpfsFileTests.cs index fd1f4f6..27e5df1 100644 --- a/tests/OwlCore.Kubo.Tests/IpfsFileTests.cs +++ b/tests/OwlCore.Kubo.Tests/IpfsFileTests.cs @@ -1,9 +1,4 @@ -using Ipfs.Http; -using OwlCore.Storage; -using System.IO; -using System.Threading.Channels; - -namespace OwlCore.Kubo.Tests +namespace OwlCore.Kubo.Tests { [TestClass] public class IpfsFileTests @@ -11,7 +6,6 @@ public class IpfsFileTests [TestMethod] public async Task BasicFileReadTest() { - var file = new IpfsFile("Qmf412jQZiuVUtdgnB36FXFX7xg5V6KEbSJ4dpQuhkLyfD", TestFixture.Client); using var stream = await file.OpenStreamAsync(); diff --git a/tests/OwlCore.Kubo.Tests/IpnsFolderTests.cs b/tests/OwlCore.Kubo.Tests/IpnsFolderTests.cs index 4d31ba5..c3940e0 100644 --- a/tests/OwlCore.Kubo.Tests/IpnsFolderTests.cs +++ b/tests/OwlCore.Kubo.Tests/IpnsFolderTests.cs @@ -1,8 +1,5 @@ -using Ipfs.Http; -using OwlCore.Storage; +using OwlCore.Storage; using System.Diagnostics; -using System.IO; -using System.Threading.Channels; namespace OwlCore.Kubo.Tests { diff --git a/tests/OwlCore.Kubo.Tests/KuboBootstrapperTests.cs b/tests/OwlCore.Kubo.Tests/KuboBootstrapperTests.cs index 5c15feb..5029502 100644 --- a/tests/OwlCore.Kubo.Tests/KuboBootstrapperTests.cs +++ b/tests/OwlCore.Kubo.Tests/KuboBootstrapperTests.cs @@ -1,6 +1,4 @@ -using System.Text; - -namespace OwlCore.Kubo.Tests +namespace OwlCore.Kubo.Tests { [TestClass] public class KuboBootstrapperTests diff --git a/tests/OwlCore.Kubo.Tests/LoopbackPubSubApi.cs b/tests/OwlCore.Kubo.Tests/LoopbackPubSubApi.cs index 348e798..050426d 100644 --- a/tests/OwlCore.Kubo.Tests/LoopbackPubSubApi.cs +++ b/tests/OwlCore.Kubo.Tests/LoopbackPubSubApi.cs @@ -1,10 +1,8 @@ -using System.Text; -using System.Threading.Channels; -using Ipfs; +using Ipfs; using Ipfs.CoreApi; -using Microsoft.VisualStudio.TestPlatform.CommunicationUtilities; using OwlCore.Extensions; using OwlCore.Kubo.Models; +using System.Text; namespace OwlCore.Kubo.Tests; @@ -32,28 +30,32 @@ public LoopbackPubSubApi(Peer senderPeer) throw new NotImplementedException(); } - public Task PublishAsync(string topic, string message, CancellationToken cancel = new()) => PublishAsync(topic, Encoding.UTF8.GetBytes(message), cancel); + public Task PublishAsync(string topic, string message, CancellationToken cancel = default) => PublishAsync(topic, Encoding.UTF8.GetBytes(message), cancel); - public Task PublishAsync(string topic, byte[] message, CancellationToken cancel = new()) => PublishAsync(topic, new MemoryStream(message), cancel); + public Task PublishAsync(string topic, byte[] message, CancellationToken cancel = default) => PublishAsync(topic, new MemoryStream(message), cancel); - public Task PublishAsync(string topic, Stream message, CancellationToken cancel = new()) + public async Task PublishAsync(string topic, Stream message, CancellationToken cancel = default) { if (_handlers.TryGetValue(topic, out var handlers)) { - foreach (var handler in handlers) + foreach (var handler in handlers.ToArray()) { - handler(new PublishedMessage(_senderPeer, topic.IntoList(), Array.Empty(), message.ToBytes(), - message, message.Length)); + var bytes = await message.ToBytesAsync(cancel); + message.Seek(0, SeekOrigin.Begin); + handler(new PublishedMessage(_senderPeer, topic.IntoList(), Array.Empty(), bytes, new MemoryStream(bytes), bytes.Length)); } } - return _loopbackApis.InParallel(x => + await _loopbackApis.InParallel(x => { if (cancel.IsCancellationRequested) return Task.CompletedTask; - if (_emittedMessageHashCodes.Add(message.GetHashCode())) - return x.PublishAsync(topic, message, cancel); + lock (_emittedMessageHashCodes) + { + if (_emittedMessageHashCodes.Add(message.GetHashCode())) + return x.PublishAsync(topic, message, cancel); + } return Task.CompletedTask; }); @@ -72,8 +74,11 @@ public Task SubscribeAsync(string topic, Action handler, Canc if (cancellationToken.IsCancellationRequested) return Task.CompletedTask; - if (_subscribedHandlersHashCodes.Add(handler.GetHashCode())) - return x.SubscribeAsync(topic, handler, cancellationToken); + lock (_subscribedHandlersHashCodes) + { + if (_subscribedHandlersHashCodes.Add(handler.GetHashCode())) + return x.SubscribeAsync(topic, handler, cancellationToken); + } return Task.CompletedTask; }); diff --git a/tests/OwlCore.Kubo.Tests/MfsFolderTests.cs b/tests/OwlCore.Kubo.Tests/MfsFolderTests.cs index 4c9932e..db8c666 100644 --- a/tests/OwlCore.Kubo.Tests/MfsFolderTests.cs +++ b/tests/OwlCore.Kubo.Tests/MfsFolderTests.cs @@ -55,6 +55,8 @@ public async Task CreateAndDeleteFileAsync() var file = await mfs.CreateFileAsync("test.bin"); await mfs.DeleteAsync(file); + var items = await mfs.GetItemsAsync(StorableType.File).ToListAsync(); + await Assert.ThrowsExceptionAsync(async () => await mfs.GetItemAsync("/test.bin")); } @@ -94,15 +96,17 @@ public async Task GetParentAsync() [TestMethod] public async Task GetPathFromRootAsync() - { var mfs = new MfsFolder("/", TestFixture.Client); + { + var mfs = new MfsFolder("/", TestFixture.Client); var folder = (MfsFolder)await mfs.CreateFolderAsync("test", overwrite: true); var subfolder = (MfsFolder)await folder.CreateFolderAsync("subfolder", overwrite: true); try { var root = await subfolder.GetRootAsync(); - var path = await root.GetRelativePathToAsync(subfolder); + Assert.IsNotNull(root); + var path = await root.GetRelativePathToAsync(subfolder); Assert.AreEqual(path, subfolder.Path); } finally diff --git a/tests/OwlCore.Kubo.Tests/MfsStreamTests.cs b/tests/OwlCore.Kubo.Tests/MfsStreamTests.cs index e29f116..317eb3f 100644 --- a/tests/OwlCore.Kubo.Tests/MfsStreamTests.cs +++ b/tests/OwlCore.Kubo.Tests/MfsStreamTests.cs @@ -1,4 +1,6 @@ -namespace OwlCore.Kubo.Tests +using OwlCore.Storage; + +namespace OwlCore.Kubo.Tests { [TestClass] public class MfsStreamTests @@ -31,7 +33,7 @@ public async Task WriteAndReadRandomData() } // Open MfsStream from file, as standard stream. - using var fileStream = await file.OpenStreamAsync(); + using var fileStream = await file.OpenReadAsync(); var buffer = new byte[256]; var bytesRead = fileStream.Read(buffer, 0, 256); @@ -67,7 +69,7 @@ public async Task WriteAndReadRandomDataAsync() await stream.WriteAsync(randomData, 0, 256); // Read data back via file. - using var fileStream = await file.OpenStreamAsync(); + using var fileStream = await file.OpenReadAsync(); var buffer = new byte[256]; var bytesRead = await fileStream.ReadAsync(buffer, 0, 256); diff --git a/tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj b/tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj index d467cec..7ba4ffc 100644 --- a/tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj +++ b/tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj @@ -9,10 +9,10 @@ - - - - + + + + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/tests/OwlCore.Kubo.Tests/TestFixture.cs b/tests/OwlCore.Kubo.Tests/TestFixture.cs index 16fbc8f..850c967 100644 --- a/tests/OwlCore.Kubo.Tests/TestFixture.cs +++ b/tests/OwlCore.Kubo.Tests/TestFixture.cs @@ -1,6 +1,6 @@ using Ipfs.Http; using OwlCore.Storage; -using OwlCore.Storage.SystemIO; +using OwlCore.Storage.System.IO; using System.Diagnostics; namespace OwlCore.Kubo.Tests; @@ -59,7 +59,8 @@ public static async Task CreateNodeAsync(SystemFolder workingD ApiUri = new Uri($"http://127.0.0.1:{apiPort}"), GatewayUri = new Uri($"http://127.0.0.1:{gatewayPort}"), BinaryWorkingFolder = workingDirectory, - RoutingMode = DhtRoutingMode.DhtClient, + RoutingMode = DhtRoutingMode.AutoClient, + LaunchConflictMode = BootstrapLaunchConflictMode.Relaunch, }; OwlCore.Diagnostics.Logger.LogInformation($"Starting node {nodeRepoName}\n"); From 1b7734947cb8b9fabfec37026969842c6a9279fb Mon Sep 17 00:00:00 2001 From: Arlo Godfrey Date: Sun, 28 Apr 2024 03:10:42 -0500 Subject: [PATCH 2/3] Update to .NET 8 --- .github/workflows/build.yml | 2 +- .github/workflows/publish.yml | 2 +- global.json | 2 +- src/OwlCore.Kubo.csproj | 2 +- tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index b72a396..b705d91 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -24,7 +24,7 @@ jobs: - name: Install .NET SDK uses: actions/setup-dotnet@v1 with: - dotnet-version: '7.0.x' + dotnet-version: '8.0.x' # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it - name: Checkout Repository diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index fde83ae..a8e0fb2 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -16,7 +16,7 @@ jobs: - name: Install .NET SDK uses: actions/setup-dotnet@v1 with: - dotnet-version: '7.0.x' + dotnet-version: '8.0.x' - name: Restore dependencies run: dotnet restore diff --git a/global.json b/global.json index 1c7274b..408e96b 100644 --- a/global.json +++ b/global.json @@ -1,6 +1,6 @@ { "sdk": { - "version": "7.0.100", + "version": "8.0.201", "rollForward": "latestFeature" } } \ No newline at end of file diff --git a/src/OwlCore.Kubo.csproj b/src/OwlCore.Kubo.csproj index 0578340..3588fa0 100644 --- a/src/OwlCore.Kubo.csproj +++ b/src/OwlCore.Kubo.csproj @@ -1,6 +1,6 @@  - netstandard2.0;net6.0;net7.0 + netstandard2.0;net6.0;net7.0;net8.0; enable 12.0 nullable diff --git a/tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj b/tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj index 7ba4ffc..21f4155 100644 --- a/tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj +++ b/tests/OwlCore.Kubo.Tests/OwlCore.Kubo.Tests.csproj @@ -1,7 +1,7 @@ - net7.0 + net8.0 enable enable From 6827228a17ac36c1d4728671c5db71bbde3c1f29 Mon Sep 17 00:00:00 2001 From: Arlo Godfrey Date: Sun, 28 Apr 2024 03:16:51 -0500 Subject: [PATCH 3/3] Update release notes for UseAcceleratedDHTClient --- src/OwlCore.Kubo.csproj | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/OwlCore.Kubo.csproj b/src/OwlCore.Kubo.csproj index 3588fa0..a04a137 100644 --- a/src/OwlCore.Kubo.csproj +++ b/src/OwlCore.Kubo.csproj @@ -46,7 +46,7 @@ Added all missing DhtRoutingMode values that have been added to Kubo as of 0.26. Added a LaunchConfigMode to KuboBootstrapper. Defines the behavior when a node is already running (when the repo is locked): Throw, Attach, or Relaunch. Added an ApiUriMode to KuboBootstrapper. Gets or sets an enum that determines how to use the supplied ApiUri: UseExisting, or OverwriteExisting. Added a GatewayUriMode to KuboBootstrapper. Gets or sets an enum that determines how to use the supplied GatewayUri: UseExisting, or OverwriteExisting. -Added a UseAcceleratedDHTClient property to KuboBootstrapper. If set to true, the client DHT will be used on startup. +Added a UseAcceleratedDHTClient property to KuboBootstrapper. If set to true, the accelerated DHT client will be used on startup. Extensive documentation has been added to aid in deciding whether you should use this. Note that enabling this will increase resource consumption. [Improvements] As part of the move to ICoreApi, all internal calls to DoCommandAsync have been removed wherever IMfsApi is used, thanks to the implementation contributed in https://github.com/ipfs-shipyard/net-ipfs-core/pull/13/.