diff --git a/CHANGES.md b/CHANGES.md index 58bb0034..27ecad43 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,3 +1,9 @@ +# 5.5.1 +* Fixed issue #300 Support DateTime2 data type for TimeStamp column (thanks to @stedel for the contribution). +* Added Sourcelink support and publish symbols to nuget.org which allows consumers of the package to debug into the sink code. +* Added troubleshooting tip in README with workaround for SqlClient issue regarding missing Microsoft.Data.SqlClient.Sni.dll in .NET Framework apps and updated `samples\AppConfigDemo` accordingly. +* Added information in README about batched sink SqlBulkCopy behavior according to issue #209. + # 5.5.0 * Implemented enhancement #208: use Microsoft.Data.SqliClient for all platforms except net452 to enable Column Encryption (thanks to @mungk for the contribution). * Fixed issue #290 MissingMethodException with .NET Standard 2.0. diff --git a/README.md b/README.md index cb8884c4..7c1cbb07 100644 --- a/README.md +++ b/README.md @@ -436,14 +436,19 @@ This column stores the event level (Error, Information, etc.). For backwards-com ### TimeStamp -This column stores the time the log event was sent to Serilog as a SQL `datetime` (default) or `datetimeoffset` type. If `datetimeoffset` should be used, this can be configured as follows. +This column stores the time the log event was sent to Serilog as a SQL `datetime` (default), `datetime2` or `datetimeoffset` type. If `datetime2` or `datetimeoffset` should be used, this can be configured as follows. ```csharp var columnOptions = new ColumnOptions(); columnOptions.TimeStamp.DataType = SqlDbType.DateTimeOffset; ``` -Please be aware that you have to configure the sink for `datetimeoffset` if the used logging database table has a `TimeStamp` column of type `datetimeoffset`. On the other hand you must not configure for `datetimeoffset` if the `TimeStamp` column is of type `datetime`. Failing to configure the data type accordingly can result in log table entries with wrong timezone offsets or no log entries being created at all due to exceptions during logging. +```csharp +var columnOptions = new ColumnOptions(); +columnOptions.TimeStamp.DataType = SqlDbType.DateTime2; +``` + +Please be aware that you have to configure the sink for `datetimeoffset` if the used logging database table has a `TimeStamp` column of type `datetimeoffset`. If the underlying database uses `datetime2` for the `TimeStamp` column, the sink must be configured to use `datetime2`. On the other hand you must not configure for `datetimeoffset` if the `TimeStamp` column is of type `datetime` or `datetime2`. Failing to configure the data type accordingly can result in log table entries with wrong timezone offsets or no log entries being created at all due to exceptions during logging. While TimeStamp may appear to be a good candidate as a clustered primary key, even relatively low-volume logging can emit identical timestamps forcing SQL Server to add a "uniqueifier" value behind the scenes (effectively an auto-incrementing identity-like integer). For frequent timestamp range-searching and sorting, a non-clustered index is better. @@ -699,6 +704,11 @@ Any Serilog application should _always_ call `Log.CloseAndFlush` before shutting AppDomain.CurrentDomain.ProcessExit += (s, e) => Log.CloseAndFlush(); ``` +### Consider batched sink SqlBulkCopy behavior + +If you initialize the sink with `WriteTo` then it uses a batched sink semantics. This means that it does not directly issue an SQL command to the database for each log call, but it collectes log events in a buffer and later asynchronously writes a bulk of them to the database using [SqlBulkCopy](https://docs.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlbulkcopy?view=dotnet-plat-ext-3.1). **If SqlBulkCopy fails to write a single row of the batch to the database, the whole batch will be lost.** Unfortunately it is not easily possible (and probably only with a significant performance impact) to find out what lines of the batch caused problems. Therefore the sink cannot easily retry the operation with the problem lines removed. Typical problems can be that data (like the log message) exceeds the field length in the database or fields which cannot be null are null in the log event. Keep this in mind when using the batched version of the sink and avoid log events to be created with data that is invalid according to your database schema. Use a wrapper class or Serilog Enrichers to validate and correct the log event data before it gets written to the database. + + ### Test outside of Visual Studio When you exit an application running in debug mode under Visual Studio, normal shutdown processes may be interrupted. Visual Studio issues a nearly-instant process kill command when it decides you're done debugging. This is a particularly common problem with ASP.NET and ASP.NET Core applications, in which Visual Studio instantly terminates the application as soon as the browser is closed. Even `finally` blocks usually fail to execute. If you aren't seeing your last few events written, try testing your application outside of Visual Studio. @@ -711,6 +721,10 @@ If you're reading about a feature that doesn't seem to work, check whether you'r Please check your NuGet references and confirm you are specifically referencing _Serilog.Sinks.MSSqlServer_. In the early days of .NET Core, there was a popular Core-specific fork of this sink, but the documentation and NuGet project URLs pointed here. Today the package is marked deprecated, but we continue to see some confusion around this. +### .NET Framework apps must reference Microsoft.Data.SqlClient + +If you are using the sink in a .NET Framework app, make sure to add a nuget package reference to Microsoft.Data.SqlClient in your app project. This is necessary due to a bug in SqlClient which can lead to exceptions about missing Microsoft assemblies. Details can be found in [issue 283](https://github.com/serilog/serilog-sinks-mssqlserver/issues/283#issuecomment-664397489) and [issue 208](https://github.com/serilog/serilog-sinks-mssqlserver/issues/208#issuecomment-664503566). + ## Querying Property Data Extracting and querying the property column directly can be helpful when looking for specific log sequences. SQL Server has query syntax supporting columns that store either XML or JSON data. diff --git a/appveyor.yml b/appveyor.yml index 0303d0a3..ac54f1fb 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -9,6 +9,7 @@ build_script: test: off artifacts: - path: artifacts/Serilog.*.nupkg +- path: artifacts/Serilog.*.snupkg deploy: - provider: NuGet api_key: @@ -18,7 +19,9 @@ deploy: - provider: GitHub auth_token: secure: p4LpVhBKxGS5WqucHxFQ5c7C8cP74kbNB0Z8k9Oxx/PMaDQ1+ibmoexNqVU5ZlmX - artifact: /Serilog.*\.nupkg/ + artifacts: + /Serilog.*\.nupkg/ + /Serilog.*\.snupkg/ tag: v$(appveyor_build_version) on: branch: master diff --git a/sample/AppConfigDemo/AppConfigDemo.csproj b/sample/AppConfigDemo/AppConfigDemo.csproj index 71f96d91..06b1b01c 100644 --- a/sample/AppConfigDemo/AppConfigDemo.csproj +++ b/sample/AppConfigDemo/AppConfigDemo.csproj @@ -12,6 +12,8 @@ 512 true true + + AnyCPU @@ -33,12 +35,36 @@ 4 + + ..\..\packages\Microsoft.Data.SqlClient.1.1.3\lib\net46\Microsoft.Data.SqlClient.dll + ..\..\packages\Microsoft.Extensions.Configuration.Abstractions.3.1.4\lib\netstandard2.0\Microsoft.Extensions.Configuration.Abstractions.dll ..\..\packages\Microsoft.Extensions.Primitives.3.1.4\lib\netstandard2.0\Microsoft.Extensions.Primitives.dll + + ..\..\packages\Microsoft.Identity.Client.3.0.8\lib\net45\Microsoft.Identity.Client.dll + + + ..\..\packages\Microsoft.IdentityModel.JsonWebTokens.5.5.0\lib\net461\Microsoft.IdentityModel.JsonWebTokens.dll + + + ..\..\packages\Microsoft.IdentityModel.Logging.5.5.0\lib\net461\Microsoft.IdentityModel.Logging.dll + + + ..\..\packages\Microsoft.IdentityModel.Protocols.5.5.0\lib\net461\Microsoft.IdentityModel.Protocols.dll + + + ..\..\packages\Microsoft.IdentityModel.Protocols.OpenIdConnect.5.5.0\lib\net461\Microsoft.IdentityModel.Protocols.OpenIdConnect.dll + + + ..\..\packages\Microsoft.IdentityModel.Tokens.5.5.0\lib\net461\Microsoft.IdentityModel.Tokens.dll + + + ..\..\packages\Newtonsoft.Json.10.0.1\lib\net45\Newtonsoft.Json.dll + ..\..\packages\Serilog.2.9.0\lib\net46\Serilog.dll @@ -48,6 +74,16 @@ + + ..\..\packages\System.Data.Common.4.3.0\lib\net451\System.Data.Common.dll + True + True + + + + + ..\..\packages\System.IdentityModel.Tokens.Jwt.5.5.0\lib\net461\System.IdentityModel.Tokens.Jwt.dll + ..\..\packages\System.Memory.4.5.4\lib\net461\System.Memory.dll @@ -58,6 +94,7 @@ ..\..\packages\System.Runtime.CompilerServices.Unsafe.4.7.1\lib\net461\System.Runtime.CompilerServices.Unsafe.dll + @@ -82,4 +119,11 @@ + + + + This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + \ No newline at end of file diff --git a/sample/AppConfigDemo/packages.config b/sample/AppConfigDemo/packages.config index fbbc92b4..029494de 100644 --- a/sample/AppConfigDemo/packages.config +++ b/sample/AppConfigDemo/packages.config @@ -1,10 +1,21 @@  + + + + + + + + + + + diff --git a/src/Serilog.Sinks.MSSqlServer/Serilog.Sinks.MSSqlServer.csproj b/src/Serilog.Sinks.MSSqlServer/Serilog.Sinks.MSSqlServer.csproj index f5636cea..758e3064 100644 --- a/src/Serilog.Sinks.MSSqlServer/Serilog.Sinks.MSSqlServer.csproj +++ b/src/Serilog.Sinks.MSSqlServer/Serilog.Sinks.MSSqlServer.csproj @@ -2,7 +2,7 @@ A Serilog sink that writes events to Microsoft SQL Server - 5.5.0 + 5.5.1 Michiel van Oudheusden;Christian Kadluba;Serilog Contributors netstandard2.0;net452;net461;net472;netcoreapp2.0;netcoreapp2.2 true @@ -18,6 +18,10 @@ Apache-2.0 https://github.com/serilog/serilog-sinks-mssqlserver git + true + true + true + snupkg win false false @@ -26,6 +30,7 @@ + all runtime; build; native; contentfiles; analyzers; buildtransitive diff --git a/src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptions.cs b/src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptions.cs index a6c6713f..0cdfed34 100644 --- a/src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptions.cs +++ b/src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptions.cs @@ -20,15 +20,15 @@ public TimeStampColumnOptions() : base() } /// - /// The TimeStamp column only supports the DateTime and DateTimeOffset data types. + /// The TimeStamp column only supports the DateTime, DateTime2 and DateTimeOffset data types. /// public new SqlDbType DataType { get => base.DataType; set { - if (value != SqlDbType.DateTime && value != SqlDbType.DateTimeOffset) - throw new ArgumentException("The Standard Column \"TimeStamp\" only supports the DateTime and DateTimeOffset formats."); + if (value != SqlDbType.DateTime && value != SqlDbType.DateTimeOffset && value != SqlDbType.DateTime2) + throw new ArgumentException("The Standard Column \"TimeStamp\" only supports the DateTime, DateTime2 and DateTimeOffset formats."); base.DataType = value; } } diff --git a/src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/Output/JsonLogEventFormatter.cs b/src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/Output/JsonLogEventFormatter.cs index 399c009d..590f7906 100644 --- a/src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/Output/JsonLogEventFormatter.cs +++ b/src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/Output/JsonLogEventFormatter.cs @@ -124,7 +124,7 @@ private void WriteTimeStampIfPresent(LogEvent logEvent, TextWriter output, ref s output.Write(precedingDelimiter); precedingDelimiter = _commaDelimiter; var colData = WritePropertyName(logEvent, output, StandardColumn.TimeStamp); - var value = _columnOptions.TimeStamp.DataType == SqlDbType.DateTime + var value = (_columnOptions.TimeStamp.DataType == SqlDbType.DateTime || _columnOptions.TimeStamp.DataType == SqlDbType.DateTime2) ? ((DateTime)colData.Value).ToString("o", CultureInfo.InvariantCulture) : ((DateTimeOffset)colData.Value).ToString("o", CultureInfo.InvariantCulture); JsonValueFormatter.WriteQuotedJsonString(value, output); diff --git a/test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptionsTests.cs b/test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptionsTests.cs index d9a92e0c..89eee0c9 100644 --- a/test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptionsTests.cs +++ b/test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptionsTests.cs @@ -40,5 +40,16 @@ public void CannotSetDataTypeNVarChar() // Act and assert - should throw Assert.Throws(() => options.TimeStamp.DataType = SqlDbType.NVarChar); } + + [Trait("Feature", "#300")] + [Fact] + public void CanSetDataTypeDateTime2() + { + // Arrange + var options = new Serilog.Sinks.MSSqlServer.ColumnOptions(); + + // Act - should not throw + options.TimeStamp.DataType = SqlDbType.DateTime2; + } } } diff --git a/test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/Output/JsonLogEventFormatterTests.cs b/test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/Output/JsonLogEventFormatterTests.cs index 7cbd7c6f..d605ecea 100644 --- a/test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/Output/JsonLogEventFormatterTests.cs +++ b/test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/Output/JsonLogEventFormatterTests.cs @@ -1,4 +1,4 @@ -using System; +using System; using System.Collections.Generic; using System.Data; using System.IO; @@ -90,6 +90,27 @@ public void FormatTimeStampColumnTypeDateTimeRendersCorrectTimeStamp() Assert.Equal(expectedResult, renderResult); } + [Fact] + [Trait("Feature", "#300")] + public void FormatTimeStampColumnTypeDateTime2RendersCorrectTimeStamp() + { + // Arrange + const string expectedResult = "{\"TimeStamp\":\"2020-07-01T09:41:10.1230000\",\"Level\":\"Information\",\"Message\":\"\",\"MessageTemplate\":\"Test message template\"}"; + _testColumnOptions.TimeStamp.DataType = SqlDbType.DateTime2; + var testLogEvent = CreateTestLogEvent(new DateTimeOffset(2020, 7, 1, 9, 41, 10, 123, TimeSpan.Zero)); + + // Act + string renderResult; + using (var outputWriter = new StringWriter()) + { + _sut.Format(testLogEvent, outputWriter); + renderResult = outputWriter.ToString(); + } + + // Assert + Assert.Equal(expectedResult, renderResult); + } + [Fact] public void FormatWithPropertiesRendersCorrectProperties() {