لرن سرا

برچسب: automatically

  • How to automatically refresh configurations with Azure App Configuration in ASP.NET Core | Code4IT

    How to automatically refresh configurations with Azure App Configuration in ASP.NET Core | Code4IT


    ASP.NET allows you to poll Azure App Configuration to always get the most updated values without restarting your applications. It’s simple, but you have to think thoroughly.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    – Davide

    In a previous article, we learned how to centralize configurations using Azure App Configuration, a service provided by Azure to share configurations in a secure way. Using Azure App Configuration, you’ll be able to store the most critical configurations in a single place and apply them to one or more environments or projects.

    We used a very simple example with a limitation: you have to restart your applications to make changes effective. In fact, ASP.NET connects to Azure App Config, loads the configurations in memory, and serves these configs until the next application restart.

    In this article, we’re gonna learn how to make configurations dynamic: by the end of this article, we will be able to see the changes to our configs reflected in our applications without restarting them.

    Since this one is a kind of improvement of the previous article, you should read it first.

    Let me summarize here the code showcased in the previous article. We have an ASP.NET Core API application whose only purpose is to return the configurations stored in an object, whose shape is this one:

    {
  "MyNiceConfig": {
    "PageSize": 6,
    "Host": {
      "BaseUrl": "https://www.mydummysite.com",
      "Password": "123-go"
    }
  }
}

    In the constructor of the API controller, I injected an IOptions<MyConfig> instance that holds the current data stored in the application.

     public ConfigDemoController(IOptions<MyConfig> config)
        => _config = config;

    The only HTTP Endpoint is a GET: it just accesses that value and returns it to the client.

    [HttpGet()]
public IActionResult Get()
{
    return Ok(_config.Value);
}

    Finally, I created a new instance of Azure App Configuration, and I used a connection string to integrate Azure App Configuration with the existing configurations by calling:

    builder.Configuration.AddAzureAppConfiguration(ConnectionString);

    Now we can move on and make configurations dynamic.

    Sentinel values: a guard value to monitor changes in the configurations

    On Azure App Configuration, you have to update the configurations manually one by one. Unfortunately, there is no way to update them in a single batch. You can import them in a batch, but you have to update them singularly.

    Imagine that you have a service that accesses an external API whose BaseUrl and API Key are stored on Az App Configuration. We now need to move to another API: we then have to update both BaseUrl and API Key. The application is running, and we want to update the info about the external API. If we updated the application configurations every time something is updated on Az App Configuration, we would end up with an invalid state – for example, we would have the new BaseUrl and the old API Key.

    Therefore, we have to define a configuration value that acts as a sort of versioning key for the whole list of configurations. In Azure App Configuration’s jargon, it’s called Sentinel.

    A Sentinel is nothing but version key: it’s a string value that is used by the application to understand if it needs to reload the whole list of configurations. Since it’s just a string, you can set any value, as long as it changes over time. My suggestion is to use the UTC date value of the moment you have updated the value, such as 202306051522. This way, in case of errors you can understand when was the last time any of these values have changed (but you won’t know which values have changed), and, depending on the pricing tier you are using, you can compare the current values with the previous ones.

    So, head back to the Configuration Explorer page and add a new value: I called it Sentinel.

    Sentinel value on Azure App Configuration

    As I said, you can use any value. For the sake of this article, I’m gonna use a simple number (just for simplicity).

    Define how to refresh configurations using ASP.NET Core app startup

    We can finally move to the code!

    If you recall, in the previous article we added a NuGet package, Microsoft.Azure.AppConfiguration.AspNetCore, and then we added Azure App Configuration as a configurations source by calling

    builder.Configuration.AddAzureAppConfiguration(ConnectionString);

    That instruction is used to load all the configurations, without managing polling and updates. Therefore, we must remove it.

    Instead of that instruction, add this other one:

    builder.Configuration.AddAzureAppConfiguration(options =>
{
    options
    .Connect(ConnectionString)
    .Select(KeyFilter.Any, LabelFilter.Null)
    // Configure to reload configuration if the registered sentinel key is modified
    .ConfigureRefresh(refreshOptions =>
              refreshOptions.Register("Sentinel", label: LabelFilter.Null, refreshAll: true)
        .SetCacheExpiration(TimeSpan.FromSeconds(3))
      );
});

    Let’s deep dive into each part:

    options.Connect(ConnectionString) just tells ASP.NET that the configurations must be loaded from that specific connection string.

    .Select(KeyFilter.Any, LabelFilter.Null) loads all keys that have no Label;

    and, finally, the most important part:

    .ConfigureRefresh(refreshOptions =>
            refreshOptions.Register(key: "Sentinel", label: LabelFilter.Null, refreshAll: true)
      .SetCacheExpiration(TimeSpan.FromSeconds(3))
    );

    Here we are specifying that all values must be refreshed (refreshAll: true) when the key with value=“Sentinel” (key: "Sentinel") is updated. Then, store those values for 3 seconds (SetCacheExpiration(TimeSpan.FromSeconds(3)).

    Here I used 3 seconds as a refresh time. This means that, if the application is used continuously, the application will poll Azure App Configuration every 3 seconds – it’s clearly a bad idea! So, pick the correct value depending on the change expectations. The default value for cache expiration is 30 seconds.

    Notice that the previous instruction adds Azure App Configuration to the Configuration object, and not as a service used by .NET. In fact, the method is builder.Configuration.AddAzureAppConfiguration. We need two more steps.

    First of all, add Azure App Configuration to the IServiceCollection object:

    builder.Services.AddAzureAppConfiguration();

    Finally, we have to add it to our existing middlewares by calling

    app.UseAzureAppConfiguration();

    The final result is this:

    public static void Main(string[] args)
{
    var builder = WebApplication.CreateBuilder(args);

    const string ConnectionString = "......";

    // Load configuration from Azure App Configuration
    builder.Configuration.AddAzureAppConfiguration(options =>
    {
        options.Connect(ConnectionString)
                .Select(KeyFilter.Any, LabelFilter.Null)
                // Configure to reload configuration if the registered sentinel key is modified
                .ConfigureRefresh(refreshOptions =>
                    refreshOptions.Register(key: "Sentinel", label: LabelFilter.Null, refreshAll: true)
                    .SetCacheExpiration(TimeSpan.FromSeconds(3)));
    });

    // Add the service to IServiceCollection
    builder.Services.AddAzureAppConfiguration();

    builder.Services.AddControllers();
    builder.Services.Configure<MyConfig>(builder.Configuration.GetSection("MyNiceConfig"));

    var app = builder.Build();

    // Add the middleware
    app.UseAzureAppConfiguration();

    app.UseHttpsRedirection();

    app.MapControllers();

    app.Run();
}

    IOptionsMonitor: accessing and monitoring configuration values

    It’s time to run the project and look at the result: some of the values are coming from Azure App Configuration.

    Default config coming from Azure App Configuration

    Now we can update them: without restarting the application, update the PageSize value, and don’t forget to update the Sentinel too. Call again the endpoint, and… nothing happens! 😯

    This is because in our controller we are using IOptions<T> instead of IOptionsMonitor<T>. As we’ve learned in a previous article, IOptionsMonitor<T> is a singleton instance that always gets the most updated config values. It also emits an event when the configurations have been refreshed.

    So, head back to the ConfigDemoController, and replace the way we retrieve the config:

    [ApiController]
[Route("[controller]")]
public class ConfigDemoController : ControllerBase
{
    private readonly IOptionsMonitor<MyConfig> _config;

    public ConfigDemoController(IOptionsMonitor<MyConfig> config)
    {
        _config = config;
        _config.OnChange(Update);
    }

    [HttpGet()]
    public IActionResult Get()
    {
        return Ok(_config.CurrentValue);
    }

    private void Update(MyConfig arg1, string? arg2)
    {
      Console.WriteLine($"Configs have been updated! PageSize is {arg1.PageSize}, " +
                $" Password is {arg1.Host.Password}");
    }
}

    When using IOptionsMonitor<T>, you can retrieve the current values of the configuration object by accessing the CurrentValue property. Also, you can define an event listener that is to be attached to the OnChange event;

    We can finally run the application and update the values on Azure App Configuration.

    Again, update one of the values, update the sentinel, and wait. After 3 seconds, you’ll see a message popping up in the console: it’s the text defined in the Update method.

    Then, call again the application (again, without restarting it), and admire the updated values!

    You can see a live demo here:

    Demo of configurations refreshed dinamically

    As you can see, the first time after updating the Sentinel value, the values are still the old ones. But, in the meantime, the values have been updated, and the cache has expired, so that the next time the values will be retrieved from Azure.

    My 2 cents on timing

    As we’ve learned, the config values are stored in a memory cache, with an expiration time. Every time the cache expires, we need to retrieve again the configurations from Azure App Configuration (in particular, by checking if the Sentinel value has been updated in the meanwhile). Don’t underestimate the cache value, as there are pros and cons of each kind of value:

    • a short timespan keeps the values always up-to-date, making your application more reactive to changes. But it also means that you are polling too often the Azure App Configuration endpoints, making your application busier and incurring limitations due to the requests count;
    • a long timespan keeps your application more performant because there are fewer requests to the Configuration endpoints, but it also forces you to have the configurations updated after a while from the update applied on Azure.

    There is also another issue with long timespans: if the same configurations are used by different services, you might end up in a dirty state. Say that you have UserService and PaymentService, and both use some configurations stored on Azure whose caching expiration is 10 minutes. Now, the following actions happen:

    1. UserService starts
    2. PaymentService starts
    3. Someone updates the values on Azure
    4. UserService restarts, while PaymentService doesn’t.

    We will end up in a situation where UserService has the most updated values, while PaymentService doesn’t. There will be a time window (in our example, up to 10 minutes) in which the configurations are misaligned.

    Also, take costs and limitations into consideration: with the Free tier you have 1000 requests per day, while with the Standard tier, you have 30.000 per hour per replica. Using the default cache expiration (30 seconds) in an application with a continuous flow of users means that you are gonna call the endpoint 2880 times per day (2 times a minute * (minutes per day = 1440)). Way more than the available value on the Free tier.

    So, think thoroughly before choosing an expiration time!

    Further readings

    This article is a continuation of a previous one, and I suggest you read the other one to understand how to set up Azure App Configuration and how to integrate it in an ASP.NET Core API application in case you don’t want to use dynamic configuration.

    🔗 Azure App Configuration and ASP.NET Core API: a smart and secure way to manage configurations | Code4IT

    This article first appeared on Code4IT 🐧

    Also, we learned that using IOptions we are not getting the most updated values: in fact, we need to use IOptionsMonitor. Check out this article to understand the other differences in the IOptions family.

    🔗 Understanding IOptions, IOptionsMonitor, and IOptionsSnapshot in ASP.NET Core | Code4IT

    Finally, I briefly talked about pricing. As of July 2023, there are just 2 pricing tiers, with different limitations.

    🔗 App Configuration pricing | Microsoft Learn

    Wrapping up

    In my opinion, smart configuration handling is essential for the hard times when you have to understand why an error is happening only in a specific environment.

    Centralizing configurations is a good idea, as it allows developers to simulate a whole environment by just changing a few values on the application.

    Making configurations live without restarting your applications manually can be a good idea, but you have to analyze it thoroughly.

    I hope you enjoyed this article! Let’s keep in touch on Twitter or LinkedIn! 🤜🤛

    Happy coding!

    🐧





    Source link

  • HTTP Logging in ASP.NET: how to automatically log all incoming HTTP requests (and its downsides!)

    HTTP Logging in ASP.NET: how to automatically log all incoming HTTP requests (and its downsides!)


    Aren’t you tired of adding manual logs to your HTTP APIs to log HTTP requests and responses? By using a built-in middleware in ASP.NET, you will be able to centralize logs management and have a clear view of all the incoming HTTP requests.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    – Davide

    Whenever we publish a service, it is important to add proper logging to the application. Logging helps us understand how the system works and behaves, and it’s a fundamental component that allows us to troubleshoot problems that occur during the actual usage of the application.

    In this blog, we have talked several times about logging. However, we mostly focused on the logs that were written manually.

    In this article, we will learn how to log incoming HTTP requests to help us understand how our APIs are being used from the outside.

    Scaffolding the empty project

    To showcase this type of logging, I created an ASP.NET API. It’s a very simple application with CRUD operations on an in-memory collection.

    [ApiController]
[Route("[controller]")]
public class BooksController : ControllerBase
{
    private readonly List<Book> booksCatalogue = Enumerable.Range(1, 5).Select(index => new Book
    {
        Id = index,
        Title = $"Book with ID {index}"
    }).ToList();

    private readonly ILogger<BooksController> _logger;

    public BooksController(ILogger<BooksController> logger)
    {
        _logger = logger;
    }
}

    These CRUD operations are exposed via HTTP APIs, following the usual verb-based convention.

    For example:

    [HttpGet("{id}")]
public ActionResult<Book> GetBook([FromRoute] int id)
{
    _logger.LogInformation("Looking if in my collection with {TotalBooksCount} books there is one with ID {SearchedId}"
            , booksCatalogue.Count, id);

    Book? book = booksCatalogue.SingleOrDefault(x => x.Id == id);

    return book switch
    {
        null => NotFound(),
        _ => Ok(book)
    };
}

    As you can see, I have added some custom logs: before searching for the element with the specified ID, I also wrote a log message such as “Looking if in my collection with 5 books there is one with ID 2”.

    Where can I find the message? For the sake of this article, I decided to use Seq!

    Seq is a popular log sink (well, as you may know, my favourite one!), that is easy to install and to integrate with .NET. I’ve thoroughly explained how to use Seq in conjunction with ASP.NET in this article and in other ones.

    In short, the most important change in your application is to add Seq as the log sink, like this:

    builder.Services.AddLogging(lb => {
    lb.AddSeq();
});

    Now, whenever I call the GET endpoint, I can see the related log messages appear in Seq:

    Custom log messages

    But sometimes it’s not enough. I want to see more details, and I want them to be applied everywhere!

    How to add HTTP Logging to an ASP.NET application

    HTTP Logging is a way of logging most of the details of the incoming HTTP operations, tracking both the requests and the responses.

    With HTTP Logging, you don’t need to manually write custom logs to access the details of incoming requests: you just need to add its related middleware, configure it as you want, and have all the required logs available for all your endpoints.

    Adding it is pretty straightforward: you first need to add the HttpLogging middleware to the list of services:

    builder.Services.AddHttpLogging(lb => { });

    so that you can use it once the WebApplication instance is built:

    There’s still a problem, though: all the logs generated via HttpLogging are, by default, ignored, as logs coming from their namespace (named Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware) are at Information log level, thus ignored because of the default configurations.

    You either have to update the appsetting.json file to tell the logging system to process logs from that namespace:

    {
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",
      "Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware": "Information"
    }
  }
}

    or, alternatively, you need to do the same when setting up the logging system in the Program class:

    builder.Services.AddLogging(lb => {
  lb.AddSeq();
+ lb.AddFilter("Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware", LogLevel.Information);
});

    We then have all our pieces in place: let’s execute the application!

    First, you can spin up the API; you should be able to see the Swagger page:

    Swagger page for our application&rsquo;s API

    From here, you can call the GET endpoint:

    Http response of the API call, as seen on Swagger

    You should now able to see all the logs in Seq:

    Logs list in Seq

    As you can see from the screenshot above, I have a log entry for the request and one for the response. Also, of course, I have the custom message I added manually in the C# method.

    Understanding HTTP Request logs

    Let’s focus on the data logged for the HTTP request.

    If we open the log related to the HTTP request, we can see all these values:

    Details of the HTTP Request

    Among these details, we can see properties such as:

    • the host name (localhost:7164)
    • the method (GET)
    • the path (/books/4)

    and much more.

    You can see all the properties as standalone items, but you can also have a grouped view of all the properties by accessing the HttpLog element:

    Details of the HTTP Log element

    Notice that for some elements we do not have access to the actual value, as the value is set to [Redacted]. This is a default configuration that prevents logging too many things (and undisclosing some values) as well as writing too much content on the log sink (the more you write, the less performant the queries become – and you also pay more!).

    Among other redacted values, you can see that even the Cookie value is not directly available – for the same reasons explained before.

    Understanding HTTP Response logs

    Of course, we can see some interesting data in the Response log:

    Details of the HTTP Response

    Here, among some other properties such as the Host Name, we can see the Status Code and the Trace Id (which, as you may notice, is the same as the one in te Request).

    As you can see, the log item does not contain the body of the response.

    Also, just as it happens with the Request, we do not have access to the list of HTTP Headers.

    How to save space, storage, and money by combining log entries

    For every HTTP operation, we end up with 2 log entries: one for the Request and one for the Response.

    However, it would be more practical to have both request and response info stored in the same log item so we can understand more easily what is happening.

    Lucky for us, this functionality is already in place. We just need to set the CombineLogs property to true when we add the HttpLogging functionality:

    builder.Services.AddHttpLogging(lb =>
{
+  lb.CombineLogs = true;
}
);

    Then, we are able to see the data for both the request and the related response in the same log element.

    Request and Response combined logs

    The downsides of using HTTP Logging

    Even though everything looks nice and pretty, adding HTTP Logging has some serious consequences.

    First of all, remember that you are doing some more operations for every incoming HTTP request. Just processing and storing the log messages can bring to an application performance downgrade – you are using parts of the processing resources to interpret the HTTP context, create the correct log entry, and store it.

    Depending on how your APIs are structured, you may need to strip out sensitive data: HTTP Logs, by default, log almost everything (except for the parts stored as Redacted). Since you don’t want to store as plain text the content of the requests, you may need to create custom logic to redact parts of the request and response you want to hide: you may need to implement a custom IHttpLoggingInterceptor.

    Finally, consider that logging occupies storage, and storage has a cost. The more you log, the higher the cost. You should define proper strategies to avoid excessive storage costs while keeping valuable logs.

    Further readings

    There is a lot more, as always. In this article, I focused on the most essential parts, but the road to having proper HTTP Logs is still long.

    You may want to start from the official documentation, of course!

    🔗 HTTP logging in ASP.NET Core | Microsoft Docs

    This article first appeared on Code4IT 🐧

    All the logs produced for this article were stored on Seq. You can find more info about installing and integrating Seq in ASP.NET Core in this article:

    🔗 Easy logging management with Seq and ILogger in ASP.NET | Code4IT

    Wrapping up

    HTTP Logging can be a good tool for understanding the application behaviour and detecting anomalies. However, as you can see, there are some important downsides that need to be considered.

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn or Twitter! 🤜🤛

    Happy coding!

    🐧





    Source link