دسته: ذخیره داده‌های موقت

  • Davide’s Code and Architecture Notes

    Davide’s Code and Architecture Notes


    When designing a software system, we naturally focus more on the happy flow. But we should carefully plan to handle errors that fall into three categories: Validation, Transient, and Fatal.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    When designing a new software system, it’s easy to focus mainly on the happy flow and forget that you must also handle errors.

    You should carefully define and design how to handle errors: depending on the use case, error handling can have a huge impact on the architecture of your software system.

    In this article, we’ll explore the three main categories of errors that we must always remember to address; for each type of error, we will showcase how addressing it can impact the software architecture differently.

    An ideal system with only the happy path

    To use a realistic example, let’s design a simple system with a single module named MainApplication: this module reads data from an external API, manipulates the data, and stores the result on the DB.

    The system is called asynchronously, via a Message Queue, by an external service – that we are going to ignore.

    The happy flow is pretty much the following:

    1. An external system inserts some data into the Queue;
    2. MainApplication reads the data from the Queue;
    3. MainApplication calls an external API to retrieve some data;
    4. MainApplication stores some data on the DB;
    5. MainApplication sends a message on the queue with the operation result.

    Happy flow for MainApplication

    Now, the happy flow is simple. But we should have covered what to do in case of an error.

    Introducing the Error Management Trio

    In general, errors that need to be handled fall into three categories (that I decided to call “the Error Management Trio”): data validation, transient errors, and faults.

    Data Validation focuses on the data used across the system, particularly the data you don’t control.

    Transient Errors occur when the application’s overall status or its dependencies temporarily change to an invalid state.

    Faults are errors that take down the whole application, and you cannot recover immediately.

    The Trio does not take into account “errors” that are not properly errors: null values, queries that do not return any value, and so on. These, in my opinion, are all legitimate statuses that represent that lack of values but are not errors that have architectural relevance.

    The Error Management Trio schema

    Data Validation: the first defence against invalid status

    The Data Validation category focuses on ensuring that relevant data is in a valid status.

    In particular, it aims at ensuring that data coming from external sources (for example, from the Body in an incoming HTTP request or from the result of a query on the database) is both syntactically and logically valid.

    Suppose that the messages we receive from the queue are in the following format:

    {
      "Username": "mr. captain",
      "BookId": 154,
      "Operation": "Add"
    }
    

    We definitely need to perform some sort of validation on the message content.

    For example:

    • The Username property must not be empty;
    • The BookId property must be a positive number;
    • The Operation property must have one of the following values: Add, Remove, Refresh;

    How does it impact our design?

    We have several choices to deal with an invalid incoming message:

    1. ignore the whole message: if it doesn’t pass the validation, discard the message;
    2. send the message back to the caller, describing the type of error
    3. try to fix it locally: if we are able to recreate a valid message, we could try to fix it and process the incoming message;
    4. try to fix it in a separate service: you will need to create a distinct service that receives the invalid message and tries to fix it: if it manages to fix the message, it re-inserts it in the original queue; otherwise, it sends a message to the response queue to notify about the impossibility to recreate a valid message.

    As you can see, even for the simple input validation, the choices we make can have an impact on the structure of the architecture.

    Suppose that you choose option #4: you will need to implement a brand new service (let’s call it ValidationFixesManager), configure a new queue, and keep track of the attempts to fix the message.

    Example of Architecture with ValidationFixesManager component

    All of this only when considering the static validation. How would you validate your business rules? How would you ensure that, for instance, the Username is valid and the user is still active on the system?

    Maybe you discover that the data stored on the database in incomplete or stale. Then you have to work out a way to handle such type of data.

    For example, you can:

    • run a background job that ensures that all the data is always valid;
    • enrich the data from the DB with newer data only when it is actually needed;
    • fine-tune the database consistency level.

    We have just demonstrated a simple but important fact: data validation looks trivial, but depending on the needs of your system, it may impact how you design your system.

    Transient Errors: temporary errors that may randomly occur

    Even if the validation passes, temporary issues may prevent your operations from completing.

    In the previous example, there are some possible cases to consider:

    1. the external API is temporarily down, and you cannot retrieve the data you need;
    2. the return queue is full, and you cannot add response messages;
    3. the application is not able to connect to the DB due to network issues;

    These kinds of issues are due to a temporary status of the system or of one of its dependencies.

    Sure, you may add automatic retries: for instance, you can use Polly to automatically retry access the API. But what if it’s not enough?

    Again, depending on your application’s requirements and the overall structure you started designing, solving this problem may bring you to unexpected paths.

    Let’s say that the external API is returning a 500 HTTP error: this is a transient error, and it does not depend on the content of the request: the API is down, an you cannot to anything to solve it.

    What can we do if all the retries fail?

    If we can just accept the situation, we can return the error to the caller and move on with the next operation.

    But if we need to keep trying until the operation goes well, we have (at least) two choices:

    1. consume the message from the Queue, try calling the API, and, if it fails, re-insert the message on the queue (ideally, with some delay);
    2. peek the message from the queue and try calling the API. If it fails, the message stays on the queue (and you need a way to read it again). Otherwise, we consider the message completed and remove it from the queue.

    These are just two of the different solutions. But, as you can see, this choice will have, in the long run, a huge effect on the future of the application, both in terms of maintainability and performance.

    Below is how the structure changes if we decide to send the failed messages back in the queue with some delay.

    The MainApplication now sends messages back on the queue

    In both cases, we must remember that trying to call a service that is temporarily down is useless: maybe it’s time to use a Circuit Breaker?

    Fatal Errors: when everything goes wrong

    There is one type of error that is often neglected but that may deeply influence how your system behaves: fatal errors.

    Examples of fatal errors are:

    • the host has consumed all the CPU or RAM;
    • the file system is corrupted;
    • the connection to an external system is interrupted due to network misconfigurations.

    In short, fatal errors are errors you have no way to solve in the short run: they happen and stop everything you are doing.

    This kind of error cannot be directly managed via application code, but you need to rely on other techniques.

    For example, to make sure you won’t consume all the available RAM, you should plan for autoscaling of your resources. So you have to design the system with autoscaling in mind: this means, for example, that the system must be stateless and the application must run on infrastructure objects that can be configured to automatically manage resources (like Azure Functions, Kubernetes, and Azure App Services). Also: do you need horizontal or vertical scaling?

    And, talking about the integrity of the system, how do you ensure that operations that were ongoing when the fatal error occurred can be completed?

    One possible solution is to use a database table to keep track of the status of each operation, so that when the application restarts, it first completes pending operations, and then starts working on new operations.

    A database keeps track of the failed operations

    A practical approach to address the Error Management Trio

    There are too many errors to manage and too much effort to cover everything!

    How can we cover everything? Well, it’s impossible: for every action we take to prevent an error, a new one may occur.

    Let’s jump back to the example we saw for handling validation errors (using a new service that tries to fix the message). What if the ValidationFixesManager service is down or the message queue is unreachable? We tried to solve a problem, but we ended up with two more to be managed!

    Let me introduce a practical approach to help you decide what needs to be addressed.

    Step 1: list all the errors you can think of. Create a table to list all the possible errors that you expect they can happen.

    You can add a column to describe the category the error falls into, as well as a Probability and Impact on the system column with a value (in this example, Low, Medium and High) that represents the probability that this error occurs and the impact it has on the overall application.

    Problem Category Probability Impact on the system
    Invalid message from queue Data Validation Medium High
    Invalid user data on DB Data Validation Low Medium
    Missing user on DB Data Validation Low Low
    API not reachable Transient High High
    DB not reachable Transient Low High
    File system corrupted Fatal Low High
    CPU limit reached Fatal Medium High

    From here, you can pick the most urgent elements to be addressed.

    Step 2: evaluate alternatives. Every error can be addressed in several ways (ignoring the error IS a valid alternative!). Take some time to explore all the alternatives.

    Again, a table can be a good companion for this step. You can describe, for example:
    the effort required to solve the error (Low, Medium, High)
    the positive and negative consequences in terms (also) of quality attributes (aka: “-ilities”). Maybe a solution works fine for data integrity but has a negative impact on maintainability.

    Step 3: use ADRs to describe how (and why) you will handle that specific error.

    Take your time to thoroughly describe, using ADR documents, the problems you are trying to solve, the solutions taken into consideration, and the final choice.

    Having everything written down in a shared file is fundamental for ensuring that, in the future, the present choices and necessities are taken into account, before saying “meh, that’s garbage!”

    Further readings

    Unfortunately, I feel that error handling is one of the most overlooked topics when designing a system. This also means that there are not lots and lots of articles and resources that explore this topic.

    But, if you use queues, one of the components you should use to manage errors is the Dead Letter queue. Here’s a good article by Dorin Baba where he explains how to use Dead Letter queues to handle errors in asynchronous systems.

    🔗 Handling errors like a pro or nah? Let’s talk about Dead Letters | Dorin Baba

    This article first appeared on Code4IT 🐧

    In this article, we used a Queue to trigger the beginning of the operation. When using Azure services, we have two types of message queues: Queues and Topics. Do you know the difference? Hint: other vendors use the same names to represent different concepts.

    🔗 Azure Service Bus: Queues vs Topics | Code4IT

    Whichever the way you chose to solve manage an error, always remember to write down the reasons that guided you to use that specific solution. An incredibly helpful way is by using ADRs.

    🔗 Tracking decision with Architecture Decision Records (ADRs) | CodeIT

    Wrapping up

    This article highlights the importance of error management and the fact that even if we all want to avoid and prevent errors in our systems, we still have to take care of them and plan according to our needs.

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn or Twitter! 🤜🤛

    Happy coding!

    🐧





    Source link

  • Easy logging management with Seq and ILogger in ASP.NET | Code4IT

    Easy logging management with Seq and ILogger in ASP.NET | Code4IT


    Seq is one of the best Log Sinks out there : it’s easy to install and configure, and can be added to an ASP.NET application with just a line of code.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    Logging is one of the most essential parts of any application.

    Wouldn’t it be great if we could scaffold and use a logging platform with just a few lines of code?

    In this article, we are going to learn how to install and use Seq as a destination for our logs, and how to make an ASP.NET 8 API application send its logs to Seq by using the native logging implementation.

    Seq: a sink and dashboard to manage your logs

    In the context of logging management, a “sink” is a receiver of the logs generated by one or many applications; it can be a cloud-based system, but it’s not mandatory: even a file on your local file system can be considered a sink.

    Seq is a Sink, and works by exposing a server that stores logs and events generated by an application. Clearly, other than just storing the logs, Seq allows you to view them, access their details, perform queries over the collection of logs, and much more.

    It’s free to use for individual usage, and comes with several pricing plans, depending on the usage and the size of the team.

    Let’s start small and install the free version.

    We have two options:

    1. Download it locally, using an installer (here’s the download page);
    2. Use Docker: pull the datalust/seq image locally and run the container on your Docker engine.

    Both ways will give you the same result.

    However, if you already have experience with Docker, I suggest you use the second approach.

    Once you have Docker installed and running locally, open a terminal.

    First, you have to pull the Seq image locally (I know, it’s not mandatory, but I prefer doing it in a separate step):

    Then, when you have it downloaded, you can start a new instance of Seq locally, exposing the UI on a specific port.

    docker run --name seq -d --restart unless-stopped -e ACCEPT_EULA=Y -p 5341:80 datalust/seq:latest
    

    Let’s break down the previous command:

    • docker run: This command is used to create and start a new Docker container.
    • --name seq: This option assigns the name seq to the container. Naming containers can make them easier to manage.
    • -d: This flag runs the container in detached mode, meaning it runs in the background.
    • --restart unless-stopped: This option ensures that the container will always restart unless it is explicitly stopped. This is useful for ensuring that the container remains running even after a reboot or if it crashes.
    • -e ACCEPT_EULA=Y: This sets an environment variable inside the container. In this case, it sets ACCEPT_EULA to Y, which likely indicates that you accept the End User License Agreement (EULA) for the software running in the container.
    • -p 5341:80: This maps port 5341 on your host machine to port 80 in the container. This allows you to access the service running on port 80 inside the container via port 5341 on your host.
    • datalust/seq:latest: This specifies the Docker image to use for the container. datalust/seq is the image name, and latest is the tag, indicating that you want to use the latest version of this image.

    So, this command runs a container named seq in the background, ensures it restarts unless stopped, sets an environment variable to accept the EULA, maps a host port to a container port, and uses the latest version of the datalust/seq image.

    It’s important to pay attention to the used port: by default, Seq uses port 5341 to interact with the UI and the API. If you prefer to use another port, feel free to do that – just remember that you’ll need some additional configuration.

    Now that Seq is installed on your machine, you can access its UI. Guess what? It’s on localhost:5341!

    Seq brand new instance

    However, Seq is “just” a container for our logs – but we have to produce them.

    A sample ASP.NET API project

    I’ve created a simple API project that exposes CRUD operations for a data model stored in memory (we don’t really care about the details).

    [ApiController]
    [Route("[controller]")]
    public class BooksController : ControllerBase
    {
        public BooksController()
        {
    
        }
    
        [HttpGet("{id}")]
        public ActionResult<Book> GetBook([FromRoute] int id)
        {
    
            Book? book = booksCatalogue.SingleOrDefault(x => x.Id == id);
            return book switch
            {
                null => NotFound(),
                _ => Ok(book)
            };
        }
    }
    

    As you can see, the details here are not important.

    Even the Main method is the default one:

    var builder = WebApplication.CreateBuilder(args);
    
    builder.Services.AddControllers();
    
    builder.Services.AddEndpointsApiExplorer();
    builder.Services.AddSwaggerGen();
    
    var app = builder.Build();
    
    if (app.Environment.IsDevelopment())
    {
        app.UseSwagger();
        app.UseSwaggerUI();
    }
    
    app.UseHttpsRedirection();
    
    app.MapControllers();
    
    app.Run();
    

    We have the Controllers, we have Swagger… well, nothing fancy.

    Let’s mix it all together.

    How to integrate Seq with an ASP.NET application

    If you want to use Seq in an ASP.NET application (may it be an API application or whatever else), you have to add it to the startup pipeline.

    First, you have to install the proper NuGet package: Seq.Extensions.Logging.

    The Seq.Extensions.Logging NuGet package

    Then, you have to add it to your Services, calling the AddSeq() method:

    var builder = WebApplication.CreateBuilder(args);
    
    builder.Services.AddControllers();
    
    builder.Services.AddEndpointsApiExplorer();
    builder.Services.AddSwaggerGen();
    
    + builder.Services.AddLogging(lb => lb.AddSeq());
    
    var app = builder.Build();
    

    Now, Seq is ready to intercept whatever kind of log arrives at the specified port (remember, in our case, we are using the default one: 5341).

    We can try it out by adding an ILogger to the BooksController constructor:

    private readonly ILogger<BooksController> _logger;
    
    public BooksController(ILogger<BooksController> logger)
    {
        _logger = logger;
    }
    

    So that we can use the _logger instance to create logs as we want, using the necessary Log Level:

    [HttpGet("{id}")]
    public ActionResult<Book> GetBook([FromRoute] int id)
    {
        _logger.LogInformation("I am Information");
        _logger.LogWarning("I am Warning");
        _logger.LogError("I am Error");
        _logger.LogCritical("I am Critical");
    
        Book? book = booksCatalogue.SingleOrDefault(x => x.Id == id);
        return book switch
        {
            null => NotFound(),
            _ => Ok(book)
        };
    }
    

    Log messages on Seq

    Using Structured Logging with ILogger and Seq

    One of the best things about Seq is that it automatically handles Structured Logging.

    [HttpGet("{id}")]
    public ActionResult<Book> GetBook([FromRoute] int id)
    {
        _logger.LogInformation("Looking if in my collection with {TotalBooksCount} books there is one with ID {SearchedId}"
     , booksCatalogue.Count, id);
    
        Book? book = booksCatalogue.SingleOrDefault(x => x.Id == id);
        return book switch
        {
            null => NotFound(),
            _ => Ok(book)
        };
    }
    

    Have a look at this line:

    _logger.LogInformation("Looking if in my collection with {TotalBooksCount} books there is one with ID {SearchedId}"
     , booksCatalogue.Count, id);
    

    This line generates a string message, replaces all the placeholders, and, on top of that, creates two properties, SearchedId and TotalBooksCount; you can now define queries using these values.

    Structured Logs in Seq allow you to view additional logging properties

    Further readings

    I have to admit it: logging management is one of my favourite topics.

    I’ve already written a sort of introduction to Seq in the past, but at that time, I did not use the native ILogger, but Serilog, a well-known logging library that added some more functionalities on top of the native logger.

    🔗 Logging with Serilog and Seq | Code4IT

    This article first appeared on Code4IT 🐧

    In particular, Serilog can be useful for propagating Correlation IDs across multiple services so that you can fetch all the logs generated by a specific operation, even though they belong to separate applications.

    🔗 How to log Correlation IDs in .NET APIs with Serilog

    Feel free to search through my blog all the articles related to logging – I’m sure you will find interesting stuff!

    Wrapping up

    I think Seq is the best tool for local development: it’s easy to download and install, supports structured logging, and can be easily added to an ASP.NET application with just a line of code.

    I usually add it to my private projects, especially when the operations I run are complex enough to require some well-structured log.

    Given how it’s easy to install, sometimes I use it for my work projects too: when I have to fix a bug, but I don’t want to use the centralized logging platform (since it’s quite complex to use), I add Seq as a destination sink, run the application, and analyze the logs in my local machine. Then, of course, I remove its reference, as I want it to be just a discardable piece of configuration.

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn, Twitter or BlueSky! 🤜🤛

    Happy coding!

    🐧





    Source link

  • Weekend Sale! 🎁

    Weekend Sale! 🎁


    At Browserling and Online Tools we love sales.

    We just created a new automated Weekend Sale.

    Now each weekend, we show a 50% discount offer to all users who visit our site.

    🔥 onlinetools.com/pricing

    🔥 browserling.com/#pricing

    Buy a subscription now and see you next time!



    Source link

  • 2 ways to generate realistic data using Bogus &vert; Code4IT

    2 ways to generate realistic data using Bogus | Code4IT


    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    In a previous article, we delved into the creation of realistic data using Bogus, an open-source library that allows you to generate data with plausible values.

    Bogus contains several properties and methods that generate realistic data such as names, addresses, birthdays, and so on.

    In this article, we will learn two ways to generate data with Bogus: both ways generate the same result; the main change is on the reusability and the modularity. But, in my opinion, it’s just a matter of preference: there is no approach absolutely better than the other. However, both methods can be preferred in specific cases.

    For the sake of this article, we are going to use Bogus to generate instances of the Book class, defined like this:

    public class Book
    {
        public Guid Id { get; set; }
        public string Title { get; set; }
        public int PagesCount { get; set; }
        public Genre[] Genres { get; set; }
        public DateOnly PublicationDate { get; set; }
        public string AuthorFirstName { get; set; }
        public string AuthorLastName { get; set; }
    }
    
    public enum Genre
    {
        Thriller, Fantasy, Romance, Biography
    }
    

    Expose a Faker inline or with a method

    It is possible to create a specific object that, using a Builder approach, allows you to generate one or more items of a specified type.

    It all starts with the Faker<T> generic type, where T is the type you want to generate.

    Once you create it, you can define the rules to be used when initializing the properties of a Book by using methods such as RuleFor and RuleForType.

    public static class BogusBookGenerator
    {
        public static Faker<Book> CreateFaker()
        {
            Faker<Book> bookFaker = new Faker<Book>()
             .RuleFor(b => b.Id, f => f.Random.Guid())
             .RuleFor(b => b.Title, f => f.Lorem.Text())
             .RuleFor(b => b.Genres, f => f.Random.EnumValues<Genre>())
             .RuleFor(b => b.AuthorFirstName, f => f.Person.FirstName)
             .RuleFor(b => b.AuthorLastName, f => f.Person.LastName)
             .RuleFor(nameof(Book.PagesCount), f => f.Random.Number(100, 800))
             .RuleForType(typeof(DateOnly), f => f.Date.PastDateOnly());
    
            return bookFaker;
        }
    }
    

    In this way, thanks to the static method, you can simply create a new instance of Faker<Book>, ask it to generate one or more books, and enjoy the result:

    Faker<Book> generator = BogusBookGenerator.CreateFaker();
    var books = generator.Generate(10);
    

    Clearly, it’s not necessary for the class to be marked as static: it all depends on what you need to achieve!

    Expose a subtype of Faker, specific for the data type to be generated

    If you don’t want to use a method (static or not static, it doesn’t matter), you can define a subtype of Faker<Book> whose customization rules are all defined in the constructor.

    public class BookGenerator : Faker<Book>
    {
        public BookGenerator()
        {
            RuleFor(b => b.Id, f => f.Random.Guid());
            RuleFor(b => b.Title, f => f.Lorem.Text());
            RuleFor(b => b.Genres, f => f.Random.EnumValues<Genre>());
            RuleFor(b => b.AuthorFirstName, f => f.Person.FirstName);
            RuleFor(b => b.AuthorLastName, f => f.Person.LastName);
            RuleFor(nameof(Book.PagesCount), f => f.Random.Number(100, 800));
            RuleForType(typeof(DateOnly), f => f.Date.PastDateOnly());
        }
    }
    

    Using this way, you can simply create a new instance of BookGenerator and, again, call the Generate method to create new book instances.

    var generator = new BookGenerator();
    var books = generator.Generate(10);
    

    Method vs Subclass: When should we use which?

    As we saw, both methods bring the same result, and their usage is almost identical.

    So, which way should I use?

    Use the method approach (the first one) when you need:

    • Simplicity: If you need to generate fake data quickly and your rules are straightforward, using a method is the easiest approach.
    • Ad-hoc Data Generation: Ideal for one-off or simple scenarios where you don’t need to reuse the same rules across your application.

    Or use the subclass (the second approach) when you need:

    • Reusability: If you need to generate the same type of fake data in multiple places, defining a subclass allows you to encapsulate the rules and reuse them easily.
    • Complex scenarios and extensibility: Better suited for more complex data generation scenarios where you might have many rules or need to extend the functionality.
    • Maintainability: Easier to maintain and update the rules in one place.

    Further readings

    If you want to learn a bit more about Bogus and use it to populate data used by Entity Framework, I recently published an article about this topic:

    🔗Seeding in-memory Entity Framework with realistic data with Bogus | Code4IT

    This article first appeared on Code4IT 🐧

    But, clearly, the best place to learn about Bogus is by reading the official documentation, that you can find on GitHub.

    🔗 Bogus repository | GitHub

    Wrapping up

    This article sort of complements the previous article about Bogus.

    I think Bogus is one of the best libraries in the .NET universe, as having realistic data can help you improve the intelligibility of the test cases you generate. Also, Bogus can be a great tool when you want to showcase demo values without accessing real data.

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn, Twitter or BlueSky! 🤜🤛

    Happy coding!

    🐧





    Source link

  • Happy May Sale! 🎁

    Happy May Sale! 🎁


    At Browserling and Online Tools we love sales.

    We just created a new automated sale called Happy May Sale.

    Now each new month on the first day we show a 50% discount offer to all users who visit our site.

    Buy a subscription now and see you next time!



    Source link

  • measuring things to ensure prosperity &vert; Code4IT

    measuring things to ensure prosperity | Code4IT


    Non-functional requirements matter, but we often forget to validate them. You can measure them by setting up Fitness Functions.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    Just creating an architecture is not enough; you should also make sure that the pieces of stuff you are building are, in the end, the once needed by your system.

    Is your system fast enough? Is it passing all the security checks? What about testability, maintainability, and other -ilities?

    Fitness Functions are components of the architecture that do not execute functional operations, but, using a set of tests and measurements, allow you to validate that the system respects all the non-functional requirements defined upfront.

    Fitness Functions: because non-functional requirements matter

    An architecture is made of two main categories of requirements: functional requirements and non-functional requirements.

    Functional requirements are the most easy to define and to test: if one of the requirements is “a user with role Admin must be able to see all data”, then writing a suite of tests for this specific requirement is pretty straightforward.

    Non-functional requirements are, for sure, as important as functional requirements, but are often overlooked or not detailed. “The system must be fast”: ok, how fast? What do you mean with “fast”? What is an acceptable value of “fast”?

    If we don’t have a clear understanding of non-functional requirements, then it’s impossible to measure them.

    And once we have defined a way to measure them, how can we ensure that we are meeting our expectations? Here’s where Fitness Functions come in handy.

    In fact, Fitness Functions are specific components that focus on non-functional requirements, executing some calculations and providing metrics that help architects and developers ensure that the system’s architecture aligns with business goals, technical requirements, and other quality attributes.

    Why Fitness Functions are crucial for future-proof architectures

    When creating an architecture, you must think of the most important -ilities for that specific case. How can you ensure that the technical choices we made meet the expectations?

    By being related to specific and measurable metrics, Fitness Functions provide a way to assess the architecture’s quality and performance, reducing the reliance on subjective opinions by using objective measurements. A metric can be a simple number (e.g., “maximum number of requests per second”), a percentage value (like “percentage of code covered by tests”) or other values that are still measurable.

    Knowing how the system behaves in regards to these measures allows architects to work on the continuous improvement of the system: teams can identify areas for improvement and make decisions based not on personal opinion but on actual data to enhance the system.

    Having a centralized place to view the historical values of a measure helps understanding if you have done progresses or, as time goes by, the quality has degraded.

    Still talking about the historical values of the measures, having a clear understanding of what is the current status of such metrics can help in identifying potential issues early in the development process, allowing teams to address them before they become critical problems.

    For example, by using Fitness Functions, you can ensure that the system is able to handle a certain amount of users per second: having proper measurements, you can identify which functionalities are less performant and, in case of high traffic, may bring the whole system down.

    You are already using Fitness Functions, but you didn’t know

    Fitness Functions sound like complex things to handle.

    Even though you can create your own functions, most probably you are already using them without knowing it. Lots of tools are available out there that cover several metrics, and I’m sure you’ve already used some of them (or, at least, you’ve already heard of them).

    Tools like SonarQube and NDepend use Fitness Functions to evaluate code quality based on metrics such as code complexity, duplication, and adherence to coding standards. Those metrics are calculated based on static analysis of the code, and teams can define thresholds under which a system can be at risk of losing maintainability. An example of metric related to code quality is Code Coverage: the higher, the better (even though 100% of code coverage does not guarantee your code is healthy).

    Tools like JMeter or K6 help you measure system performance under various conditions: having a history of load testing results can help ensure that, as you add new functionalities to the system, the performance on some specific modules does not downgrade.

    All in all, most of the Fitness Functions can be set to be part of CI/CD pipelines: for example, you can configure a CD pipeline to block the deployment of the code on a specific system if the load testing results of the new code are worse than the previous version. Or you could block a Pull Request if the code coverage percentage is getting lower.

    Further readings

    A good way to start experimenting with Load Testing is by running them locally. A nice open-source project is K6: you can install it on your local machine, define the load phases, and analyze the final result.

    🔗 Getting started with Load testing with K6 on Windows 11 | Code4IT

    This article first appeared on Code4IT 🐧

    But, even if you don’t really care about load testing (maybe because your system is not expected to handle lots of users), I’m sure you still care about code quality and their tests. When using .NET, you can collect code coverage reports using Cobertura. Then, if you are using Azure DevOps, you may want to stop a Pull Request if the code coverage percentage has decreased.

    Here’s how to do all of this:

    🔗 Cobertura, YAML, and Code Coverage Protector: how to view Code Coverage report on Azure DevOps | Code4IT

    Wrapping up

    Sometimes, there are things that we use every day, but we don’t know how to name them: Fitness Functions are one of them – and they are the foundation of future-proof software systems.

    You can create your own Fitness Functions based on whatever you can (and need to) measure: from average page loading to star-rated customer satisfaction. In conjunction with a clear dashboard, you can provide a clear view of the history of such metrics.

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn, Twitter or BlueSky! 🤜🤛

    Happy coding!

    🐧





    Source link

  • End of April Sale! 🎁

    End of April Sale! 🎁


    At Browserling and Online Tools, we love sales.

    We just created new automated sale – now each year, on the last day of April, we run an End of April Sale!

    🔥 onlinetools.com/pricing

    🔥 browserling.com/#pricing

    Buy a subscription now and see you next time!



    Source link

  • HTTP Logging in ASP.NET: how to automatically log all incoming HTTP requests (and its downsides!)

    HTTP Logging in ASP.NET: how to automatically log all incoming HTTP requests (and its downsides!)


    Aren’t you tired of adding manual logs to your HTTP APIs to log HTTP requests and responses? By using a built-in middleware in ASP.NET, you will be able to centralize logs management and have a clear view of all the incoming HTTP requests.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    Whenever we publish a service, it is important to add proper logging to the application. Logging helps us understand how the system works and behaves, and it’s a fundamental component that allows us to troubleshoot problems that occur during the actual usage of the application.

    In this blog, we have talked several times about logging. However, we mostly focused on the logs that were written manually.

    In this article, we will learn how to log incoming HTTP requests to help us understand how our APIs are being used from the outside.

    Scaffolding the empty project

    To showcase this type of logging, I created an ASP.NET API. It’s a very simple application with CRUD operations on an in-memory collection.

    [ApiController]
    [Route("[controller]")]
    public class BooksController : ControllerBase
    {
        private readonly List<Book> booksCatalogue = Enumerable.Range(1, 5).Select(index => new Book
        {
            Id = index,
            Title = $"Book with ID {index}"
        }).ToList();
    
        private readonly ILogger<BooksController> _logger;
    
        public BooksController(ILogger<BooksController> logger)
        {
            _logger = logger;
        }
    }
    

    These CRUD operations are exposed via HTTP APIs, following the usual verb-based convention.

    For example:

    [HttpGet("{id}")]
    public ActionResult<Book> GetBook([FromRoute] int id)
    {
        _logger.LogInformation("Looking if in my collection with {TotalBooksCount} books there is one with ID {SearchedId}"
                , booksCatalogue.Count, id);
    
        Book? book = booksCatalogue.SingleOrDefault(x => x.Id == id);
    
        return book switch
        {
            null => NotFound(),
            _ => Ok(book)
        };
    }
    

    As you can see, I have added some custom logs: before searching for the element with the specified ID, I also wrote a log message such as “Looking if in my collection with 5 books there is one with ID 2”.

    Where can I find the message? For the sake of this article, I decided to use Seq!

    Seq is a popular log sink (well, as you may know, my favourite one!), that is easy to install and to integrate with .NET. I’ve thoroughly explained how to use Seq in conjunction with ASP.NET in this article and in other ones.

    In short, the most important change in your application is to add Seq as the log sink, like this:

    builder.Services.AddLogging(lb => {
        lb.AddSeq();
    });
    

    Now, whenever I call the GET endpoint, I can see the related log messages appear in Seq:

    Custom log messages

    But sometimes it’s not enough. I want to see more details, and I want them to be applied everywhere!

    How to add HTTP Logging to an ASP.NET application

    HTTP Logging is a way of logging most of the details of the incoming HTTP operations, tracking both the requests and the responses.

    With HTTP Logging, you don’t need to manually write custom logs to access the details of incoming requests: you just need to add its related middleware, configure it as you want, and have all the required logs available for all your endpoints.

    Adding it is pretty straightforward: you first need to add the HttpLogging middleware to the list of services:

    builder.Services.AddHttpLogging(lb => { });
    

    so that you can use it once the WebApplication instance is built:

    There’s still a problem, though: all the logs generated via HttpLogging are, by default, ignored, as logs coming from their namespace (named Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware) are at Information log level, thus ignored because of the default configurations.

    You either have to update the appsetting.json file to tell the logging system to process logs from that namespace:

    {
      "Logging": {
        "LogLevel": {
          "Default": "Information",
          "Microsoft.AspNetCore": "Warning",
          "Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware": "Information"
        }
      }
    }
    

    or, alternatively, you need to do the same when setting up the logging system in the Program class:

    builder.Services.AddLogging(lb => {
      lb.AddSeq();
    + lb.AddFilter("Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware", LogLevel.Information);
    });
    

    We then have all our pieces in place: let’s execute the application!

    First, you can spin up the API; you should be able to see the Swagger page:

    Swagger page for our application&rsquo;s API

    From here, you can call the GET endpoint:

    Http response of the API call, as seen on Swagger

    You should now able to see all the logs in Seq:

    Logs list in Seq

    As you can see from the screenshot above, I have a log entry for the request and one for the response. Also, of course, I have the custom message I added manually in the C# method.

    Understanding HTTP Request logs

    Let’s focus on the data logged for the HTTP request.

    If we open the log related to the HTTP request, we can see all these values:

    Details of the HTTP Request

    Among these details, we can see properties such as:

    • the host name (localhost:7164)
    • the method (GET)
    • the path (/books/4)

    and much more.

    You can see all the properties as standalone items, but you can also have a grouped view of all the properties by accessing the HttpLog element:

    Details of the HTTP Log element

    Notice that for some elements we do not have access to the actual value, as the value is set to [Redacted]. This is a default configuration that prevents logging too many things (and undisclosing some values) as well as writing too much content on the log sink (the more you write, the less performant the queries become – and you also pay more!).

    Among other redacted values, you can see that even the Cookie value is not directly available – for the same reasons explained before.

    Understanding HTTP Response logs

    Of course, we can see some interesting data in the Response log:

    Details of the HTTP Response

    Here, among some other properties such as the Host Name, we can see the Status Code and the Trace Id (which, as you may notice, is the same as the one in te Request).

    As you can see, the log item does not contain the body of the response.

    Also, just as it happens with the Request, we do not have access to the list of HTTP Headers.

    How to save space, storage, and money by combining log entries

    For every HTTP operation, we end up with 2 log entries: one for the Request and one for the Response.

    However, it would be more practical to have both request and response info stored in the same log item so we can understand more easily what is happening.

    Lucky for us, this functionality is already in place. We just need to set the CombineLogs property to true when we add the HttpLogging functionality:

    builder.Services.AddHttpLogging(lb =>
    {
    +  lb.CombineLogs = true;
    }
    );
    

    Then, we are able to see the data for both the request and the related response in the same log element.

    Request and Response combined logs

    The downsides of using HTTP Logging

    Even though everything looks nice and pretty, adding HTTP Logging has some serious consequences.

    First of all, remember that you are doing some more operations for every incoming HTTP request. Just processing and storing the log messages can bring to an application performance downgrade – you are using parts of the processing resources to interpret the HTTP context, create the correct log entry, and store it.

    Depending on how your APIs are structured, you may need to strip out sensitive data: HTTP Logs, by default, log almost everything (except for the parts stored as Redacted). Since you don’t want to store as plain text the content of the requests, you may need to create custom logic to redact parts of the request and response you want to hide: you may need to implement a custom IHttpLoggingInterceptor.

    Finally, consider that logging occupies storage, and storage has a cost. The more you log, the higher the cost. You should define proper strategies to avoid excessive storage costs while keeping valuable logs.

    Further readings

    There is a lot more, as always. In this article, I focused on the most essential parts, but the road to having proper HTTP Logs is still long.

    You may want to start from the official documentation, of course!

    🔗 HTTP logging in ASP.NET Core | Microsoft Docs

    This article first appeared on Code4IT 🐧

    All the logs produced for this article were stored on Seq. You can find more info about installing and integrating Seq in ASP.NET Core in this article:

    🔗 Easy logging management with Seq and ILogger in ASP.NET | Code4IT

    Wrapping up

    HTTP Logging can be a good tool for understanding the application behaviour and detecting anomalies. However, as you can see, there are some important downsides that need to be considered.

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn or Twitter! 🤜🤛

    Happy coding!

    🐧





    Source link

  • How to create Custom Attributes, and why they are useful &vert; Code4IT

    How to create Custom Attributes, and why they are useful | Code4IT


    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    In C#, attributes are used to describe the meaning of some elements, such as classes, methods, and interfaces.

    I’m sure you’ve already used them before. Examples are:

    • the [Required] attribute when you define the properties of a model to be validated;
    • the [Test] attribute when creating Unit Tests using NUnit;
    • the [Get] and the [FromBody] attributes used to define API endpoints.

    As you can see, all the attributes do not specify the behaviour, but rather, they express the meaning of a specific element.

    In this article, we will learn how to create custom attributes in C# and some possible interesting usages of such custom attributes.

    Create a custom attribute by inheriting from System.Attribute

    Creating a custom attribute is pretty straightforward: you just need to create a class that inherits from System.Attribute.

    [AttributeUsage(AttributeTargets.Interface | AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = true)]
    public class ApplicationModuleAttribute : Attribute
    {
     public Module BelongingModule { get; }
    
     public ApplicationModuleAttribute(Module belongingModule)
       {
     BelongingModule = belongingModule;
       }
    }
    
    public enum Module
    {
     Authentication,
     Catalogue,
     Cart,
     Payment
    }
    

    Ideally, the class name should end with the suffix -Attribute: in this way, you can use the attribute using the short form [ApplicationModule] rather than using the whole class name, like [ApplicationModuleAttribute]. In fact, C# attributes can be resolved by convention.

    Depending on the expected usage, a custom attribute can have one or more constructors and can expose one or more properties. In this example, I created a constructor that accepts an enum.
    I can then use this attribute by calling [ApplicationModule(Module.Cart)].

    Define where a Custom Attribute can be applied

    Have a look at the attribute applied to the class definition:

    [AttributeUsage(AttributeTargets.Interface | AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = true)]
    

    This attribute tells us that the ApplicationModule can be applied to interfaces, classes, and methods.

    System.AttributeTargets is an enum that enlists all the points you can attach to an attribute. The AttributeTargets enum is defined as:

    [Flags]
    public enum AttributeTargets
    {
     Assembly = 1,
     Module = 2,
     Class = 4,
     Struct = 8,
     Enum = 16,
     Constructor = 32,
     Method = 64,
     Property = 128,
     Field = 256,
     Event = 512,
     Interface = 1024,
     Parameter = 2048,
     Delegate = 4096,
     ReturnValue = 8192,
     GenericParameter = 16384,
     All = 32767
    }
    

    Have you noticed it? It’s actually a Flagged enum, whose values are powers of 2: this trick allows us to join two or more values using the OR operator.

    There’s another property to notice: AllowMultiple. When set to true, this property tells us that it’s possible to use apply more than one attribute of the same type to the same element, like this:

    [ApplicationModule(Module.Cart)]
    [ApplicationModule(Module.Catalogue)]
    public class ItemDetailsService { }
    

    Or, if you want, you can inline them:

    [ApplicationModule(Module.Cart), ApplicationModule(Module.Catalogue)]
    public class ItemDetailsService { }
    

    Practical usage of Custom Attributes

    You can use custom attributes to declare which components or business areas an element belongs to.

    In the previous example, I defined an enum that enlists all the business modules supported by my application:

    public enum Module
    {
        Authentication,
        Catalogue,
        Cart,
        Payment
    }
    

    This way, whenever I define an interface, I can explicitly tell which components it belongs to:

    [ApplicationModule(Module.Catalogue)]
    public interface IItemDetails
    {
        [ApplicationModule(Module.Catalogue)]
        string ShowItemDetails(string itemId);
    }
    
    [ApplicationModule(Module.Cart)]
    public interface IItemDiscounts
    {
        [ApplicationModule(Module.Cart)]
        bool CanHaveDiscounts(string itemId);
    }
    

    Not only that: I can have one single class implement both interfaces and mark it as related to both the Catalogue and the Cart areas.

    [ApplicationModule(Module.Cart)]
    [ApplicationModule(Module.Catalogue)]
    public class ItemDetailsService : IItemDetails, IItemDiscounts
    {
        [ApplicationModule(Module.Catalogue)]
        public string ShowItemDetails(string itemId) => throw new NotImplementedException();
    
        [ApplicationModule(Module.Cart)]
        public bool CanHaveDiscounts(string itemId) => throw new NotImplementedException();
    }
    

    Notice that I also explicitly enriched the two inner methods with the related attribute – even if it’s not necessary.

    Further readings

    As you noticed, the AttributeTargets is a Flagged Enum. Don’t you know what they are and how to define them? I’ve got you covered! I wrote two articles about Enums, and you can find info about Flagged Enums in both articles:

    🔗 5 things you should know about enums in C# | Code4IT

    and
    🔗 5 more things you should know about enums in C# | Code4IT

    This article first appeared on Code4IT 🐧

    There are some famous but not-so-obvious examples of attributes that you should know: DebuggerDisplay and InternalsVisibleTo.

    DebuggerDisplay can be useful for improving your debugging sessions.

    🔗 Simplify debugging with DebuggerDisplay attribute dotNET | Code4IT

    IntenalsVisibleTo can be used to give access to internal classes to external projects:;for example, you can use that attribute when writing unit tests.

    🔗 Testing internal members with InternalsVisibleTo | Code4IT

    Wrapping up

    In this article, I showed you how to create custom attributes in C# to specify which modules a class or a method belongs to. This trick can be useful if you want to speed up the analysis of your repository: if you need to retrieve all the classes that are used for the Cart module (for example, because you want to move them to an external library), you can just search for Module.Cart across the repository and have a full list of elements.

    In particular, this approach can be useful for the exposed components, such as API controllers. Knowing that two or more modules use the same Controller can help you understand if a change in the API structure is necessary.

    Another good usage of this attribute is automatic documentation: you could create a tool that automatically enlists all the interfaces, API endpoints, and classes grouped by the belonging module. The possibilities are infinite!

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn, Twitter or BlueSky! 🤜🤛

    Happy coding!

    🐧





    Source link

  • like Mermaid, but better. Syntax, installation, and practical usage tips &vert; Code4IT

    like Mermaid, but better. Syntax, installation, and practical usage tips | Code4IT


    D2 is an open-source tool to design architectural layouts using a declarative syntax. It’s a textual format, which can also be stored under source control. Let’s see how it works, how you can install it, and some practical usage tips.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    When defining the architecture of a system, I believe in the adage that says that «A picture is worth a thousand words».

    Proper diagramming helps in understanding how the architecture is structured, the dependencies between components, how the different components communicate, and their responsibilities.

    A clear architectural diagram can also be useful for planning. Once you have a general idea of the components, you can structure the planning according to the module dependencies and the priorities.

    A lack of diagramming leads to a “just words” definition: how many times have you heard people talk about modules that do not exist or do not work as they were imagining?

    The whole team can benefit from having a common language: a clear diagram brings clear thoughts, helping all the stakeholders (developers, architects, managers) understand the parts that compose a system.

    I tried several approaches: both online WYSIWYG tools like Draw.IO and DSL like Structurizr and Mermaid. For different reasons, I wasn’t happy with any of them.

    Then I stumbled upon D2: its rich set of elements makes it my new go-to tool for describing architectures. Let’s see how it works!

    A quick guide to D2 syntax

    Just like the more famous Mermaid, when using D2, you have to declare all the elements and connections as textual nodes.

    You can generate diagrams online by using the Playground section available on the official website, or you can install it locally (as you will see later).

    Elements: the basic components of every diagram

    Elements are defined as a set of names that can be enriched with a label and other metadata.

    Here’s an example of the most straightforward configurations for standalone elements.

    service
    
    user: Application User
    
    job: {
      shape: hexagon
    }
    

    For each element, you can define its internal name (service), a label (user: Application User) and a shape (shape: hexagon).

    A simple diagram with only two unrelated elements

    Other than that, I love the fact that you can define elements to be displayed as multiple instances: this can be useful when a service has multiple instances of the same type, and you want to express it clearly without the need to manually create multiple elements.

    You can do it by setting the multiple property to true.

    apiGtw: API Gateway {
      shape: cloud
    }
    be: BackEnd {
      style.multiple: true
    }
    
    apiGtw -> be
    

    Simple diagram with multiple backends

    Grouping: nesting elements hierarchically

    You may want to group elements. You can do that by using a hierarchical structure.

    In the following example, the main container represents my e-commerce application, composed of a website and a background job. The website is composed of a frontend, a backend, and a database.

    ecommerce: E-commerce {
      website: User Website {
        frontend
        backend
        database: DB {
          shape: cylinder
        }
      }
    
      job: {
        shape: hexagon
      }
    }
    

    As you can see from the diagram definition, elements can be nested in a hierarchical structure using the {} symbols. Of course, you can still define styles and labels to nested elements.

    Diagram with nested elements

    Connections: making elements communicate

    An architectural diagram is helpful only if it can express connections between elements.

    To connect two elements, you must use the --, the -> or the <- connector. You have to link their IDs, not their labels.

    ecommerce: E-commerce {
        website: User Website {
            frontend
        backend
        database: DB {
            shape: cylinder
        }
        frontend -> backend
        backend -> database: retrieve records {
            style.stroke: red
        }
      }
    
      job: {
          shape: hexagon
      }
      job -> website.database: update records
    }
    

    The previous example contains some interesting points.

    • Elements within the same container can be referenced directly using their ID: frontend -> backend.
    • You can add labels to a connection: backend -> database: retrieve records.
    • You can apply styles to a connection, like choosing the arrow colour with style.stroke: red.
    • You can create connections between elements from different containers: job -> website.database.

    Connections between elements from different containers

    When referencing items from different containers, you must always include the container ID: job -> website.database works, but job -> database doesn’t because database is not defined (so it gets created from scratch).

    SQL Tables: represent the table schema

    An interesting part of D2 diagrams is the possibility of adding the description of SQL tables.

    Obviously, the structure cannot be validated: the actual syntax depends on the database vendor.

    However, having the table schema defined in the diagram can be helpful in reasoning around the dependencies needed to complete a development.

    serv: Products Service
    
    db: Database Schema {
      direction: right
      shape: cylinder
      userTable: dbo.user {
        shape: sql_table
        Id: int {constraint: primary_key}
        FirstName: text
        LastName: text
        Birthday: datetime2
      }
    
      productsTable: dbo.products {
        shape: sql_table
        Id: int {constraint: primary_key}
        Owner: int {constraint: foreign_key}
        Description: text
      }
    
      productsTable.Owner -> userTable.Id
    }
    
    serv -> db.productsTable: Retrieve products by user id
    

    Diagram with database tables

    Notice how you can also define constraints to an element, like {constraint: foreign_key}, and specify the references from one table to another.

    How to install and run D2 locally

    D2 is a tool written in Go.

    Go is not natively present in every computer, so you have to install it. You can learn how to install it from the official page.

    Once Go is ready, you can install D2 in several ways. I use Windows 11, so my preferred installation approach is to use a .msi installer, as described here.

    If you are on macOS, you can use Homebrew to install it by running:

    Regardless of the Operating System, you can have Go directly install D2 by running the following command:

    go install oss.terrastruct.com/d2@latest
    

    It’s even possible to install it via Docker. However, this approach is quite complex, so I prefer installing D2 directly with the other methods I explained before.

    You can find more information about the several installation approaches on the GitHub page of the project.

    Use D2 via command line

    To work with D2 diagrams, you need to create a file with the .d2 extension. That file will contain the textual representation of the diagrams, following the syntax we saw before.

    Once D2 is installed and the file is present in the file system (in my case, I named the file my-diagram.d2), you can use the console to generate the diagram locally – remember, I’m using Windows11, so I need to run the exe file:

    d2.exe --watch .\my-diagram.d2
    

    Now you can open your browser, head to the localhost page displayed on the shell, and see how D2 renders the local file. Thanks to the --watch flag, you can update the file locally and see the result appear on the browser without the need to restart the application.

    When the diagram is ready, you can export it as a PNG or SVG by running

    d2.exe .\my-diagram.d2 my-wonderful-design.png
    

    Create D2 Diagrams on Visual Studio Code

    Another approach is to install the D2 extension on VS Code.

    D2 extension on Visual Studio Code

    Thanks to this extension, you can open any D2 file and, by using the command palette, see a preview of the final result. You can also format the document to have the diagram definition tidy and well-structured.

    D2 extension command palette

    How to install and use D2 Diagrams on Obsidian

    Lastly, D2 can be easily integrated with tools like Obsidian. Among the community plugins, you can find the official D2 plugin.

    D2 plugin for Obsidian

    As you can imagine, Go is required on your machine.
    And, if necessary, you are required to explicitly set the path to the bin folder of Go. In my case, I had to set it to C:\Users\BelloneDavide\go\bin\.

    D2 plugin settings for Obsidian

    To insert a D2 diagram in a note generated with Obsidian, you have to use d2 as a code fence language.

    Practical tips for using D2

    D2 is easy to use once you have a basic understanding of how to create elements and connections.

    However, some tips may be useful to ease the process of creating the diagrams. Or, at least, these tips helped me write and maintain my diagrams.

    Separate elements and connections definition

    A good approach is to declare the application’s structure first, and then list all the connections between elements unless the elements are within the same components and are not expected to change.

    ecommerce: E-commerce {
      website: User Website {
        backend
        database: DB {
          shape: cylinder
        }
    
        backend -> database: retrieve records {
          style.stroke: red
        }
      }
    
      job -> website.database: update records
    }
    

    Here, the connection between backend and database is internal to the website element, so it makes sense to declare it directly within the website element.

    However, the other connection between the job and the database is cross-element. In the long run, it may bring readability problems.

    So, you could update it like this:

    ecommerce: E-commerce {
     website: User Website {
     backend
     database: DB {
     shape: cylinder
     }
    
     backend -> database: retrieve records {
     style.stroke: red
     }
     }
    
    - job -> website.database: update records
    }
    
    + ecommerce.job -> ecommerce.website.database: update records
    

    This tip can be extremely useful when you have more than one element with the same name belonging to different parents.

    Needless to say, since the order of the connection declarations does not affect the final rendering, write them in an organized way that best fits your needs. In general, I prefer creating sections (using comments to declare the area), and grouping connections by the outbound module.

    Pick a colour theme (and customize it, if you want!)

    D2 allows you to specify a theme for the diagram. There are some predefined themes (which are a set of colour palettes), each with a name and an ID.

    To use a theme, you have to specify it in the vars element on top of the diagram:

    vars: {
      d2-config: {
        theme-id: 103
      }
    }
    

    103 is the theme named “Earth tones”, using a brown-based palette that, when applied to the diagram, renders it like this.

    Diagram using the 103 colour palette

    However, if you have a preferred colour palette, you can use your own colours by overriding the default values:

    vars: {
      d2-config: {
        # Terminal theme code
        theme-id: 103
        theme-overrides: {
          B4: "#C5E1A5"
        }
      }
    }
    

    Diagram with a colour overridden

    You can read more about themes and customizations here.

    What is that B4 key overridden in the previous example? Unfortunately, I don’t know: you must try all the variables to understand how the diagram is rendered.

    Choose the right layout engine

    You can choose one of the three supported layout engines to render the elements in a different way (more info here).

    DAGRE and ELK are open source, but quite basic. TALA is more sophisticated, but it requires a paid licence.

    Here’s an example of how the same diagram is rendered using the three different engines.

    A comparison betweel DAGRE, ELK and TALA layout engines

    You can decide which engine to use by declaring it in the layout-engine element:

    vars: {
      d2-config: {
        layout-engine: tala
      }
    }
    

    Choosing the right layout engine can be beneficial because sometimes some elements are not rendered correctly: here’s a weird rendering with the DAGRE engine.

    DAGRE engine with a weird rendering

    Use variables to simplify future changes

    D2 allows you to define variables in a single place and have the same value repeated everywhere it’s needed.

    So, for example, instead of having

    mySystem: {
      reader: Magazine Reader
      writer: Magazine Writer
    }
    

    With the word “Magazine” repeated, you can move it to a variable, so that it can change in the future:

    vars: {
      entityName: Magazine
    }
    
    mySystem: {
      reader: ${entityName} Reader
      writer: ${entityName} Writer
    }
    

    If in the future you’ll have to handle not only Magazines but also other media types, you can simply replace the value of entityName in one place and have it updated all over the diagram.

    D2 vs Mermaid: a comparison

    D2 and Mermaid are similar but have some key differences.

    They both are diagram-as-a-code tools, meaning that the definition of a diagram is expressed as a text file, thus making it available under source control.

    Mermaid is already supported by many tools, like Azure DevOps wikis, GitHub pages, and so on.
    On the contrary, D2 must be installed (along with the Go language).

    Mermaid is quite a “close” system: even if it allows you to define some basic styles, it’s not that flexible.

    On the contrary, D2 allows you to choose a theme for the whole diagram, as well as choosing different layout engines.
    Also, D2 has some functionalities that are (currently) missing on Mermaid:

    Mermaid, on the contrary, allows us to define more types of diagrams: State Diagrams, Gantt, Mindmaps, and so on. Also, as we saw, it’s already supported on many platforms.

    So, my (current) choice is: use D2 for architectural diagrams, and use Mermaid for everything else.

    I haven’t tried D2 for Sequence Diagrams yet, so I won’t express an opinion on that.

    Further readings

    D2 is available online with a playground you can use to try things out in a sandboxed environment.

    🔗 D2 Playground

    All the documentation can be found on GitHub or on the official website:

    🔗 D2 documentation

    And, if you want, you can use icons to create better diagrams: D2 exposes a set of SVG icons that can be easily integrated into your diagrams. You can find them here:

    🔗 D2 predefined icons

    This article first appeared on Code4IT 🐧

    Ok, but diagrams have to live in a context. How can you create useful and maintainable documentation for your future self?

    A good way to document your architectural choices is to define ADRs (Architecture Decision Records), as explained here:

    🔗 Tracking decision with Architecture Decision Records (ADRs) | Code4IT

    And, of course, just the architectural diagram is not enough: you should also describe the dependencies, the constraints, the deployment strategies, and so on. Arc42 is a template that can guide you to proper system documentation:

    🔗 Arc42 Documentation, for a comprehensive description of your project | Code4IT

    Wrapping up

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn, Twitter or BlueSky! 🤜🤛

    Happy coding!

    🐧





    Source link