بلاگ

  • Matrix Sentinels: Building Dynamic Particle Trails with TSL

    Matrix Sentinels: Building Dynamic Particle Trails with TSL


    While experimenting with particle systems, I challenged myself to create particles with tails, similar to snakes moving through space. At first, I didn’t have access to TSL, so I tested basic ideas, like using noise derivatives and calculating previous steps for each particle, but none of them worked as expected.

    I spent a long time pondering how to make it work, but all my solutions involved heavy testing with WebGL and GPGPU, which seemed like it would require too much code for a simple proof of concept. That’s when TSL (Three.js Shader Language) came into play. With its Compute Shaders, I was able to compute arrays and feed the results into materials, making it easier to test ideas quickly and efficiently. This allowed me to accomplish the task without much time lost.

    Now, let’s dive into the step-by-step process of building the particle system, from setting up the environment to creating the trails and achieving that fluid movement.

    Step 1: Set Up the Particle System

    First, we’ll define the necessary uniforms that will be used to create and control the particles in the system.

    uniforms = {
        color: uniform( new THREE.Color( 0xffffff ).setRGB( 1, 1, 1 ) ),
        size: uniform( 0.489 ),
    
        uFlowFieldInfluence: uniform( 0.5 ),
        uFlowFieldStrength: uniform( 3.043 ),
        uFlowFieldFrequency: uniform( 0.207 ),
    }

    Next, create the variables that will define the parameters of the particle system. The “tails_count” variable determines how many segments each snake will have, while the “particles_count” defines the total number of segments in the scene. The “story_count” variable represents the number of frames used to store the position data for each segment. Increasing this value will increase the distance between segments, as we will store the position history of each one. The “story_snake” variable holds the history of one snake, while “full_story_length” stores the history for all snakes. These variables will be enough to bring the concept to life.

    tails_count = 7 //  n-1 point tails
    particles_count = this.tails_count * 200 // need % tails_count
    story_count = 5 // story for 1 position
    story_snake = this.tails_count * this.story_count
    full_story_length = ( this.particles_count / this.tails_count ) * this.story_snake

    Next, we need to create the buffers required for the computational shaders. The most important buffer to focus on is the “positionStoryBuffer,” which will store the position history of all segments. To understand how it works, imagine a train: the head of the train sets the direction, and the cars follow in the same path. By saving the position history of the head, we can use that data to determine the position of each car by referencing its position in the history.

    const positionsArray = new Float32Array( this.particles_count * 3 )
    const lifeArray = new Float32Array( this.particles_count )
    
    const positionInitBuffer = instancedArray( positionsArray, 'vec3' );
    const positionBuffer = instancedArray( positionsArray, 'vec3' );
    
    // Tails
    const positionStoryBuffer = instancedArray( new Float32Array( this.particles_count * this.tails_count * this.story_count ), 'vec3' );
    
    const lifeBuffer = instancedArray( lifeArray, 'float' );

    Now, let’s create the particle system with a material. I chose a standard material because it allows us to use an emissiveNode, which will interact with Bloom effects. For each segment, we’ll use a sphere and disable frustum culling to ensure the particles don’t accidentally disappear off the screen.

    const particlesMaterial = new THREE.MeshStandardNodeMaterial( {
        metalness: 1.0,
        roughness: 0
    } );
        
    particlesMaterial.emissiveNode = color(0x00ff00)
    
    const sphereGeometry = new THREE.SphereGeometry( 0.1, 32, 32 );
    
    const particlesMesh = this.particlesMesh = new THREE.InstancedMesh( sphereGeometry, particlesMaterial, this.particles_count );
    particlesMesh.instanceMatrix.setUsage( THREE.DynamicDrawUsage );
    particlesMesh.frustumCulled = false;
    
    this.scene.add( this.particlesMesh )
    

    Step 2: Initialize Particle Positions

    To initialize the positions of the particles, we’ll use a computational shader to reduce CPU usage and speed up page loading. We randomly generate the particle positions, which form a pseudo-cube shape. To keep the particles always visible on screen, we assign them a lifetime after which they disappear and won’t reappear from their starting positions. The “cycleStep” helps us assign each snake its own random positions, ensuring the tails are generated in the same location as the head. Finally, we send this data to the computation process.

    const computeInit = this.computeInit = Fn( () => {
        const position = positionBuffer.element( instanceIndex )
        const positionInit = positionInitBuffer.element( instanceIndex );
        const life = lifeBuffer.element( instanceIndex )
    
        // Position
        position.xyz = vec3(
            hash( instanceIndex.add( uint( Math.random() * 0xffffff ) ) ),
            hash( instanceIndex.add( uint( Math.random() * 0xffffff ) ) ),
            hash( instanceIndex.add( uint( Math.random() * 0xffffff ) ) )
        ).sub( 0.5 ).mul( vec3( 5, 5, 5 ) );
    
        // Copy Init
        positionInit.assign( position )
    
        const cycleStep = uint( float( instanceIndex ).div( this.tails_count ).floor() )
    
        // Life
        const lifeRandom = hash( cycleStep.add( uint( Math.random() * 0xffffff ) ) )
        life.assign( lifeRandom )
    
    } )().compute( this.particles_count );
    
    this.renderer.computeAsync( this.computeInit ).then( () => {
        this.initialCompute = true
    } )
    Initialization of particle position

    Step 3: Compute Position History

    For each frame, we compute the position history for each segment. The key aspect of the “computePositionStory” function is that new positions are recorded only from the head of the snake, and all positions are shifted one step forward using a queue algorithm.

    const computePositionStory = this.computePositionStory = Fn( () => {
        const positionStory = positionStoryBuffer.element( instanceIndex )
    
        const cycleStep = instanceIndex.mod( uint( this.story_snake ) )
        const lastPosition = positionBuffer.element( uint( float( instanceIndex.div( this.story_snake ) ).floor().mul( this.tails_count ) ) )
    
        If( cycleStep.equal( 0 ), () => { // Head
            positionStory.assign( lastPosition )
        } )
    
        positionStoryBuffer.element( instanceIndex.add( 1 ) ).assign( positionStoryBuffer.element( instanceIndex ) )
    
    } )().compute( this.full_story_length );

    Step 4: Update Particle Positions

    Next, we update the positions of all particles, taking into account the recorded history of their positions. First, we use simplex noise to generate the new positions of the particles, allowing our snakes to move smoothly through space. Each particle also has its own lifetime, during which it moves and eventually resets to its original position. The key part of this function is determining which particle is the head and which is the tail. For the head, we generate a new position based on simplex noise, while for the tail, we use positions from the saved history.

    const computeUpdate = this.computeUpdate = Fn( () => {
    
        const position = positionBuffer.element( instanceIndex )
        const positionInit = positionInitBuffer.element( instanceIndex )
    
        const life = lifeBuffer.element( instanceIndex );
    
        const _time = time.mul( 0.2 )
    
        const uFlowFieldInfluence = this.uniforms.uFlowFieldInfluence
        const uFlowFieldStrength = this.uniforms.uFlowFieldStrength
        const uFlowFieldFrequency = this.uniforms.uFlowFieldFrequency
    
        If( life.greaterThanEqual( 1 ), () => {
            life.assign( life.mod( 1 ) )
            position.assign( positionInit )
    
        } ).Else( () => {
            life.addAssign( deltaTime.mul( 0.2 ) )
        } )
    
        // Strength
        const strength = simplexNoise4d( vec4( position.mul( 0.2 ), _time.add( 1 ) ) ).toVar()
        const influence = uFlowFieldInfluence.sub( 0.5 ).mul( -2.0 ).toVar()
        strength.assign( smoothstep( influence, 1.0, strength ) )
    
        // Flow field
        const flowField = vec3(
            simplexNoise4d( vec4( position.mul( uFlowFieldFrequency ).add( 0 ), _time ) ),
            simplexNoise4d( vec4( position.mul( uFlowFieldFrequency ).add( 1.0 ), _time ) ),
            simplexNoise4d( vec4( position.mul( uFlowFieldFrequency ).add( 2.0 ), _time ) )
        ).normalize()
    
        const cycleStep = instanceIndex.mod( uint( this.tails_count ) )
    
        If( cycleStep.equal( 0 ), () => { // Head
            const newPos = position.add( flowField.mul( deltaTime ).mul( uFlowFieldStrength ) /* * strength */ )
            position.assign( newPos )
        } ).Else( () => { // Tail
            const prevTail = positionStoryBuffer.element( instanceIndex.mul( this.story_count ) )
            position.assign( prevTail )
        } )
    
    } )().compute( this.particles_count );

    To display the particle positions, we’ll create a simple function called “positionNode.” This function will not only output the positions but also apply a slight magnification effect to the head of the snake.

    particlesMaterial.positionNode = Fn( () => {
        const position = positionBuffer.element( instanceIndex );
    
        const cycleStep = instanceIndex.mod( uint( this.tails_count ) )
        const finalSize = this.uniforms.size.toVar()
    
        If( cycleStep.equal( 0 ), () => {
            finalSize.addAssign( 0.5 )
        } )
    
        return positionLocal.mul( finalSize ).add( position )
    } )()

    The final element will be to update the calculations on each frame.

    async update( deltaTime ) {
    
        // Compute update
        if( this.initialCompute) {
            await this.renderer.computeAsync( this.computePositionStory )
            await this.renderer.computeAsync( this.computeUpdate )
        }
    }

    Conclusion

    Now, you should be able to easily create position history buffers for other problem-solving tasks, and with TSL, this process becomes quick and efficient. I believe this project has potential for further development, such as transferring position data to model bones. This could enable the creation of beautiful, flying dragons or similar effects in 3D space. For this, a custom bone structure tailored to the project would be needed.



    Source link

  • Python – Monte Carlo Simulation – Useful code

    Python – Monte Carlo Simulation – Useful code


    Python can be used for various tasks. One of these is Monte Carlo simulation for future stock analysis. In the video below this is exactly what is happening. 🙂

    10K simulations in 30 buckets for KO look like that.

    Instead of explaining the video and its code (available also in GitHub), I will concentrate on why it is better to use log returns than simple returns in stock analysis. Which is actually part of the video as well. Below are the 3 main reasons:

    1. Time-Additivity

    Log returns sum over time, making multi-period calculations effortless. A 10% gain followed by a 10% loss doesn’t cancel out with simple returns—but it nearly does with logs.

    2. Symmetry Matters

    A +10% and -10% return aren’t true inverses in simple terms. Logs fix this, ensuring consistent math for gains and losses.

    3. Better for Modeling

    Log returns follow a near-normal distribution, crucial for statistical models like Monte Carlo simulations.

    When to Use Simple Returns?

    Code Highlights



    Source link

  • Davide’s Code and Architecture Notes

    Davide’s Code and Architecture Notes


    When designing a software system, we naturally focus more on the happy flow. But we should carefully plan to handle errors that fall into three categories: Validation, Transient, and Fatal.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    When designing a new software system, it’s easy to focus mainly on the happy flow and forget that you must also handle errors.

    You should carefully define and design how to handle errors: depending on the use case, error handling can have a huge impact on the architecture of your software system.

    In this article, we’ll explore the three main categories of errors that we must always remember to address; for each type of error, we will showcase how addressing it can impact the software architecture differently.

    An ideal system with only the happy path

    To use a realistic example, let’s design a simple system with a single module named MainApplication: this module reads data from an external API, manipulates the data, and stores the result on the DB.

    The system is called asynchronously, via a Message Queue, by an external service – that we are going to ignore.

    The happy flow is pretty much the following:

    1. An external system inserts some data into the Queue;
    2. MainApplication reads the data from the Queue;
    3. MainApplication calls an external API to retrieve some data;
    4. MainApplication stores some data on the DB;
    5. MainApplication sends a message on the queue with the operation result.

    Happy flow for MainApplication

    Now, the happy flow is simple. But we should have covered what to do in case of an error.

    Introducing the Error Management Trio

    In general, errors that need to be handled fall into three categories (that I decided to call “the Error Management Trio”): data validation, transient errors, and faults.

    Data Validation focuses on the data used across the system, particularly the data you don’t control.

    Transient Errors occur when the application’s overall status or its dependencies temporarily change to an invalid state.

    Faults are errors that take down the whole application, and you cannot recover immediately.

    The Trio does not take into account “errors” that are not properly errors: null values, queries that do not return any value, and so on. These, in my opinion, are all legitimate statuses that represent that lack of values but are not errors that have architectural relevance.

    The Error Management Trio schema

    Data Validation: the first defence against invalid status

    The Data Validation category focuses on ensuring that relevant data is in a valid status.

    In particular, it aims at ensuring that data coming from external sources (for example, from the Body in an incoming HTTP request or from the result of a query on the database) is both syntactically and logically valid.

    Suppose that the messages we receive from the queue are in the following format:

    {
      "Username": "mr. captain",
      "BookId": 154,
      "Operation": "Add"
    }
    

    We definitely need to perform some sort of validation on the message content.

    For example:

    • The Username property must not be empty;
    • The BookId property must be a positive number;
    • The Operation property must have one of the following values: Add, Remove, Refresh;

    How does it impact our design?

    We have several choices to deal with an invalid incoming message:

    1. ignore the whole message: if it doesn’t pass the validation, discard the message;
    2. send the message back to the caller, describing the type of error
    3. try to fix it locally: if we are able to recreate a valid message, we could try to fix it and process the incoming message;
    4. try to fix it in a separate service: you will need to create a distinct service that receives the invalid message and tries to fix it: if it manages to fix the message, it re-inserts it in the original queue; otherwise, it sends a message to the response queue to notify about the impossibility to recreate a valid message.

    As you can see, even for the simple input validation, the choices we make can have an impact on the structure of the architecture.

    Suppose that you choose option #4: you will need to implement a brand new service (let’s call it ValidationFixesManager), configure a new queue, and keep track of the attempts to fix the message.

    Example of Architecture with ValidationFixesManager component

    All of this only when considering the static validation. How would you validate your business rules? How would you ensure that, for instance, the Username is valid and the user is still active on the system?

    Maybe you discover that the data stored on the database in incomplete or stale. Then you have to work out a way to handle such type of data.

    For example, you can:

    • run a background job that ensures that all the data is always valid;
    • enrich the data from the DB with newer data only when it is actually needed;
    • fine-tune the database consistency level.

    We have just demonstrated a simple but important fact: data validation looks trivial, but depending on the needs of your system, it may impact how you design your system.

    Transient Errors: temporary errors that may randomly occur

    Even if the validation passes, temporary issues may prevent your operations from completing.

    In the previous example, there are some possible cases to consider:

    1. the external API is temporarily down, and you cannot retrieve the data you need;
    2. the return queue is full, and you cannot add response messages;
    3. the application is not able to connect to the DB due to network issues;

    These kinds of issues are due to a temporary status of the system or of one of its dependencies.

    Sure, you may add automatic retries: for instance, you can use Polly to automatically retry access the API. But what if it’s not enough?

    Again, depending on your application’s requirements and the overall structure you started designing, solving this problem may bring you to unexpected paths.

    Let’s say that the external API is returning a 500 HTTP error: this is a transient error, and it does not depend on the content of the request: the API is down, an you cannot to anything to solve it.

    What can we do if all the retries fail?

    If we can just accept the situation, we can return the error to the caller and move on with the next operation.

    But if we need to keep trying until the operation goes well, we have (at least) two choices:

    1. consume the message from the Queue, try calling the API, and, if it fails, re-insert the message on the queue (ideally, with some delay);
    2. peek the message from the queue and try calling the API. If it fails, the message stays on the queue (and you need a way to read it again). Otherwise, we consider the message completed and remove it from the queue.

    These are just two of the different solutions. But, as you can see, this choice will have, in the long run, a huge effect on the future of the application, both in terms of maintainability and performance.

    Below is how the structure changes if we decide to send the failed messages back in the queue with some delay.

    The MainApplication now sends messages back on the queue

    In both cases, we must remember that trying to call a service that is temporarily down is useless: maybe it’s time to use a Circuit Breaker?

    Fatal Errors: when everything goes wrong

    There is one type of error that is often neglected but that may deeply influence how your system behaves: fatal errors.

    Examples of fatal errors are:

    • the host has consumed all the CPU or RAM;
    • the file system is corrupted;
    • the connection to an external system is interrupted due to network misconfigurations.

    In short, fatal errors are errors you have no way to solve in the short run: they happen and stop everything you are doing.

    This kind of error cannot be directly managed via application code, but you need to rely on other techniques.

    For example, to make sure you won’t consume all the available RAM, you should plan for autoscaling of your resources. So you have to design the system with autoscaling in mind: this means, for example, that the system must be stateless and the application must run on infrastructure objects that can be configured to automatically manage resources (like Azure Functions, Kubernetes, and Azure App Services). Also: do you need horizontal or vertical scaling?

    And, talking about the integrity of the system, how do you ensure that operations that were ongoing when the fatal error occurred can be completed?

    One possible solution is to use a database table to keep track of the status of each operation, so that when the application restarts, it first completes pending operations, and then starts working on new operations.

    A database keeps track of the failed operations

    A practical approach to address the Error Management Trio

    There are too many errors to manage and too much effort to cover everything!

    How can we cover everything? Well, it’s impossible: for every action we take to prevent an error, a new one may occur.

    Let’s jump back to the example we saw for handling validation errors (using a new service that tries to fix the message). What if the ValidationFixesManager service is down or the message queue is unreachable? We tried to solve a problem, but we ended up with two more to be managed!

    Let me introduce a practical approach to help you decide what needs to be addressed.

    Step 1: list all the errors you can think of. Create a table to list all the possible errors that you expect they can happen.

    You can add a column to describe the category the error falls into, as well as a Probability and Impact on the system column with a value (in this example, Low, Medium and High) that represents the probability that this error occurs and the impact it has on the overall application.

    Problem Category Probability Impact on the system
    Invalid message from queue Data Validation Medium High
    Invalid user data on DB Data Validation Low Medium
    Missing user on DB Data Validation Low Low
    API not reachable Transient High High
    DB not reachable Transient Low High
    File system corrupted Fatal Low High
    CPU limit reached Fatal Medium High

    From here, you can pick the most urgent elements to be addressed.

    Step 2: evaluate alternatives. Every error can be addressed in several ways (ignoring the error IS a valid alternative!). Take some time to explore all the alternatives.

    Again, a table can be a good companion for this step. You can describe, for example:
    the effort required to solve the error (Low, Medium, High)
    the positive and negative consequences in terms (also) of quality attributes (aka: “-ilities”). Maybe a solution works fine for data integrity but has a negative impact on maintainability.

    Step 3: use ADRs to describe how (and why) you will handle that specific error.

    Take your time to thoroughly describe, using ADR documents, the problems you are trying to solve, the solutions taken into consideration, and the final choice.

    Having everything written down in a shared file is fundamental for ensuring that, in the future, the present choices and necessities are taken into account, before saying “meh, that’s garbage!”

    Further readings

    Unfortunately, I feel that error handling is one of the most overlooked topics when designing a system. This also means that there are not lots and lots of articles and resources that explore this topic.

    But, if you use queues, one of the components you should use to manage errors is the Dead Letter queue. Here’s a good article by Dorin Baba where he explains how to use Dead Letter queues to handle errors in asynchronous systems.

    🔗 Handling errors like a pro or nah? Let’s talk about Dead Letters | Dorin Baba

    This article first appeared on Code4IT 🐧

    In this article, we used a Queue to trigger the beginning of the operation. When using Azure services, we have two types of message queues: Queues and Topics. Do you know the difference? Hint: other vendors use the same names to represent different concepts.

    🔗 Azure Service Bus: Queues vs Topics | Code4IT

    Whichever the way you chose to solve manage an error, always remember to write down the reasons that guided you to use that specific solution. An incredibly helpful way is by using ADRs.

    🔗 Tracking decision with Architecture Decision Records (ADRs) | CodeIT

    Wrapping up

    This article highlights the importance of error management and the fact that even if we all want to avoid and prevent errors in our systems, we still have to take care of them and plan according to our needs.

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn or Twitter! 🤜🤛

    Happy coding!

    🐧





    Source link

  • Motion Highlights #5 | Codrops

    Motion Highlights #5 | Codrops


    The

    New

    Collective

    🎨✨💻 Stay ahead of the curve with handpicked, high-quality frontend development and design news, picked freshly every single day. No fluff, no filler—just the most relevant insights, inspiring reads, and updates to keep you in the know.

    Prefer a weekly digest in your inbox? No problem, we got you covered. Just subscribe here.



    Source link

  • Python – Reading Financial Data From Internet – Useful code

    Python – Reading Financial Data From Internet – Useful code


    Reading financial data from the internet is sometimes challenging. In this short article with two python snippets, I will show how to read it from Wikipedia and from and from API, delivering in JSON format:

    This is how the financial json data from the api looks like.

    Reading the data from the API is actually not tough, if you have experience reading JSON, with nested lists. If not, simply try with trial and error and eventually you will succeed:

    With the reading from wikipedia, it is actually even easier – the site works flawlessly with pandas, and if you count the tables correctly, you would get what you want:

    You might want to combine both sources, just in case:

    The YouTube video for this article is here:
    https://www.youtube.com/watch?v=Uj95BgimHa8
    The GitHub code is there – GitHub

    Enjoy it! 🙂



    Source link

  • GPT Function Calling: 5 Underrated Use Cases | by Max Brodeur-Urbas


    OpenAI’s backend converting messy unstructured data to structured data via functions

    OpenAI’s “Function Calling” might be the most groundbreaking yet under appreciated feature released by any software company… ever.

    Functions allow you to turn unstructured data into structured data. This might not sound all that groundbreaking but when you consider that 90% of data processing and data entry jobs worldwide exist for this exact reason, it’s quite a revolutionary feature that went somewhat unnoticed.

    Have you ever found yourself begging GPT (3.5 or 4) to spit out the answer you want and absolutely nothing else? No “Sure, here is your…” or any other useless fluff surrounding the core answer. GPT Functions are the solution you’ve been looking for.

    How are Functions meant to work?

    OpenAI’s docs on function calling are extremely limited. You’ll find yourself digging through their developer forum for examples of how to use them. I dug around the forum for you and have many example coming up.

    Here’s one of the only examples you’ll be able to find in their docs:

    functions = [
    {
    "name": "get_current_weather",
    "description": "Get the current weather in a given location",
    "parameters": {
    "type": "object",
    "properties": {
    "location": {
    "type": "string",
    "description": "The city and state, e.g. San Francisco, CA",
    },
    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
    },
    "required": ["location"],
    },
    }
    ]

    A function definition is a rigid JSON format that defines a function name, description and parameters. In this case, the function is meant to get the current weather. Obviously GPT isn’t able to call this actual API (since it doesn’t exist) but using this structured response you’d be able to connect the real API hypothetically.

    At a high level however, functions provide two layers of inference:

    Picking the function itself:

    You may notice that functions are passed into the OpenAI API call as an array. The reason you provide a name and description to each function are so GPT can decide which to use based on a given prompt. Providing multiple functions in your API call is like giving GPT a Swiss army knife and asking it to cut a piece of wood in half. It knows that even though it has a pair of pliers, scissors and a knife, it should use the saw!

    Function definitions contribute towards your token count. Passing in hundreds of functions would not only take up the majority of your token limit but also result in a drop in response quality. I often don’t even use this feature and only pass in 1 function that I force it to use. It is very nice to have in certain use cases however.

    Picking the parameter values based on a prompt:

    This is the real magic in my opinion. GPT being able to choose the tool in it’s tool kit is amazing and definitely the focus of their feature announcement but I think this applies to more use cases.

    You can imagine a function like handing GPT a form to fill out. It uses its reasoning, the context of the situation and field names/descriptions to decide how it will fill out each field. Designing the form and the additional information you pass in is where you can get creative.

    GPT filling out your custom form (function parameters)

    One of the most common things I use functions for to extract specific values from a large chunk of text. The sender’s address from an email, a founders name from a blog post, a phone number from a landing page.

    I like to imagine I’m searching for a needle in a haystack except the LLM burns the haystack, leaving nothing but the needle(s).

    GPT Data Extraction Personified.

    Use case: Processing thousands of contest submissions

    I built an automation that iterated over thousands of contest submissions. Before storing these in a Google sheet I wanted to extract the email associated with the submission. Heres the function call I used for extracting their email.

    {
    "name":"update_email",
    "description":"Updates email based on the content of their submission.",
    "parameters":{
    "type":"object",
    "properties":{
    "email":{
    "type":"string",
    "description":"The email provided in the submission"
    }
    },
    "required":[
    "email"
    ]
    }
    }

    Assigning unstructured data a score based on dynamic, natural language criteria is a wonderful use case for functions. You could score comments during sentiment analysis, essays based on a custom grading rubric, a loan application for risk based on key factors. A recent use case I applied scoring to was scoring of sales leads from 0–100 based on their viability.

    Use Case: Scoring Sales leads

    We had hundreds of prospective leads in a single google sheet a few months ago that we wanted to tackle from most to least important. Each lead contained info like company size, contact name, position, industry etc.

    Using the following function we scored each lead from 0–100 based on our needs and then sorted them from best to worst.

    {
    "name":"update_sales_lead_value_score",
    "description":"Updates the score of a sales lead and provides a justification",
    "parameters":{
    "type":"object",
    "properties":{
    "sales_lead_value_score":{
    "type":"number",
    "description":"An integer value ranging from 0 to 100 that represents the quality of a sales lead based on these criteria. 100 is a perfect lead, 0 is terrible. Ideal Lead Criteria:\n- Medium sized companies (300-500 employees is the best range)\n- Companies in primary resource heavy industries are best, ex. manufacturing, agriculture, etc. (this is the most important criteria)\n- The higher up the contact position, the better. VP or Executive level is preferred."
    },
    "score_justification":{
    "type":"string",
    "description":"A clear and conscise justification for the score provided based on the custom criteria"
    }
    }
    },
    "required":[
    "sales_lead_value_score",
    "score_justification"
    ]
    }

    Define custom buckets and have GPT thoughtfully consider each piece of data you give it and place it in the correct bucket. This can be used for labelling tasks like selecting the category of youtube videos or for discrete scoring tasks like assigning letter grades to homework assignments.

    Use Case: Labelling news articles.

    A very common first step in data processing workflows is separating incoming data into different streams. A recent automation I built did exactly this with news articles scraped from the web. I wanted to sort them based on the topic of the article and include a justification for the decision once again. Here’s the function I used:

    {
    "name":"categorize",
    "description":"Categorize the input data into user defined buckets.",
    "parameters":{
    "type":"object",
    "properties":{
    "category":{
    "type":"string",
    "enum":[
    "US Politics",
    "Pandemic",
    "Economy",
    "Pop culture",
    "Other"
    ],
    "description":"US Politics: Related to US politics or US politicians, Pandemic: Related to the Coronavirus Pandemix, Economy: Related to the economy of a specific country or the world. , Pop culture: Related to pop culture, celebrity media or entertainment., Other: Doesn't fit in any of the defined categories. "
    },
    "justification":{
    "type":"string",
    "description":"A short justification explaining why the input data was categorized into the selected category."
    }
    },
    "required":[
    "category",
    "justification"
    ]
    }
    }

    Often times when processing data, I give GPT many possible options and want it to select the best one based on my needs. I only want the value it selected, no surrounding fluff or additional thoughts. Functions are perfect for this.

    Use Case: Finding the “most interesting AI news story” from hacker news

    I wrote another medium article here about how I automated my entire Twitter account with GPT. Part of that process involves selecting the most relevant posts from the front pages of hacker news. This post selection step leverages functions!

    To summarize the functions portion of the use case, we would scrape the first n pages of hacker news and ask GPT to select the post most relevant to “AI news or tech news”. GPT would return only the headline and the link selected via functions so that I could go on to scrape that website and generate a tweet from it.

    I would pass in the user defined query as part of the message and use the following function definition:

    {
    "name":"find_best_post",
    "description":"Determine the best post that most closely reflects the query.",
    "parameters":{
    "type":"object",
    "properties":{
    "best_post_title":{
    "type":"string",
    "description":"The title of the post that most closely reflects the query, stated exactly as it appears in the list of titles."
    }
    },
    "required":[
    "best_post_title"
    ]
    }
    }

    Filtering is a subset of categorization where you categorize items as either true or false based on a natural language condition. A condition like “is Spanish” will be able to filter out all Spanish comments, articles etc. using a simple function and conditional statement immediately after.

    Use Case: Filtering contest submission

    The same automation that I mentioned in the “Data Extraction” section used ai-powered-filtering to weed out contest submissions that didn’t meet the deal-breaking criteria. Things like “must use typescript” were absolutely mandatory for the coding contest at hand. We used functions to filter out submissions and trim down the total set being processed by 90%. Here is the function definition we used.

    {
    "name":"apply_condition",
    "description":"Used to decide whether the input meets the user provided condition.",
    "parameters":{
    "type":"object",
    "properties":{
    "decision":{
    "type":"string",
    "enum":[
    "True",
    "False"
    ],
    "description":"True if the input meets this condition 'Does submission meet the ALL these requirements (uses typescript, uses tailwindcss, functional demo)', False otherwise."
    }
    },
    "required":[
    "decision"
    ]
    }
    }

    If you’re curious why I love functions so much or what I’ve built with them you should check out AgentHub!

    AgentHub is the Y Combinator-backed startup I co-founded that let’s you automate any repetitive or complex workflow with AI via a simple drag and drop no-code platform.

    “Imagine Zapier but AI-first and on crack.” — Me

    Automations are built with individual nodes called “Operators” that are linked together to create power AI pipelines. We have a catalogue of AI powered operators that leverage functions under the hood.

    Our current AI-powered operators that use functions!

    Check out these templates to see examples of function use-cases on AgentHub: Scoring, Categorization, Option-Selection,

    If you want to start building AgentHub is live and ready to use! We’re very active in our discord community and are happy to help you build your automations if needed.

    Feel free to follow the official AgentHub twitter for updates and myself for AI-related content.





    Source link

  • Easy logging management with Seq and ILogger in ASP.NET | Code4IT

    Easy logging management with Seq and ILogger in ASP.NET | Code4IT


    Seq is one of the best Log Sinks out there : it’s easy to install and configure, and can be added to an ASP.NET application with just a line of code.

    Table of Contents

    Just a second! 🫷
    If you are here, it means that you are a software developer.
    So, you know that storage, networking, and domain management have a cost .

    If you want to support this blog, please ensure that you have disabled the adblocker for this site.
    I configured Google AdSense to show as few ADS as possible – I don’t want to bother you with lots of ads, but I still need to add some to pay for the resources for my site.

    Thank you for your understanding.
    Davide

    Logging is one of the most essential parts of any application.

    Wouldn’t it be great if we could scaffold and use a logging platform with just a few lines of code?

    In this article, we are going to learn how to install and use Seq as a destination for our logs, and how to make an ASP.NET 8 API application send its logs to Seq by using the native logging implementation.

    Seq: a sink and dashboard to manage your logs

    In the context of logging management, a “sink” is a receiver of the logs generated by one or many applications; it can be a cloud-based system, but it’s not mandatory: even a file on your local file system can be considered a sink.

    Seq is a Sink, and works by exposing a server that stores logs and events generated by an application. Clearly, other than just storing the logs, Seq allows you to view them, access their details, perform queries over the collection of logs, and much more.

    It’s free to use for individual usage, and comes with several pricing plans, depending on the usage and the size of the team.

    Let’s start small and install the free version.

    We have two options:

    1. Download it locally, using an installer (here’s the download page);
    2. Use Docker: pull the datalust/seq image locally and run the container on your Docker engine.

    Both ways will give you the same result.

    However, if you already have experience with Docker, I suggest you use the second approach.

    Once you have Docker installed and running locally, open a terminal.

    First, you have to pull the Seq image locally (I know, it’s not mandatory, but I prefer doing it in a separate step):

    Then, when you have it downloaded, you can start a new instance of Seq locally, exposing the UI on a specific port.

    docker run --name seq -d --restart unless-stopped -e ACCEPT_EULA=Y -p 5341:80 datalust/seq:latest
    

    Let’s break down the previous command:

    • docker run: This command is used to create and start a new Docker container.
    • --name seq: This option assigns the name seq to the container. Naming containers can make them easier to manage.
    • -d: This flag runs the container in detached mode, meaning it runs in the background.
    • --restart unless-stopped: This option ensures that the container will always restart unless it is explicitly stopped. This is useful for ensuring that the container remains running even after a reboot or if it crashes.
    • -e ACCEPT_EULA=Y: This sets an environment variable inside the container. In this case, it sets ACCEPT_EULA to Y, which likely indicates that you accept the End User License Agreement (EULA) for the software running in the container.
    • -p 5341:80: This maps port 5341 on your host machine to port 80 in the container. This allows you to access the service running on port 80 inside the container via port 5341 on your host.
    • datalust/seq:latest: This specifies the Docker image to use for the container. datalust/seq is the image name, and latest is the tag, indicating that you want to use the latest version of this image.

    So, this command runs a container named seq in the background, ensures it restarts unless stopped, sets an environment variable to accept the EULA, maps a host port to a container port, and uses the latest version of the datalust/seq image.

    It’s important to pay attention to the used port: by default, Seq uses port 5341 to interact with the UI and the API. If you prefer to use another port, feel free to do that – just remember that you’ll need some additional configuration.

    Now that Seq is installed on your machine, you can access its UI. Guess what? It’s on localhost:5341!

    Seq brand new instance

    However, Seq is “just” a container for our logs – but we have to produce them.

    A sample ASP.NET API project

    I’ve created a simple API project that exposes CRUD operations for a data model stored in memory (we don’t really care about the details).

    [ApiController]
    [Route("[controller]")]
    public class BooksController : ControllerBase
    {
        public BooksController()
        {
    
        }
    
        [HttpGet("{id}")]
        public ActionResult<Book> GetBook([FromRoute] int id)
        {
    
            Book? book = booksCatalogue.SingleOrDefault(x => x.Id == id);
            return book switch
            {
                null => NotFound(),
                _ => Ok(book)
            };
        }
    }
    

    As you can see, the details here are not important.

    Even the Main method is the default one:

    var builder = WebApplication.CreateBuilder(args);
    
    builder.Services.AddControllers();
    
    builder.Services.AddEndpointsApiExplorer();
    builder.Services.AddSwaggerGen();
    
    var app = builder.Build();
    
    if (app.Environment.IsDevelopment())
    {
        app.UseSwagger();
        app.UseSwaggerUI();
    }
    
    app.UseHttpsRedirection();
    
    app.MapControllers();
    
    app.Run();
    

    We have the Controllers, we have Swagger… well, nothing fancy.

    Let’s mix it all together.

    How to integrate Seq with an ASP.NET application

    If you want to use Seq in an ASP.NET application (may it be an API application or whatever else), you have to add it to the startup pipeline.

    First, you have to install the proper NuGet package: Seq.Extensions.Logging.

    The Seq.Extensions.Logging NuGet package

    Then, you have to add it to your Services, calling the AddSeq() method:

    var builder = WebApplication.CreateBuilder(args);
    
    builder.Services.AddControllers();
    
    builder.Services.AddEndpointsApiExplorer();
    builder.Services.AddSwaggerGen();
    
    + builder.Services.AddLogging(lb => lb.AddSeq());
    
    var app = builder.Build();
    

    Now, Seq is ready to intercept whatever kind of log arrives at the specified port (remember, in our case, we are using the default one: 5341).

    We can try it out by adding an ILogger to the BooksController constructor:

    private readonly ILogger<BooksController> _logger;
    
    public BooksController(ILogger<BooksController> logger)
    {
        _logger = logger;
    }
    

    So that we can use the _logger instance to create logs as we want, using the necessary Log Level:

    [HttpGet("{id}")]
    public ActionResult<Book> GetBook([FromRoute] int id)
    {
        _logger.LogInformation("I am Information");
        _logger.LogWarning("I am Warning");
        _logger.LogError("I am Error");
        _logger.LogCritical("I am Critical");
    
        Book? book = booksCatalogue.SingleOrDefault(x => x.Id == id);
        return book switch
        {
            null => NotFound(),
            _ => Ok(book)
        };
    }
    

    Log messages on Seq

    Using Structured Logging with ILogger and Seq

    One of the best things about Seq is that it automatically handles Structured Logging.

    [HttpGet("{id}")]
    public ActionResult<Book> GetBook([FromRoute] int id)
    {
        _logger.LogInformation("Looking if in my collection with {TotalBooksCount} books there is one with ID {SearchedId}"
     , booksCatalogue.Count, id);
    
        Book? book = booksCatalogue.SingleOrDefault(x => x.Id == id);
        return book switch
        {
            null => NotFound(),
            _ => Ok(book)
        };
    }
    

    Have a look at this line:

    _logger.LogInformation("Looking if in my collection with {TotalBooksCount} books there is one with ID {SearchedId}"
     , booksCatalogue.Count, id);
    

    This line generates a string message, replaces all the placeholders, and, on top of that, creates two properties, SearchedId and TotalBooksCount; you can now define queries using these values.

    Structured Logs in Seq allow you to view additional logging properties

    Further readings

    I have to admit it: logging management is one of my favourite topics.

    I’ve already written a sort of introduction to Seq in the past, but at that time, I did not use the native ILogger, but Serilog, a well-known logging library that added some more functionalities on top of the native logger.

    🔗 Logging with Serilog and Seq | Code4IT

    This article first appeared on Code4IT 🐧

    In particular, Serilog can be useful for propagating Correlation IDs across multiple services so that you can fetch all the logs generated by a specific operation, even though they belong to separate applications.

    🔗 How to log Correlation IDs in .NET APIs with Serilog

    Feel free to search through my blog all the articles related to logging – I’m sure you will find interesting stuff!

    Wrapping up

    I think Seq is the best tool for local development: it’s easy to download and install, supports structured logging, and can be easily added to an ASP.NET application with just a line of code.

    I usually add it to my private projects, especially when the operations I run are complex enough to require some well-structured log.

    Given how it’s easy to install, sometimes I use it for my work projects too: when I have to fix a bug, but I don’t want to use the centralized logging platform (since it’s quite complex to use), I add Seq as a destination sink, run the application, and analyze the logs in my local machine. Then, of course, I remove its reference, as I want it to be just a discardable piece of configuration.

    I hope you enjoyed this article! Let’s keep in touch on LinkedIn, Twitter or BlueSky! 🤜🤛

    Happy coding!

    🐧





    Source link

  • Integrating Psychology into Software Development | by Ulas Can Cengiz


    14 min read

    Nov 10, 2023

    Photo by Bret Kavanaugh on Unsplash

    Imagine sitting down at your desk to untangle a particularly complex piece of software code. Your eyes scan lines packed with logical operations and function calls. Somewhere in this intricate weave, a bug lurks, derailing the application’s performance. This scenario, familiar to many developers, isn’t just a test of technical skill; it’s a psychological challenge. The frustration and cognitive fatigue that often accompany such tasks can cloud judgment and prolong resolution. It’s in moments like these that the intersection of psychology and software development comes into sharp focus.

    Cognitive load theory, originally applied to educational psychology, has profound implications for managing complexity in software projects. It posits that our working memory has a limited capacity for processing new information. In the context of software development, this translates to the need for clean, readable code and well-architected systems that minimize the cognitive load on developers. By understanding and applying this theory, we can create development environments that reduce unnecessary complexity and allow developers to allocate their cognitive resources…



    Source link

  • Weekend Sale! 🎁

    Weekend Sale! 🎁


    At Browserling and Online Tools we love sales.

    We just created a new automated Weekend Sale.

    Now each weekend, we show a 50% discount offer to all users who visit our site.

    🔥 onlinetools.com/pricing

    🔥 browserling.com/#pricing

    Buy a subscription now and see you next time!



    Source link

  • Python – Simple Stock Analysis with yfinance – Useful code

    Python – Simple Stock Analysis with yfinance – Useful code


    Sometimes, the graphs of stocks are useful. Sometimes these are not. In general, do your own research, none of this is financial advice.

    And while doing that, if you want to analyze stocks with just a few lines of python, this article might help? This simple yet powerful script helps you spot potential buy and sell opportunities for Apple (AAPL) using two classic technical indicators: moving averages and RSI.

    Understanding the Strategy

    1. SMA Crossover: The Trend Following Signal

    The script first calculates two Simple Moving Averages (SMA):

    The crossover strategy is simple:

    This works because moving averages smooth out price noise, helping identify the overall trend direction.

    2. RSI: The Overbought/Oversold Indicator

    The Relative Strength Index (RSI) measures whether a stock is overbought or oversold:

    By combining SMA crossovers (trend confirmation) and RSI extremes (timing), we get stronger signals.

    This plot is generated with less than 40 lines of python code

    The code looks like that:

    The code above, but in way more details is explained in the YT video below:

    https://www.youtube.com/watch?v=m0ayASmrZmE

    And it is available in GitHub as well.



    Source link