Tag: Streaming
-
Data streaming technologies overview
Posted on August 29, 2024, Level beginner Resource Length medium
The ability to process high volumes of data (big data) in real-time has become crucial for many organizations, and this is where data streaming technologies come into the picture. These technologies allow large amounts of data to be processed in real-time or near real-time as it is generated, enabling businesses to gain immediate insights and make time-sensitive data-driven decisions. By Darryn Campbell.
Tags streaming big-data cloud event-driven software
-
The benefits of event-driven architecture
Posted on March 7, 2024, Level intermediate Resource Length medium
Event-driven architecture (EDA) is a software design pattern used in microservices or other decoupled services or apps. Each service in the ecosystem can asynchronously publish and subscribe to events via an event broker. By Darryn Campbell.
Tags web-development app-development queues streaming event-driven
-
Real-time data architecture patterns
Posted on December 27, 2023, Level intermediate Resource Length long
The rapid proliferation and increased volume of data across industries has magnified the need for organizations to have a solid strategy in place for processing and managing real-time data. Improving overall data capabilities enables teams to operate more efficiently, and emerging technologies have even created a smoother pathway for bringing real-time data closer to business users, which plays a critical role in effective decision-making. By Miguel Garcia.
Tags streaming web-development app-development devops software-architecture
-
Streams in Scala - introductory guide
Posted on September 19, 2023, Level intermediate Resource Length medium
Streams in Scala provide a lazy evaluation mechanism where elements are computed on-demand rather than being eagerly evaluated and stored in memory. This allows for efficient memory utilization, especially when dealing with large datasets or potentially infinite sequences of data. By Aniefiok Akpan.
Tags akka scala programming learning streaming queues
-
Achieving unbrickable remote firmware updates on MCUs with a Microvisor architecture
Posted on May 16, 2023, Level intermediate Resource Length medium
For many years, microcontrollers have been a staple in various products, continuously revolutionizing their feature sets, reliability, and performance. Moore's Law has brought 16- and 32-bit processing to even the smallest and most affordable consumer products. The presence of larger memory and CPU power has allowed the use of real-time operating systems (RTOS) where previously developers had to rely on "bare metal" coding. However, as products have evolved to become connected devices in the context of IoT, it has revealed fundamental shortcomings in the traditional methods of software development for microcontrollers. By Jonathan Williams.
Tags big-data iot streaming robotics cloud infosec
-
A serverless query engine from spare parts
Posted on May 7, 2023, Level intermediate Resource Length medium
An open-source implementation of a Data Lake with DuckDB and AWS Lambdas. In this post we will show how to build a simple end-to-end application in the cloud on a serverless infrastructure. The purpose is simple: we want to show that we can develop directly against the cloud while minimizing the cognitive overhead of designing and building infrastructure. By Ciro Greco.
Tags data-science streaming apis database serverless open-source
-
Comparing Avro vs Protobuf for data serialization
Posted on April 18, 2023, Level beginner Resource Length short
Data serialization is a crucial aspect of modern distributed systems because it enables the efficient communication and storage of structured data. In this article, we will discuss two popular serialization formats: Avro and Protocol Buffers, Protobuf for short, and compare their strengths and weaknesses to help you make an informed decision about which one to use in your projects. By Daniel Selans.
Tags json queues messaging app-development streaming apache
-
Using Vulcan codecs with Kafka Java APIs
Posted on April 17, 2023, Level intermediate Resource Length medium
For those that aren't familiar, Vulcan is a functional Avro encoding library that uses the official Apache Avro library under the hood. The difference between this and the official Avro build plugins approach is that the types are defined in plain Scala. Then the Avro schema is generated from those instead of defining the Avro schema and getting code generated at compile time that adheres to that schema. By César Enrique.
Tags apache java messaging app-development streaming scala
-
Real-time data linkage via Linked Data Event Streams
Posted on April 12, 2023, Level intermediate Resource Length long
Real-time interchanging data across domains and applications is challenging; data format incompatibility, latency and outdated data sets, quality issues, and lack of metadata and context. A Linked Data Event Stream (LDES) is a new data publishing approach which allows you to publish any dataset as a collection of immutable objects. The focus of an LDES is to allow clients to replicate the history of a dataset and efficiently synchronize with its latest changes. By towardsai.net.
Tags data-science streaming performance how-to big-data apache
-
Kubernetes events: In-depth guide and examples
Posted on January 4, 2023, Level intermediate Resource Length long
Kubernetes events help you understand how Kubernetes resource decisions are made and they can be helpful for debugging. Learn more about k8s events in this in-depth guide. By Tyler Charboneau.
Tags apis devops app-development open-source streaming event-driven
-
Live streaming commerce: A playbook
Posted on October 14, 2022, Level beginner Resource Length long
To better assist customers with product discovery, retailers are discovering and leveraging innovative ways like live streaming. Live streaming commerce is an interactive social commerce tool that combines video streaming and TV entertainment formats like talk shows and chat. By Shantala Raman.
Tags miscellaneous streaming cio how-to cloud
-
How to visualize your Apache Kafka data the easy way with stream lineage
Posted on July 24, 2022, Level intermediate Resource Length long
Understanding how data flows and is transformed across the different layers of an organization's application and data stack is one of the most challenging governance problems companies are facing today. Who is producing data? By David Araujo and Julia Peng.
Tags app-development messaging apis apache streaming