[{"data":1,"prerenderedAt":5884},["ShallowReactive",2],{"/framework/blog/signal-processing-app":3,"navigation-framework":182,"/framework/blog/signal-processing-blog":1216,"/developers/templates/etl/gaussian_filtering_pythonrelated":1315,"surr-/framework/blog/signal-processing":5881},{"id":4,"title":5,"author":6,"body":14,"description":24,"extension":161,"meta":162,"navigation":176,"path":177,"seo":178,"sitemap":179,"stem":180,"__hash__":181},"content/framework/blog/912.signal-processing.md","Signal Processing in Real-time: Bridging the Gap Between Ideal Sampling and Real-World Data Streams",{"id":7,"url":8,"name":9,"description":10,"img":11,"provider":12,"linkedin":13},"olivier","olivier-ruas","Olivier Ruas","Algorithm and Data Processing Magician","/assets/pictures/image_ruas_pathway.jpg","s3","https://www.linkedin.com/in/oruas/",{"type":15,"value":16,"toc":151},"minimark",[17,21,25,34,47,52,55,59,62,85,89,92,96,107,112,115,118,121,125,128,131,135,138],[18,19,5],"h1",{"id":20},"signal-processing-in-real-time-bridging-the-gap-between-ideal-sampling-and-real-world-data-streams",[22,23,24],"p",{},"Understand the complexities of performing signal processing in realtime. Learn how to build robust signal processing systems that can handle irregular data streams.",[22,26,27,28,33],{},"Signal processing is crucial for extracting valuable insights from raw data. It is used across a range of data stream processing applications, such as sensor data, ",[29,30,32],"a",{"href":31},"/blog/iot-data-analytics","IoT data analytics"," and financial time series. The typical signal processing setup is built on the assumption that data is regularly sampled at precise intervals. However, realtime data processing does not meet this assumption: signals are collected from disparate, decoupled data sources leading to irregular data sampling and out-of-order data points. In this less predictable environment, traditional signal processing algorithms designed for regularly sampled data may not suffice. You will need a realtime signal processing system that is able to accommodate out-of-order and irregularly sampled data points.",[22,35,36,37,41,42,46],{},"This article will explore signal processing fundamentals, how they differ in context from real-world data stream processing, and how Pathway tackles this challenge to unlock effective digital signal processing in asynchronous and less structured scenarios. It is the foundation for our signal-processing tutorials on ",[29,38,40],{"href":39},"/developers/templates/etl/gaussian_filtering_python","applying a Gaussian filter on out-of-order data points"," and ",[29,43,45],{"href":44},"/developers/templates/etl/upsampling","how to combine data streams with upsampling",".",[48,49,51],"h2",{"id":50},"what-is-signal-processing","What is Signal Processing?",[22,53,54],{},"Signal processing is a broad field that encompasses a wide range of techniques used to manipulate signals, typically in the time or frequency domains. Usually, signals are considered time-varying data, where the time axis represents the progression of discrete samples.",[48,56,58],{"id":57},"common-operations-in-signal-processing","Common Operations in Signal Processing",[22,60,61],{},"There are a number of common operations in signal processing:",[63,64,65,73,79],"ul",{},[66,67,68,72],"li",{},[69,70,71],"strong",{},"Smoothing and Filtering",": Smoothing and filtering in data processing are used to reduce noise, remove unwanted variations, and reveal underlying patterns in the data for improved analysis and interpretation. Smoothing techniques reduce noise and irregularities in a signal, making it easier to analyze trends and patterns.",[66,74,75,78],{},[69,76,77],{},"Feature Extraction",": Feature extraction aims to identify and extract relevant information or features from a signal, which can be used for further analysis or classification.",[66,80,81,84],{},[69,82,83],{},"Transformations",": Signal transformations, such as Fourier transforms, convert signals from the time domain to the frequency domain and vice versa, revealing different characteristics.",[48,86,88],{"id":87},"what-is-digital-signal-processing","What is Digital Signal Processing?",[22,90,91],{},"Digital signal processing is a subfield of signal processing working with digital signals (as opposed to analog signal processing, which deals with analog signals).",[48,93,95],{"id":94},"digital-signal-processing-vs-data-stream-processing","Digital Signal Processing vs. Data Stream Processing",[22,97,98,99,103,104,106],{},"In digital signal processing, signals are typically represented by discrete samples taken at regular intervals. The typical DSP setup is highly concentrated and coordinated: a dedicated process meticulously samples, processes, and sends data using a global clock for synchronization. Everything follows a regular and precise timing pattern: think of an audio processor or broadband modem in the low layers. These systems operate under the common assumption that the sampling rates are perfect and consistent: with a sample every second, the ",[100,101,102],"code",{},"i","th data point has been sampled at second ",[100,105,102],{}," precisely. This synchrony and absence of failure makes it possible to wait for all the points to be gathered before any computation is performed.",[22,108,109],{},[69,110,111],{},"In many real-world data processing applications, this idealized scenario is unrealistic.",[22,113,114],{},"First, the sensors themselves are not perfect. Not only the captors may be faulty, including the internal clock slowly drifting apart from the \"real\" time, but data may be sampled at irregular intervals, leading to variations in the time between samples. The resulting data is noisy and contains gaps.",[22,116,117],{},"The data collection adds another layer of imperfection. Data packets may be lost or delayed, which means we must deal with incomplete and out-of-order data streams.",[22,119,120],{},"In complex systems, which integrate data samples from several sources data misalignment is as serious as signal noise! This challenge requires a different approach to data processing compared to idealized signal processing, where the focus is on precise time alignment.",[48,122,124],{"id":123},"signal-processing-with-pathway","Signal Processing with Pathway",[22,126,127],{},"Because they assume a perfect sampling, standard DSP systems do not support the addition of late data points. Out-of-order and irregularly sampled points will be buffered until all points are received, delaying the response until then; there is no way to push a late point through the system.",[22,129,130],{},"Pathway is built to work with real-world and realtime sensor data. Its incremental engine enables you to build reactive data products that will automatically update themselves whenever the out-of-order data points arrive. As a developer, you only have to describe the logic once and Pathway will handle the data updates under the hood. In practice, this means that you can manipulate your streaming data as if it were static, i.e. as if all the data is already gathered. You can write your processing logic without thinking about the order in which data will arrive. Pathway will handle any irregular or out-of-order data by updating its results automatically whenever these points arrive.",[48,132,134],{"id":133},"building-realtime-signal-processing-pipelines-with-pathway","Building Realtime Signal Processing Pipelines with Pathway",[22,136,137],{},"Now that you understand the complexities of working with realtime signal processing, you may want to get some hands-on experience by working through one of our signal processing tutorials:",[139,140,141,146],"ol",{},[66,142,143],{},[29,144,145],{"href":39},"Applying a Gaussian filter on out-of-order data",[66,147,148],{},[29,149,150],{"href":44},"Upsampling your Signal Processing Data: transforming multiple data streams.",{"title":152,"searchDepth":153,"depth":153,"links":154},"",2,[155,156,157,158,159,160],{"id":50,"depth":153,"text":51},{"id":57,"depth":153,"text":58},{"id":87,"depth":153,"text":88},{"id":94,"depth":153,"text":95},{"id":123,"depth":153,"text":124},{"id":133,"depth":153,"text":134},"md",{"keywords":163,"layout":170,"thumbnail":171,"tags":173,"date":174,"related":175,"hidden":176},[164,165,166,167,168,169],"time series","signal processing","resampling","filter","gaussian","upsampling","blog",{"src":172},"/assets/content/blog/iot-data-analytics/manul-iot.png",[170],"2023-10-16",[39,44],true,"/framework/blog/signal-processing",{"title":5,"description":24},{"loc":177},"framework/blog/912.signal-processing","hyElI5ztHo9cIyk85VIn2DJ_bQDjG3VM_Dwy3qSlDHo",[183],{"title":184,"path":185,"stem":186,"children":187,"page":196},"Framework","/framework","framework",[188],{"title":189,"path":190,"stem":191,"children":192,"meta":1215},"Blog","/framework/blog","framework/blog/1.index",[193,197,207,216,242,255,275,292,304,314,334,343,352,365,375,390,405,414,423,438,447,458,468,477,488,499,510,519,528,537,546,558,567,576,585,604,617,631,643,652,661,670,679,690,702,711,720,729,735,749,758,767,776,784,793,807,817,826,835,844,853,862,871,880,889,898,907,917,926,935,943,952,961,970,979,988,997,1007,1016,1025,1034,1043,1052,1061,1072,1080,1089,1097,1106,1123,1132,1147,1156,1165,1180,1189,1198,1206],{"title":189,"path":190,"stem":191,"meta":194},{"layout":195,"aside":196,"toc":196,"single":176},"default",false,{"title":198,"path":199,"stem":200,"meta":201},"Pathway is now in Open Beta","/framework/blog/pathway-open-beta-announced","framework/blog/1.pathway-open-beta-announced",{"layout":170,"thumbnail":202,"tags":204,"date":206,"hidden":176},{"src":203,"provider":12},"/assets/blog/thumbnails/pathway-thumbnail.gif",[170,205],"open beta","2022-12-05",{"title":208,"path":209,"stem":210,"meta":211},"Pathway named 2021 i-Lab Laureate","/framework/blog/ilab2021","framework/blog/1000.Ilab2021",{"layout":170,"aside":196,"thumbnail":212,"tags":214,"date":215,"hidden":176},{"src":213,"provider":12,"contain":176},"/assets/blog/thumbnails/I-Lab-LOGO.jpg",[170],"2021-07-09",{"title":217,"path":218,"stem":219,"meta":220},"Multimodal RAG with Gemini","/framework/blog/gemini-rag","framework/blog/1001.gemini-rag",{"aside":176,"layout":170,"thumbnail":221,"date":224,"tags":225,"keywords":228,"notebook_export_path":240,"run_template":241,"hidden":176},{"src":222,"fit":223},"/assets/content/showcases/gemini_rag/Blog_Banner.png","contain","2024-08-06",[226,227],"showcase","llm",[229,230,231,232,233,234,235,236,237,238,239],"LLM","RAG","GPT","OpenAI","Gemini","multimodal RAG","MM-RAG","unstructured","notebook","Gemini RAG","RAG Gemini","notebooks/showcases/multimodal-rag-using-Gemini.ipynb","/developers/templates/rag/template-multimodal-rag",{"title":243,"path":244,"stem":245,"meta":246},"Langchain and Pathway: RAG Apps with always-up-to-date knowledge","/framework/blog/langchain-integration","framework/blog/1002.langchain-integration",{"layout":170,"date":247,"hidden":176,"thumbnail":248,"tags":250,"notebook_export_path":252,"keywords":253},"2024-05-18",{"src":249},"/assets/content/showcases/vectorstore/Langchain-Pathway.png",[226,227,251],"engineering","notebooks/showcases/langchain-integration.ipynb",[229,230,231,232,254,237],"LangChain",{"title":256,"path":257,"stem":258,"children":259,"meta":271},"LlamaIndex and Pathway: RAG Apps with always-up-to-date knowledge","/framework/blog/llamaindex-pathway","framework/blog/1003.llamaindex-pathway",[260],{"title":256,"path":257,"stem":258,"meta":261},{"layout":170,"date":262,"thumbnail":263,"tags":265,"keywords":267,"docker_github_link":270,"hidden":176},"2024-01-12",{"src":264},"/assets/content/showcases/vectorstore/llamaindexpathway.png",[226,227,266,251],"case-study",[229,230,231,232,268,269],"LlamaIndex","docker","https://github.com/pathway-labs/realtime-indexer-qa-chat/tree/main",{"layout":170,"date":262,"thumbnail":272,"tags":273,"keywords":274,"docker_github_link":270,"hidden":176},{"src":264},[226,227,266,251],[229,230,231,232,268,269],{"title":276,"path":277,"stem":278,"meta":279},"Real-time Enterprise RAG with SharePoint","/framework/blog/enterprise_rag_sharepoint","framework/blog/1004.enterprise_rag_sharepoint",{"aside":176,"layout":170,"thumbnail":280,"tags":282,"date":283,"updatedDate":284,"keywords":285,"docker_github_link":291},{"src":281,"fit":223},"/assets/content/showcases/enterprise_sharepoint_rag/Enterprise_RAG-thumbnail.png",[226,227,266,251],"2024-07-15","2025-03-24",[229,230,286,287,288,289,290],"Real-time RAG","LiveAI™","Enterprise RAG","Docker","SharePoint","https://github.com/pathwaycom/llm-app/tree/main/templates/question_answering_rag",{"title":293,"path":294,"stem":295,"meta":296},"Evaluating RAG applications with RAGAS","/framework/blog/evaluating-rag","framework/blog/1005.evaluating-rag",{"layout":170,"date":297,"thumbnail":298,"tags":300,"notebook_export_path":301,"keywords":302,"hidden":176},"2025-03-13",{"src":299},"/assets/content/showcases/evals/rag-evals.png",[226,227,251],"notebooks/tutorials/rag-evaluations.ipynb",[229,230,231,232,303,237],"evaluations",{"title":305,"path":306,"stem":307,"meta":308},"How to use ChatGPT API in Python for your real-time data","/framework/blog/chatgpt-python-api-real-time-data","framework/blog/1010.chatgpt-python-api-real-time-data",{"aside":196,"layout":170,"thumbnail":309,"tags":311,"date":313,"related":196,"hidden":176},{"src":310,"provider":12},"/assets/blog/thumbnails/chatgpt-python-api-real-time-data.png",[170,312,251],"tutorial","2023-08-28",{"title":315,"path":316,"stem":317,"meta":318},"How La Poste uses Pathway microservices to deliver high-quality ETAs","/framework/blog/pathway-laposte-microservices","framework/blog/1020.pathway-laposte-microservices",{"layout":319,"thumbnail":320,"date":322,"tags":323,"keywords":325,"hidden":176},"template",{"src":321,"provider":12,"contain":176},"/assets/blog/thumbnails/pathway-laposte-microservices-th.png","2025-05-15",[226,324],"data-pipeline",[326,327,328,329,330,331,332,333],"ETL","Microservices","Microservice architecture","Micropipelines","Delta tables","S3","La Poste","ETA",{"title":335,"path":336,"stem":337,"meta":338},"50× Faster Local Embeddings with Batch UDFs","/framework/blog/local-embeddings-batch-udfs","framework/blog/1025.local-embeddings-batch-udfs",{"layout":170,"thumbnail":339,"date":341,"tags":342,"hidden":176},{"src":340,"provider":12,"contain":176},"/assets/blog/thumbnails/batch-udfs.png","2025-07-17",[170,251],{"title":344,"path":345,"stem":346,"meta":347},"Real-Time OCR with PaddleOCR and Pathway","/framework/blog/paddleocr","framework/blog/699.paddleocr",{"single":176,"aside":196,"layout":170,"date":348,"thumbnail":349,"tags":351},"2026-02-02",{"src":350},"/assets/content/blog/thumbnails/paddleocr-thumbnail.png",[170,312,251,186],{"title":353,"path":354,"stem":355,"meta":356},"Build LLM/RAG pipelines with YAML templates by Pathway","/framework/blog/llm-yaml-templates","framework/blog/700.llm-yaml-templates",{"layout":170,"date":357,"tags":358,"thumbnail":360,"related":362,"hidden":176},"2024-10-29",[359,251],"news",{"src":361,"provider":12},"/assets/blog/thumbnails/llm-yaml-templates-th.png",[363,364],"/blog/pathway-showcased-during-intel-ai-summit","/blog/pathway-joins-the-opea-to-accelerate-generative-ai-adoption",{"title":366,"path":367,"stem":368,"meta":369},"Pathway is Now Available on Microsoft Azure!","/framework/blog/azure-aci-deploy","framework/blog/701.azure-aci-deploy",{"layout":170,"date":370,"tags":371,"thumbnail":372,"related":374,"hidden":176},"2024-11-27",[359],{"src":373,"provider":12},"/assets/blog/thumbnails/pathway-azure-marketplace-th.png",[363,364],{"title":376,"path":377,"stem":378,"meta":379},"Gemini 2.0 for Document Ingestion and Analytics with Pathway","/framework/blog/gemini2-document-ingestion-and-analytics","framework/blog/878.gemini2-document-ingestion-and-analytics",{"layout":170,"date":380,"thumbnail":381,"tags":383,"coauthors":384,"hidden":176},"2025-02-20",{"src":382,"provider":12},"/assets/blog/thumbnails/gemini-2-th.png",[170,251],[385],{"name":386,"description\"":387,"img":388,"provider":12,"linkedin":389},"Berke Can Rizai","LLM Research Engineer","/assets/pictures/image_pathway_team.png","https://www.linkedin.com/in/berke-can-rizai/",{"title":391,"path":392,"stem":393,"meta":394},"Pathway’s Apache Iceberg Connectors for Real-Time Data Pipelines","/framework/blog/pathway-apache-iceberg-connectors","framework/blog/879.pathway-apache-iceberg-connectors",{"layout":170,"date":395,"thumbnail":396,"tags":398,"coauthors":399,"hidden":176},"2025-02-11",{"src":397,"provider":12},"/assets/blog/thumbnails/azure-iceberg-th.png",[170,312],[400],{"name":401,"description":402,"img":403,"provider":12,"linkedin":404},"Sergey Kulik","Lead Software Research Engineer at Pathway","/assets/pictures/image_kulik_pathway.jpg","https://www.linkedin.com/in/sergey-kulik-72506a33/",{"title":406,"path":407,"stem":408,"meta":409},"Real-Time AI Pipeline with DeepSeek, Ollama and Pathway","/framework/blog/deepseek-ollama","framework/blog/880.deepseek-ollama",{"single":176,"aside":196,"layout":170,"date":410,"thumbnail":411,"tags":413,"hidden":176},"2025-02-05",{"src":412,"provider":12},"/assets/blog/thumbnails/real-time-rag-pipeline-th.png",[170,312,251],{"title":415,"path":416,"stem":417,"meta":418},"Power and Deploy RAG Agent Tools with Pathway","/framework/blog/deploy-rag-agent-tools-with-pathway","framework/blog/881.deploy-rag-agent-tools-with-pathway",{"layout":170,"date":419,"thumbnail":420,"tags":422,"hidden":176},"2025-01-16",{"src":421,"provider":12},"/assets/blog/thumbnails/power-and-deploy-th.png",[170,251],{"title":424,"path":425,"stem":426,"meta":427},"How Text Embeddings help suggest similar words","/framework/blog/how-text-embeddings-help-suggest-similar-words","framework/blog/881.how-text-embeddings-help-suggest-similar-words",{"layout":170,"date":395,"thumbnail":428,"tags":430,"coauthors":432},{"src":429,"provider":12},"/assets/blog/thumbnails/pathway-text-embeddings-th.png",[431],"community",[433],{"name":434,"description":435,"img":436,"provider":12,"linkedin":437},"Yashasvee Taiwade","Student at Indian Institute of Technology, Bombay","/assets/blog/avatars/yashasvee-avatar.png","https://www.linkedin.com/in/yashasvee-taiwade-84826a337/",{"title":439,"path":440,"stem":441,"meta":442},"2025 PATHWAY CEO PREDICTIONS","/framework/blog/pathway-predictions-2025","framework/blog/882.pathway-predictions-2025",{"layout":170,"date":443,"thumbnail":444,"tags":446,"hidden":176},"2024-12-20",{"src":445,"provider":12},"/assets/blog/thumbnails/pathway-2025-predictions-th.png",[170],{"title":448,"path":449,"stem":450,"meta":451},"Scalable Alternative to Apache Kafka and Flink for Advanced Streaming: Build Real-Time Systems with NATS and Pathway","/framework/blog/build-real-time-systems-nats-pathway-alternative-kafka-flink","framework/blog/883.build-real-time-systems-nats-pathway-alternative-kafka-flink",{"layout":170,"date":452,"thumbnail":453,"tags":455,"coauthors":456,"hidden":176},"2024-12-11",{"src":454,"provider":12},"/assets/blog/thumbnails/pathway-nats-th.png",[312],[457],{"name":401,"description":402,"img":403,"linkedin":404},{"title":459,"path":460,"stem":461,"meta":462},"Achieve Sub-Second Latency with your S3 Storage without Kafka","/framework/blog/kafka-alternative","framework/blog/884.kafka-alternative",{"layout":170,"redirect":463,"date":464,"thumbnail":465,"tags":467,"hidden":176},"/developers/templates/etl/kafka-alternative","2024-08-27",{"src":466},"/assets/content/showcases/kafka-alternatives/kafka-alternative-banner.png",[312,251,266],{"title":469,"path":470,"stem":471,"meta":472},"Computing the Option Greeks using Pathway and Databento","/framework/blog/option-greeks","framework/blog/885.option-greeks",{"layout":170,"redirect":473,"date":224,"thumbnail":474,"tags":476,"hidden":176},"/developers/templates/etl/option-greeks",{"src":475},"/assets/content/showcases/option-greeks/option-greeks.svg",[312,251],{"title":478,"path":479,"stem":480,"meta":481},"Harnessing the Power of Now With Real-Time Analytics with Zuzanna Stamirowska & Hélène Stanway","/framework/blog/datacamp-real-time-analytics","framework/blog/886.datacamp-real-time-analytics",{"layout":170,"redirection":176,"date":482,"thumbnail":483,"tags":485,"hidden":176},"2024-11-13",{"src":484},"https://i3.ytimg.com/vi/6LcRxim5W1Y/maxresdefault.jpg",[486,487],"video","podcast",{"title":489,"path":490,"stem":491,"meta":492},"Adaptive RAG: cut your LLM costs without sacrificing accuracy","/framework/blog/adaptive-rag","framework/blog/887.adaptive-rag",{"layout":170,"redirect":493,"date":494,"thumbnail":495,"tags":497,"hidden":176},"/developers/templates/rag/adaptive-rag","2024-03-28",{"src":496},"/assets/content/blog/adaptive-rag-plots/visual-abstract.png",[170,498,251,312,266],"feature",{"title":500,"path":501,"stem":502,"meta":503},"Pathway Slide Search is now available on Intel Tiber Cloud","/framework/blog/intel-summit","framework/blog/890.intel-summit",{"layout":170,"date":504,"thumbnail":505,"tags":507,"related":508,"hidden":176},"2024-09-23",{"src":506,"provider":12},"/assets/blog/thumbnails/intel-accelerator-program-th.png",[359,266],[363,509],"/blog/intel-liftoff",{"title":511,"path":512,"stem":513,"meta":514},"Pathway Joins Linux Foundation’s OPEA alongside Intel and HuggingFace","/framework/blog/pathway-joins-the-opea-to-accelerate-generative-ai-adoption","framework/blog/891.pathway-joins-the-opea-to-accelerate-generative-ai-adoption",{"layout":170,"date":515,"thumbnail":516,"tags":518,"hidden":176},"2024-08-29",{"src":517,"provider":12},"/assets/blog/thumbnails/pathway-joins-opea-th.png",[359],{"title":520,"path":521,"stem":522,"meta":523},"Pathway showcased during Intel AI Summit","/framework/blog/pathway-showcased-during-intel-ai-summit","framework/blog/892.pathway-showcased-during-intel-ai-summit",{"layout":170,"date":524,"thumbnail":525,"tags":527,"hidden":176},"2024-09-02",{"src":526,"provider":12},"/assets/blog/thumbnails/intel-ai-summit-blog-th.png",[359],{"title":529,"path":530,"stem":531,"meta":532},"BNP Paribas Talk: AI and Real-time data processing in the banking industry, Vivatech 2024","/framework/blog/bnp-talk-at-vivatech","framework/blog/893.bnp-talk-at-vivatech",{"layout":170,"date":533,"thumbnail":534,"tags":536,"hidden":176},"2024-05-22",{"src":535,"provider":12},"/assets/blog/thumbnails/bnp-talk-at-vivatech-th.png",[486,487],{"title":538,"path":539,"stem":540,"meta":541},"Pathway at Modern Data Stack Meetup: Enabling Real-Time Operational Analytics for La Poste Groupe","/framework/blog/modern-data-network","framework/blog/894.modern-data-network",{"layout":170,"date":542,"thumbnail":543,"tags":545,"hidden":176},"2024-01-31",{"src":544,"provider":12},"/assets/blog/thumbnails/modern-data-network-th.png",[359,266],{"title":547,"path":548,"stem":549,"meta":550},"Transdev Project kickoff in Dunkirk, France","/framework/blog/project-kickoff-in-dunkirk","framework/blog/895.project-kickoff-in-dunkirk",{"layout":170,"date":551,"thumbnail":552,"tags":554,"related":555,"hidden":176},"2024-01-26",{"src":553,"provider":12},"/assets/blog/thumbnails/project-kickoff-in-dunkirk-th.png",[359,266],[556,557],"/blog/intel-accelerator-program","/blog/innovation-challenge",{"title":559,"path":560,"stem":561,"meta":562},"Pathway has been selected for Intel Liftoff","/framework/blog/intel-liftoff","framework/blog/896.intel-liftoff",{"layout":170,"date":563,"thumbnail":564,"tags":565,"related":566,"hidden":176},"2024-06-01",{"src":506,"provider":12},[359],[556,557],{"title":568,"path":569,"stem":570,"meta":571},"Pathway has been selected for the Intel’s accelerator program for deep tech startups","/framework/blog/intel-accelerator-program","framework/blog/897.intel-accelerator-program",{"layout":170,"date":572,"thumbnail":573,"tags":574,"related":575,"hidden":176},"2024-04-17",{"src":506,"provider":12},[359],[557],{"title":577,"path":578,"stem":579,"meta":580},"Finalist of the NATO Innovation Challenge!","/framework/blog/innovation-challenge","framework/blog/899.innovation-challenge",{"layout":170,"date":581,"thumbnail":582,"tags":584,"hidden":176},"2023-11-14",{"src":583,"provider":12},"/assets/blog/thumbnails/innovation-challenge-th.png",[359],{"title":586,"path":587,"stem":588,"meta":589},"Realtime Classification with Nearest Neighbors","/framework/blog/realtime-classification","framework/blog/900.realtime-classification",{"layout":170,"date":590,"thumbnail":591,"tags":593,"keywords":595,"hidden":176},"2022-10-26",{"src":592,"provider":12},"/assets/blog/thumbnails/pathway-card.png",[312,594],"machine-learning",[596,597,598,599,600,601,602,603],"Classification","regression","KNN","LSH","index","Locality Sensitive Hashing","MNIST","euclidean distance",{"title":605,"path":606,"stem":607,"meta":608},"Building End-to-End RAG with NPCI’s AI Leader","/framework/blog/building-end-to-end-rag-from-hardware-to-application","framework/blog/901.building-end-to-end-rag-from-hardware-to-application",{"keywords":609,"layout":170,"thumbnail":613,"tags":615,"date":616,"hidden":176},[610,611,230,612],"data preprocessing","adaptive RAG","similarity search",{"src":614,"provider":12},"/assets/blog/thumbnails/building-end-to-end-rag-from-hardware-to-app-th.png",[170,266],"2024-04-29",{"title":618,"path":619,"stem":620,"meta":621},"Machine Unlearning for LLMs: Build Apps that Self-Correct in Real-Time ","/framework/blog/machine-unlearning","framework/blog/902.machine-unlearning",{"keywords":622,"layout":170,"tags":627,"date":628,"thumbnail":629,"hidden":176},[623,624,625,611,230,626,600],"machine unlearning","LLMs","chatbot","adaptive index",[170],"2024-04-19",{"src":630},"/assets/content/blog/ML-unlearning-forgotten.svg",{"title":632,"path":633,"stem":634,"meta":635},"Batch processing vs stream processing","/framework/blog/batch-processing-vs-stream-processing","framework/blog/903.batch-processing-vs-stream-processing",{"keywords":636,"layout":170,"thumbnail":640,"tags":641,"date":642,"hidden":176},[637,638,639],"batch processing","stream processing","data streaming",{"src":592,"provider":12},[170],"2024-04-18",{"title":644,"path":645,"stem":646,"meta":647},"The Power of Real-Time Visualization for Logistics IoT","/framework/blog/power-of-real-time-visualization-for-logistics","framework/blog/903.power-of-real-time-visualization-for-logistics",{"redirection":176,"thumbnail":648,"tags":650,"date":651,"hidden":176},{"src":649,"provider":12},"/assets/blog/thumbnails/power-of-real-time-visualization-for-logistics-th.png",[266],"2024-04-16",{"title":653,"path":654,"stem":655,"meta":656},"Supply Chain Optimization with Integrated IoT Data","/framework/blog/supply-chain-optimization-with-integrated-iot-data","framework/blog/904.supply-chain-optimization-with-integrated-iot-data",{"redirection":176,"thumbnail":657,"tags":659,"date":660,"hidden":176},{"src":658,"provider":12},"/assets/blog/thumbnails/supply-chain-optimization-with-integrated-iot-data-th.png",[170],"2024-03-21",{"title":662,"path":663,"stem":664,"meta":665},"Build a real-time RAG chatbot using Google Drive and Sharepoint","/framework/blog/build-a-realtime-rag-chatbot-streamlit","framework/blog/905.build-a-realtime-rag-chatbot-streamlit",{"aside":196,"single":176,"redirection":176,"thumbnail":666,"tags":668,"date":669,"hidden":176},{"src":667,"provider":12},"/assets/blog/thumbnails/title_streamlit-pathway-llm-google-doc-th.svg",[170,266],"2024-03-07",{"title":671,"path":672,"stem":673,"meta":674},"Redefining AI’s Learning Curve: The Art of Unlearning","/framework/blog/tech-talks-podcast","framework/blog/906.tech-talks-podcast",{"redirection":176,"thumbnail":675,"tags":677,"date":678,"hidden":176},{"src":676},"https://techblogwriter.co.uk/wp-content/uploads/2023/12/Zuzanna-Stamirowska-CEO-and-Co-Founder-of-Pathway-1.jpg",[487],"2023-12-25",{"title":680,"path":681,"stem":682,"meta":683},"RAG - Streaming Application Pathway Legal Assistant","/framework/blog/rag-streaming-app-pathway-legacy-assistant","framework/blog/907.rag-streaming-app-pathway-legacy-assistant",{"layout":170,"keywords":684,"thumbnail":686,"tags":687,"date":689,"hidden":176},[685],"rag",{"src":592,"provider":12},[688],"pathway","2023-12-29",{"title":691,"path":692,"stem":693,"meta":694},"Retrieval Augmented Generation: Beginner’s Guide to RAG Apps","/framework/blog/retrieval-augmented-generation-beginners-guide-rag-apps","framework/blog/908.retrieval-augmented-generation-beginners-guide-rag-apps",{"layout":170,"keywords":695,"thumbnail":699,"tags":701,"date":689,"hidden":176},[685,696,697,698],"rag llm","vector index","retrieval augmented generation",{"src":700,"provider":12},"/assets/blog/thumbnails/rag-beginners-guide-th.png",[170,685],{"title":703,"path":704,"stem":705,"meta":706},"Guest speaker on the Digitalisation World podcast","/framework/blog/digitalisation-world-podcast","framework/blog/909.digitalisation-world-podcast",{"redirection":176,"thumbnail":707,"tags":709,"date":710,"hidden":176},{"src":708,"provider":12},"/assets/blog/thumbnails/digitalisation-world-th.png",[487],"2023-11-22",{"title":712,"path":713,"stem":714,"meta":715},"Guest speaker on The Futurists podcast","/framework/blog/futurist-podcast","framework/blog/910.futurist-podcast",{"layout":170,"aside":196,"thumbnail":716,"tags":718,"date":719,"hidden":176},{"src":717,"provider":12},"/assets/blog/thumbnails/thefuturists-th.webp",[170,487],"2023-11-10",{"title":721,"path":722,"stem":723,"meta":724},"How to build a real-time LLM app without vector databases","/framework/blog/build-a-real-time-llm-app-without-vector-databases-using-pathway","framework/blog/911.build-a-real-time-llm-app-without-vector-databases-using-pathway",{"aside":196,"single":176,"redirection":176,"thumbnail":725,"tags":727,"date":728,"hidden":176},{"src":726,"provider":12},"/assets/blog/thumbnails/relatime-app-th.png",[170],"2023-10-19",{"title":5,"path":177,"stem":180,"meta":730},{"keywords":731,"layout":170,"thumbnail":732,"tags":733,"date":174,"related":734,"hidden":176},[164,165,166,167,168,169],{"src":172},[170],[39,44],{"title":736,"path":737,"stem":738,"meta":739},"Kafka vs RabbitMQ for Data Streaming","/framework/blog/kafka-vs-rabbit-mq-for-data-streaming","framework/blog/913.kafka-vs-rabbit-mq-for-data-streaming",{"keywords":740,"layout":170,"thumbnail":743,"tags":745,"date":746,"related":747,"hidden":176},[741,742,639],"kafka","rabbitmq",{"src":744,"provider":12},"/assets/blog/thumbnails/rabbitmqvskafka-th.png",[170],"2023-09-29",[748],"/blog/building-enterprise-search-apis-with-llms-for-production",{"title":750,"path":751,"stem":752,"meta":753},"Building LLM enterprise search APIs","/framework/blog/llm-enterprise-search","framework/blog/914.llm-enterprise-search",{"layout":170,"thumbnail":754,"tags":756,"date":757,"hidden":176},{"src":755,"provider":12},"/assets/blog/thumbnails/llm-with-search-apis-th.png",[170],"2023-08-23",{"title":759,"path":760,"stem":761,"meta":762},"Pathway: Fastest Data Processing Engine - 2023 Benchmarks","/framework/blog/streaming-benchmarks-pathway-fastest-engine-on-the-market","framework/blog/915.streaming-benchmarks-pathway-fastest-engine-on-the-market",{"layout":170,"thumbnail":763,"tags":765,"date":766,"hidden":176},{"src":764,"contain":176},"/assets/content/blog/benchmark/bm-preview.png",[170],"2023-07-17",{"title":768,"path":769,"stem":770,"meta":771},"Tutorial: Distributed computing with live streaming data","/framework/blog/distributed-computing-with-live-streaming-data","framework/blog/918.distributed-computing-with-live-streaming-data",{"layout":170,"thumbnail":772,"tags":774,"date":775,"hidden":176},{"src":773},"/assets/content/blog/distributed-computing-with-live-streaming-data-th.png",[170,312],"2023-06-16",{"title":777,"path":778,"stem":779,"meta":780},"IoT Data Analytics: Processing Real-World Data in Real Time","/framework/blog/iot-data-analytics","framework/blog/918.iot-data-analytics",{"layout":170,"thumbnail":781,"tags":782,"date":783,"hidden":176},{"src":172},[170],"2023-06-21",{"title":785,"path":786,"stem":787,"meta":788},"Guest Speaker on Intel Business Podcast with Darren Pulsipher, Chief Solutions Architect","/framework/blog/intel-business-podcast","framework/blog/919.intel-business-podcast",{"layout":170,"thumbnail":789,"tags":791,"date":792,"hidden":176},{"src":790},"https://img.youtube.com/vi/Yn-EUssQG04/maxresdefault.jpg",[170,487],"2023-05-23",{"title":794,"path":795,"stem":796,"meta":797},"Pathway is Featured in Gartner’s Market Guide for Event Stream Processing","/framework/blog/market-guide-event-stream-processing","framework/blog/920.market-guide-event-stream-processing",{"layout":170,"tags":798,"date":799,"thumbnail":800,"enterprise":176,"keywords":802,"hidden":176},[359],"2023-05-15",{"src":801,"contain":176,"padding":176},"/assets/content/blog/gartner-th.png",[803,688,804,805,806],"gartner","event","stream","processing",{"title":808,"path":809,"stem":810,"meta":811},"Guest Speaker on Charbon Podcast (in French)","/framework/blog/charbon-podcast-with-claire","framework/blog/921.charbon-podcast-with-claire",{"layout":170,"thumbnail":812,"tags":814,"lang":815,"date":816,"hidden":176},{"src":813},"/assets/content/blog/charbon-podcast-with-claire.png",[170,487],"french","2023-04-26",{"title":818,"path":819,"stem":820,"meta":821},"Podcast speaker on SuperDataScience Podcast #669","/framework/blog/superdatascience-with-adrian","framework/blog/922.superdatascience-with-adrian",{"layout":170,"thumbnail":822,"tags":824,"date":825,"hidden":176},{"src":823},"https://img.youtube.com/vi/AjxLnll8QHg/maxresdefault.jpg",[170,487],"2023-04-11",{"title":827,"path":828,"stem":829,"meta":830},"Pathway featured in Sifted briefing on GenAI","/framework/blog/pathway-featured-in-sifted-briefing-on-genai","framework/blog/923.pathway-featured-in-sifted-briefing-on-genai",{"layout":170,"thumbnail":831,"tags":833,"date":834,"hidden":176},{"src":832},"/assets/content/blog/sifted-th.png",[170,359],"2023-03-09",{"title":836,"path":837,"stem":838,"meta":839},"Improving asset utilization with Pathway: combining IoT data with real-time data processing","/framework/blog/improving-asset-utilization","framework/blog/924.improving-asset-utilization",{"layout":170,"thumbnail":840,"tags":842,"date":843,"hidden":176},{"src":841},"/assets/content/blog/improving-asset-utilzation-th.png",[170],"2023-04-06",{"title":845,"path":846,"stem":847,"meta":848},"What is Fuzzy Join and How Can it Help You Make Sense of Your Data?","/framework/blog/fuzzy-join","framework/blog/925.fuzzy-join",{"layout":170,"thumbnail":849,"tags":851,"date":852,"hidden":176},{"src":850},"/assets/content/blog/fuzzy-join/th-fuzzy-join.svg",[170,312,251],"2023-04-05",{"title":854,"path":855,"stem":856,"meta":857},"Discussing supply chain analytics on the Data Engineering Podcast","/framework/blog/data-engineering-podcast","framework/blog/926.data-engineering-podcast",{"layout":170,"thumbnail":858,"tags":860,"date":861,"hidden":176},{"src":859},"/assets/content/blog/data-engineering-podcast.jpg",[170,487],"2022-10-16",{"title":863,"path":864,"stem":865,"meta":866},"The value of real-time analytics - IoT for all podcast","/framework/blog/iot-for-all-podcast","framework/blog/927.Iot-for-all-podcast",{"layout":170,"thumbnail":867,"tags":869,"date":870,"hidden":176},{"src":868},"https://i3.ytimg.com/vi/4TSK0fpMC4I/maxresdefault.jpg",[170,487],"2023-02-20",{"title":872,"path":873,"stem":874,"meta":875},"Guest speaker at Maddyness Keynote","/framework/blog/maddy-keynote-2023","framework/blog/928.Maddy-keynote-2023",{"layout":170,"thumbnail":876,"tags":878,"date":879,"hidden":176},{"src":877},"/assets/content/blog/maddy-keynote.jpg",[170,359],"2023-02-08",{"title":881,"path":882,"stem":883,"meta":884},"Guest speakers on Female Foundry Podcast","/framework/blog/female-foundry-podcast","framework/blog/929.Female-Foundry-Podcast",{"layout":170,"thumbnail":885,"tags":887,"date":888,"hidden":176},{"src":886},"/assets/content/blog/female-foundry-podcast.jpg",[170,487],"2023-02-11",{"title":890,"path":891,"stem":892,"meta":893},"Podcast speaker on Super Data Science Podcast","/framework/blog/liquidneuralnetworks","framework/blog/930.LiquidNeuralNetworks",{"layout":170,"date":894,"thumbnail":895,"tags":897,"hidden":176},"2022-12-02",{"src":896},"/assets/content/blog/LiquidNeurealnetwork.jpg",[170,486,487],{"title":899,"path":900,"stem":901,"meta":902},"Pathway Premieres at PyData Global","/framework/blog/pydata","framework/blog/932.PyData",{"layout":170,"date":903,"thumbnail":904,"tags":906,"hidden":176},"2022-12-01",{"src":905,"contain":176},"/assets/content/blog/logo-conference-pydata-global-2021.png",[170],{"title":908,"path":909,"stem":910,"meta":911},"Pathway at ODSC West","/framework/blog/odsc","framework/blog/934.ODSC",{"layout":170,"date":912,"thumbnail":913,"tags":915,"hidden":176},"2022-11-01",{"src":914},"/assets/content/blog/ODSCWest.png",[170,916],"culture",{"title":918,"path":919,"stem":920,"meta":921},"Guest speaker on IoT For All Podcast","/framework/blog/iotforall","framework/blog/936.IoTforAll",{"layout":170,"thumbnail":922,"tags":924,"date":925,"hidden":176},{"src":923},"/assets/content/blog/IoT-for-all.png",[170,486,487],"2022-08-01",{"title":927,"path":928,"stem":929,"meta":930},"Pathway graduated from CDL-Montreal!","/framework/blog/cdl","framework/blog/938.CDL",{"layout":170,"date":931,"thumbnail":932,"tags":934,"hidden":176},"2022-06-01",{"src":933},"/assets/content/blog/Pathway-CDL-Montreal.jpeg",[170],{"title":936,"path":937,"stem":938,"meta":939},"Pathway helps La Poste reduce IoT costs by 50%","/framework/blog/laposte","framework/blog/940.LaPoste",{"layout":170,"date":931,"thumbnail":940,"tags":942,"hidden":176},{"src":941,"contain":176,"padding":176},"/assets/content/blog/ROILaposte.png",[170],{"title":944,"path":945,"stem":946,"meta":947},"Why Pathway","/framework/blog/why-pathway","framework/blog/945.why-pathway",{"layout":170,"date":948,"thumbnail":949,"tags":951,"hidden":176},"2022-05-24",{"src":950},"/assets/content/blog/Pathwaybanner.jpg",[170],{"title":953,"path":954,"stem":955,"meta":956},"Pathway wins the #Spring50 Pitch Contest!","/framework/blog/spring50","framework/blog/950.spring50",{"layout":170,"date":957,"thumbnail":958,"tags":960,"hidden":176},"2022-05-13",{"src":959,"provider":12,"contain":176},"/assets/blog/thumbnails/digital-50-contest.png",[170],{"title":962,"path":963,"stem":964,"meta":965},"Pathway is a WomenTech EU Laureate","/framework/blog/womentecheu","framework/blog/960.WomenTechEU",{"layout":170,"date":966,"thumbnail":967,"tags":969,"hidden":176},"2022-03-01",{"src":968,"provider":12},"/assets/blog/thumbnails/co-funded-th.png",[170],{"title":971,"path":972,"stem":973,"meta":974},"Pathway is a Gartner Representative Vendor","/framework/blog/gartner","framework/blog/970.Gartner",{"layout":170,"date":975,"thumbnail":976,"tags":978,"enterprise":176,"hidden":176},"2021-10-19",{"src":977,"provider":12,"contain":176},"/assets/blog/thumbnails/gartner-th.png",[170],{"title":980,"path":981,"stem":982,"meta":983},"Pathway joins Agoranov, French Science and Tech incubator - in Paris, France","/framework/blog/agoranov","framework/blog/980.Agoranov",{"layout":170,"date":984,"thumbnail":985,"tags":987,"hidden":176},"2021-10-09",{"src":986,"provider":12,"contain":176},"/assets/blog/thumbnails/Agoranov-Logo.png",[170],{"title":989,"path":990,"stem":991,"meta":992},"Pathway has been selected by Hello Tomorrow as a Deep Tech Pioneer ","/framework/blog/hellotomorrow","framework/blog/990.hellotomorrow",{"layout":170,"date":993,"thumbnail":994,"tags":996,"hidden":176},"2021-09-27",{"src":995,"provider":12,"contain":176},"/assets/blog/thumbnails/hello-tomorrow-th.png",[170],{"title":998,"path":999,"stem":1000,"meta":1001},"Gartner® recognizes Pathway as an Emerging Visionary in GenAI Engineering","/framework/blog/gartner-gen-ai-engineering","framework/blog/996.gartner-gen-ai-engineering",{"layout":170,"date":1002,"thumbnail":1003,"tags":1005,"related":1006},"2024-11-14",{"src":1004,"provider":12},"/assets/blog/thumbnails/th-pathway-gartner-emerging-market-quadrants.png",[359],[557],{"title":1008,"path":1009,"stem":1010,"meta":1011},"Adaptive Agents for Real-Time RAG: Domain-Specific AI for Legal, Finance & Healthcare","/framework/blog/adaptive-agents-rag","framework/blog/adaptive-agents-rag",{"layout":170,"date":1012,"thumbnail":1013,"tags":1015},"2025-04-02",{"src":1014,"provider":12},"/assets/blog/thumbnails/adaptive-agents-rag-th.png",[431],{"title":1017,"path":1018,"stem":1019,"meta":1020},"How AI Agents in Finance Are Transforming Financial Due Diligence: FA3STER","/framework/blog/ai-agents-finance-due-diligence","framework/blog/ai-agents-finance-due-diligence",{"layout":170,"date":1021,"thumbnail":1022,"tags":1024},"2025-03-26",{"src":1023,"provider":12},"/assets/blog/thumbnails/ai-agents-finance-due-diligence-th.png",[431],{"title":1026,"path":1027,"stem":1028,"meta":1029},"Financial Report Analysis with LiveAI™","/framework/blog/ai-financial-report-analysis","framework/blog/ai-financial-report-analysis",{"layout":170,"date":1030,"thumbnail":1031,"tags":1033},"2025-09-03",{"src":1032,"provider":12},"/assets/blog/thumbnails/earnings-call-transcript-analysis-framework-th.png",[431],{"title":1035,"path":1036,"stem":1037,"meta":1038},"LiveAI™ for SEC Filings Analysis","/framework/blog/ai-for-sec-filings","framework/blog/ai-for-sec-filings",{"layout":170,"date":1039,"thumbnail":1040,"tags":1042},"2025-07-09",{"src":1041,"provider":12},"/assets/blog/thumbnails/ai-for-sec-filings-th.png",[431],{"title":1044,"path":1045,"stem":1046,"meta":1047},"AI Paper Reviewer | LiveAI™ for Conference Classification","/framework/blog/ai-paper-reviewer","framework/blog/ai-paper-reviewer",{"layout":170,"date":1048,"thumbnail":1049,"tags":1051},"2025-06-19",{"src":1050,"provider":12},"/assets/blog/thumbnails/ai-paper-reviewer-th.png",[431],{"title":1053,"path":1054,"stem":1055,"meta":1056},"LiveAI™ Tools for Equity Research & Compliance Management","/framework/blog/ai-tools-for-equity-analysis","framework/blog/ai-tools-for-equity-analysis",{"layout":170,"date":1057,"thumbnail":1058,"tags":1060},"2025-04-23",{"src":1059,"provider":12},"/assets/blog/thumbnails/live-ai-legal-compliance-financial-analysis-th.png",[431],{"title":1062,"path":1063,"stem":1064,"meta":1065},"Pathway’s BDH solves Sudoku Extreme with 97.4% accuracy, while leading LLMs are close to 0","/framework/blog/beyond-transformers-sudoku-bench","framework/blog/beyond-transformers-sudoku-bench",{"date":1066,"thumbnail":1067,"tags":1069,"related":196,"redirect":1071},"2026-03-17",{"src":1068,"provider":12},"/assets/blog/thumbnails/the-sudoku-test-th.png",[1070],"bdh","/research/beyond-transformers-sudoku-bench",{"title":1073,"path":1074,"stem":1075,"meta":1076},"Build RAG Apps in YAML - Recording from the Intel AI DevSummit","/framework/blog/build-rag-apps-in-yaml-recording-from-the-intel-ai-devsummit","framework/blog/build-rag-apps-in-yaml-recording-from-the-intel-ai-devsummit",{"aside":196,"layout":170,"date":297,"thumbnail":1077,"tags":1079,"hidden":176},{"src":1078},"https://i3.ytimg.com/vi/OEEZ5qSHJSs/hqdefault.jpg",[486,312,251],{"title":1081,"path":1082,"stem":1083,"meta":1084},"Data Discovery to Data Pipeline Process","/framework/blog/data-discovery-to-data-pipeline-process","framework/blog/data-discovery-to-data-pipeline-process",{"layout":170,"date":1085,"thumbnail":1086,"tags":1088},"2025-02-28",{"src":1087,"provider":12},"/assets/blog/thumbnails/data-discovery-th.png",[431],{"title":1090,"path":1091,"stem":1092,"meta":1093},"Exploring Kappa Architecture with Pathway","/framework/blog/exploring-kappa-architecture-with-pathway","framework/blog/exploring-kappa-architecture-with-pathway",{"single":176,"aside":196,"layout":170,"date":1085,"thumbnail":1094,"tags":1096},{"src":1095,"provider":12},"/assets/blog/thumbnails/kappa-architecture-th.png",[431],{"title":1098,"path":1099,"stem":1100,"meta":1101},"Resilient Financial API Integration with LiveAI™","/framework/blog/financial-api-integration-live-ai","framework/blog/financial-api-integration-live-ai",{"layout":170,"date":1102,"thumbnail":1103,"tags":1105},"2025-05-23",{"src":1104,"provider":12},"/assets/blog/thumbnails/financial-api-integration-live-ai-th.png",[431],{"title":1107,"path":1108,"stem":1109,"meta":1110},"LiveAI™ for Financial Intelligence with Event-Based State Machine","/framework/blog/financial-intelligence-with-event-based-state-machine","framework/blog/financial-intelligence-with-event-based-state-machine",{"keywords":1111,"layout":170,"date":1119,"thumbnail":1120,"tags":1122},[1112,1113,1114,1115,1116,1117,1118],"Large vector stores","Event-Based State Machine","dynamic data sources","reliability","hallucination","Financial Intelligence","LiveAI™ framework","2025-06-12",{"src":1121,"provider":12},"/assets/blog/thumbnails/financial-intelligence-with-event-based-state-machine-th.png",[431],{"title":1124,"path":1125,"stem":1126,"meta":1127},"Building a Real-Time Radiology AI System with Pathway and LandingAI","/framework/blog/landing-ai","framework/blog/landing-ai",{"layout":170,"date":1128,"thumbnail":1129,"tags":1131},"2025-10-02",{"src":1130,"provider":12},"/assets/blog/thumbnails/landing-ai-th.png",[431],{"title":1133,"path":1134,"stem":1135,"meta":1136},"Pathway MCP Server: Live Indexing & Analytics for your Agents","/framework/blog/live-ai-mcp-server","framework/blog/live-ai-mcp-server",{"aside":196,"layout":170,"date":1137,"thumbnail":1138,"tags":1140,"coauthors":1141,"hidden":176},"2025-08-22",{"src":1139,"contain":176},"/assets/content/blog/MCP/mcp-server-thumbnail.png",[312,251],[1142],{"name":1143,"description\"":1144,"img":1145,"provider":12,"linkedin":1146},"Mudit Srivastava","Director of Growth","/assets/authors/mudit-srivastava.jpg","https://www.linkedin.com/in/muditjps/",{"title":1148,"path":1149,"stem":1150,"meta":1151},"Multi Agentic RAG & LiveAI™ for Finance and Legal Solutions","/framework/blog/live-ai-multi-agentic-rag","framework/blog/live-ai-multi-agentic-rag",{"layout":170,"date":1152,"thumbnail":1153,"tags":1155},"2025-04-09",{"src":1154,"provider":12},"/assets/blog/thumbnails/live-ai-multi-agentic-rag-th.png",[431],{"title":1157,"path":1158,"stem":1159,"meta":1160},"Multi Agent RAG with Interleaved Retrieval and Reasoning for Long Docs","/framework/blog/multi-agent-rag-interleaved-retrieval-reasoning","framework/blog/multi-agent-rag-interleaved-retrieval-reasoning",{"layout":170,"date":1161,"thumbnail":1162,"tags":1164},"2025-03-19",{"src":1163,"provider":12},"/assets/blog/thumbnails/multi-agent-rag-system-for-long-document-question-answering-th.png",[431],{"title":1166,"path":1167,"stem":1168,"meta":1169},"Real-Time Multimodal Data Processing with Pathway and Docling","/framework/blog/multimodal-data-processing","framework/blog/multimodal-data-processing",{"aside":196,"layout":170,"date":1170,"thumbnail":1171,"tags":1173,"coauthors":1174,"hidden":176},"2025-05-30",{"src":1172,"contain":176},"/assets/content/blog/docling-parser/docling-th.png",[312,251],[1175],{"name":1176,"description\"":1177,"img":1178,"linkedin":1179},"Albert Roethel","AI Engineer","/assets/content/blog/avatars/albert.jpg","https://www.linkedin.com/in/albertroethel/",{"title":1181,"path":1182,"stem":1183,"meta":1184},"Regulatory Compliance Automation with LiveAI™","/framework/blog/regulatory-compliance-automation-live-ai","framework/blog/regulatory-compliance-automation-live-ai",{"layout":170,"date":1185,"thumbnail":1186,"tags":1188,"related":196},"2025-04-30",{"src":1187,"provider":12},"/assets/blog/thumbnails/ai-legal-document-analysis-th.png",[431],{"title":1190,"path":1191,"stem":1192,"meta":1193},"Unlocking data stream processing [Part 1] - real-time linear regression","/framework/blog/unlocking-data-stream-processing-1","framework/blog/unlocking-data-stream-processing-1",{"layout":170,"redirection":176,"date":1194,"thumbnail":1195,"tags":1197,"hidden":176},"2023-02-16",{"src":1196},"/assets/content/tutorials/unlocking-data-stream-processing-1-th.png",[312,251],{"title":1199,"path":1200,"stem":1201,"meta":1202},"Unlocking data stream processing [Part 2] - realtime server logs monitoring with a sliding window","/framework/blog/unlocking-data-stream-processing-2","framework/blog/unlocking-data-stream-processing-2",{"layout":170,"redirection":176,"date":834,"thumbnail":1203,"tags":1205,"hidden":176},{"src":1204},"/assets/content/tutorials/unlocking-data-stream-processing-2-th.png",[312,251],{"title":1207,"path":1208,"stem":1209,"meta":1210},"Unlocking data stream processing [Part 3] - data enrichment with fuzzy joins","/framework/blog/unlocking-data-stream-processing-3","framework/blog/unlocking-data-stream-processing-3",{"layout":170,"redirection":176,"date":1211,"thumbnail":1212,"tags":1214,"hidden":176},"2023-05-09",{"src":1213},"/assets/content/tutorials/unlocking-data-stream-processing-3-th.png",[312,251],{"layout":195,"aside":196,"toc":196,"single":176},{"id":1217,"title":5,"author":1218,"body":1219,"date":1307,"description":24,"extension":161,"hidden":176,"keywords":1308,"meta":1309,"navigation":176,"path":177,"seo":1311,"stem":180,"tags":1312,"thumbnail":1313,"__hash__":1314},"blog/framework/blog/912.signal-processing.md",{"id":7,"url":8,"name":9,"description":10,"img":11,"provider":12,"linkedin":13},{"type":15,"value":1220,"toc":1299},[1221,1223,1225,1229,1235,1237,1239,1241,1243,1257,1259,1261,1263,1269,1273,1275,1277,1279,1281,1283,1285,1287,1289],[18,1222,5],{"id":20},[22,1224,24],{},[22,1226,27,1227,33],{},[29,1228,32],{"href":31},[22,1230,36,1231,41,1233,46],{},[29,1232,40],{"href":39},[29,1234,45],{"href":44},[48,1236,51],{"id":50},[22,1238,54],{},[48,1240,58],{"id":57},[22,1242,61],{},[63,1244,1245,1249,1253],{},[66,1246,1247,72],{},[69,1248,71],{},[66,1250,1251,78],{},[69,1252,77],{},[66,1254,1255,84],{},[69,1256,83],{},[48,1258,88],{"id":87},[22,1260,91],{},[48,1262,95],{"id":94},[22,1264,98,1265,103,1267,106],{},[100,1266,102],{},[100,1268,102],{},[22,1270,1271],{},[69,1272,111],{},[22,1274,114],{},[22,1276,117],{},[22,1278,120],{},[48,1280,124],{"id":123},[22,1282,127],{},[22,1284,130],{},[48,1286,134],{"id":133},[22,1288,137],{},[139,1290,1291,1295],{},[66,1292,1293],{},[29,1294,145],{"href":39},[66,1296,1297],{},[29,1298,150],{"href":44},{"title":152,"searchDepth":153,"depth":153,"links":1300},[1301,1302,1303,1304,1305,1306],{"id":50,"depth":153,"text":51},{"id":57,"depth":153,"text":58},{"id":87,"depth":153,"text":88},{"id":94,"depth":153,"text":95},{"id":123,"depth":153,"text":124},{"id":133,"depth":153,"text":134},"2023-10-16T00:00:00.000Z",[164,165,166,167,168,169],{"layout":170,"related":1310},[39,44],{"title":5,"description":24},[170],{"src":172},"ytiXSbyEuLXZSdxR73p7DzYqphLcY1E4gM0-E1M7MuY",[1316],{"id":1317,"title":1318,"author":1319,"body":1320,"description":5863,"extension":161,"meta":5864,"navigation":176,"path":39,"seo":5877,"sitemap":5878,"stem":5879,"__hash__":5880},"content/2.developers/7.templates/ETL/100.gaussian_filtering_python.md","Gaussian Filtering in Real-time: Signal processing with out-of-order data streams",{"id":7,"url":8,"name":9,"description":10,"img":11,"provider":12,"linkedin":13},{"type":15,"value":1321,"toc":5850},[1322,1325,1333,1336,1347,1349,1355,1362,1366,1382,1389,1395,1402,1405,1593,1596,1604,1614,1698,1704,1727,1873,1877,1880,1888,1907,1910,1915,1918,1921,1946,2338,2341,2525,2531,2534,2551,2696,2713,2719,2722,2728,2731,2737,2753,2771,2950,2954,2968,3207,3213,3312,3316,3324,3385,3560,3565,3568,3572,3575,3578,3584,3587,3590,3861,3866,3869,3872,4280,4283,4431,4436,4439,4732,4746,4967,4973,4976,4979,5045,5157,5178,5183,5186,5240,5370,5375,5378,5381,5463,5610,5615,5618,5621,5817,5822,5825,5831,5835,5838,5847],[18,1323,1318],{"id":1324},"gaussian-filtering-in-real-time-signal-processing-with-out-of-order-data-streams",[22,1326,1327,1328,1332],{},"In this tutorial, you will learn how to perform signal processing on out-of-order signal data. Specifically, you will apply a Gaussian filter on a signal data stream with irregular sampling.\nThis is an important skill to master because most real-world signal processing scenarios do not follow the regular and precise timing assumptions on which many standard signal processing systems are built.\nRead our blog on ",[29,1329,1331],{"href":1330},"/blog/signal-processing","the challenges of real-time signal processing"," for more context.",[22,1334,1335],{},"The tutorial will walk you through the following 3 steps:",[139,1337,1338,1341,1344],{},[66,1339,1340],{},"Preprocess the noisy data using interval joins;",[66,1342,1343],{},"Apply a Gaussian filter to smoothen the signal;",[66,1345,1346],{},"Dive deeper into how Pathway handles out-of-order data points.",[48,1348,124],{"id":123},[22,1350,1351,1352],{},"Standard DSP systems do not support the addition of late data points because they assume a perfectly regular sampling. This allows these systems to wait until all signals to arrive before starting the processing computation.\nBut in real-world and real-time settings, these assumptions are rarely guaranteed. Data points may arrive late or, worse, not at all, in which case the computation would never start.\nPathway allows you to manipulate streaming data as if it were static, i.e., as if all the data is already gathered.\n",[69,1353,1354],{},"You don't have to worry about the late or out-of-order data: Pathway will handle it by updating its results whenever late or out-of-order points are added to the system.",[22,1356,1357,1358,1361],{},"However, you still have to deal with data misalignment.\nTherefore, ",[69,1359,1360],{},"you must consider the signal as a time series",", and data points should be seen as a list of (time, value) pairs rather than an array of regularly sampled values.\nWhile the time variable makes signal processing more complex, Pathway offers several functions to make signal processing on realtime data streams as simple as it can be.",[48,1363,1365],{"id":1364},"gaussian-filtering-in-python-preprocess-the-data","Gaussian Filtering in Python: Preprocess the data",[22,1367,1368,1369,1375,1376,1381],{},"Because of the out-of-order data points, the use of IIR (",[29,1370,1374],{"href":1371,"rel":1372},"https://en.wikipedia.org/w/index.php?title=Infinite_impulse_response&oldid=1079405495",[1373],"nofollow","Infinite Impulse Response",") filters, which depend on the sequential order of data, is impractical as their outputs cannot be efficiently recomputed when late points arrive.\nInstead, Pathway offers ways to easily implement FIR (",[29,1377,1380],{"href":1378,"rel":1379},"https://en.wikipedia.org/w/index.php?title=Finite_impulse_response&oldid=1172063361",[1373],"Finite Impulse Response",") filters, where each data point influences the filter's response for a limited duration.\nConsequently, you should primarily employ kernel smoothing techniques to process signals in a manner that accommodates unordered or non-sequential data points, ensuring accurate and efficient analysis.",[22,1383,1384,1385,46],{},"Furthermore, without sample numbers, you cannot rely on a prev/next order among your data points: not only some points may be missing, but there may also be an important temporal gap between two consecutive points.\nYou need to connect your point to other points close in time.\nTo do so, Pathway provides ",[29,1386,1388],{"href":1387},"/developers/user-guide/temporal-data/interval-join","interval joins",[22,1390,1391,1392,46],{},"Interval joins merge records by comparing the timestamp from each record of two tables and evaluating whether the difference falls within a specified time window or ",[100,1393,1394],{},"interval",[22,1396,1397],{},[1398,1399],"img",{"alt":1400,"src":1401},"Illustration of interval join","assets/content/tutorials/fleet_eta_interval_join/illustrative_interval_join.png",[22,1403,1404],{},"Interval joins are a great tool to gather all the close points together:",[1406,1407,1411],"pre",{"className":1408,"code":1409,"language":1410,"meta":152,"style":152},"language-python shiki shiki-themes material-theme-palenight","results = table_a.interval_join_left(\n    table_b,\n    pw.left.timestamp,\n    pw.right.timestamp,\n    pw.temporal.interval(-50, 50),\n    pw.left.timestamp != pw.right.timestamp\n).select(\n    pw.left.timestamp,\n    neighbor_timestamp=pw.right.timestamp,\n)\n","python",[100,1412,1413,1438,1446,1465,1481,1511,1539,1550,1565,1587],{"__ignoreMap":152},[1414,1415,1418,1422,1426,1429,1431,1435],"span",{"class":1416,"line":1417},"line",1,[1414,1419,1421],{"class":1420},"s0W1g","results ",[1414,1423,1425],{"class":1424},"sAklC","=",[1414,1427,1428],{"class":1420}," table_a",[1414,1430,46],{"class":1424},[1414,1432,1434],{"class":1433},"sdLwU","interval_join_left",[1414,1436,1437],{"class":1424},"(\n",[1414,1439,1440,1443],{"class":1416,"line":153},[1414,1441,1442],{"class":1433},"    table_b",[1414,1444,1445],{"class":1424},",\n",[1414,1447,1449,1452,1454,1458,1460,1463],{"class":1416,"line":1448},3,[1414,1450,1451],{"class":1433},"    pw",[1414,1453,46],{"class":1424},[1414,1455,1457],{"class":1456},"s-wAU","left",[1414,1459,46],{"class":1424},[1414,1461,1462],{"class":1456},"timestamp",[1414,1464,1445],{"class":1424},[1414,1466,1468,1470,1472,1475,1477,1479],{"class":1416,"line":1467},4,[1414,1469,1451],{"class":1433},[1414,1471,46],{"class":1424},[1414,1473,1474],{"class":1456},"right",[1414,1476,46],{"class":1424},[1414,1478,1462],{"class":1456},[1414,1480,1445],{"class":1424},[1414,1482,1484,1486,1488,1491,1493,1495,1498,1502,1505,1508],{"class":1416,"line":1483},5,[1414,1485,1451],{"class":1433},[1414,1487,46],{"class":1424},[1414,1489,1490],{"class":1456},"temporal",[1414,1492,46],{"class":1424},[1414,1494,1394],{"class":1433},[1414,1496,1497],{"class":1424},"(-",[1414,1499,1501],{"class":1500},"sx098","50",[1414,1503,1504],{"class":1424},",",[1414,1506,1507],{"class":1500}," 50",[1414,1509,1510],{"class":1424},"),\n",[1414,1512,1514,1516,1518,1520,1522,1524,1527,1530,1532,1534,1536],{"class":1416,"line":1513},6,[1414,1515,1451],{"class":1433},[1414,1517,46],{"class":1424},[1414,1519,1457],{"class":1456},[1414,1521,46],{"class":1424},[1414,1523,1462],{"class":1456},[1414,1525,1526],{"class":1424}," !=",[1414,1528,1529],{"class":1433}," pw",[1414,1531,46],{"class":1424},[1414,1533,1474],{"class":1456},[1414,1535,46],{"class":1424},[1414,1537,1538],{"class":1456},"timestamp\n",[1414,1540,1542,1545,1548],{"class":1416,"line":1541},7,[1414,1543,1544],{"class":1424},").",[1414,1546,1547],{"class":1433},"select",[1414,1549,1437],{"class":1424},[1414,1551,1553,1555,1557,1559,1561,1563],{"class":1416,"line":1552},8,[1414,1554,1451],{"class":1433},[1414,1556,46],{"class":1424},[1414,1558,1457],{"class":1456},[1414,1560,46],{"class":1424},[1414,1562,1462],{"class":1456},[1414,1564,1445],{"class":1424},[1414,1566,1568,1572,1574,1577,1579,1581,1583,1585],{"class":1416,"line":1567},9,[1414,1569,1571],{"class":1570},"s7ZW3","    neighbor_timestamp",[1414,1573,1425],{"class":1424},[1414,1575,1576],{"class":1433},"pw",[1414,1578,46],{"class":1424},[1414,1580,1474],{"class":1456},[1414,1582,46],{"class":1424},[1414,1584,1462],{"class":1456},[1414,1586,1445],{"class":1424},[1414,1588,1590],{"class":1416,"line":1589},10,[1414,1591,1592],{"class":1424},")\n",[22,1594,1595],{},"However, they create an entry for each point in the interval:",[1406,1597,1602],{"className":1598,"code":1600,"language":1601},[1599],"language-text","timestamp | neighbor_timestamp\n------------------------------\n...\n100       | 77\n100       | 97\n100       | 132\n...\n------------------------------\n","text",[100,1603,1600],{"__ignoreMap":152},[22,1605,1606,1607,1610,1611,46],{},"To gather all the entries corresponding to the same interval, we could use a ",[100,1608,1609],{},"groupby"," followed by a ",[100,1612,1613],{},"reduce",[1406,1615,1617],{"className":1408,"code":1616,"language":1410,"meta":152,"style":152},"results = results.groupby(results.timestamp).reduce(\n    results.timestamp,\n    neighbors=pw.reducers.tuple(pw.this.neighbors_timestamps),\n)\n",[100,1618,1619,1648,1659,1694],{"__ignoreMap":152},[1414,1620,1621,1623,1625,1628,1630,1632,1635,1638,1640,1642,1644,1646],{"class":1416,"line":1417},[1414,1622,1421],{"class":1420},[1414,1624,1425],{"class":1424},[1414,1626,1627],{"class":1420}," results",[1414,1629,46],{"class":1424},[1414,1631,1609],{"class":1433},[1414,1633,1634],{"class":1424},"(",[1414,1636,1637],{"class":1433},"results",[1414,1639,46],{"class":1424},[1414,1641,1462],{"class":1456},[1414,1643,1544],{"class":1424},[1414,1645,1613],{"class":1433},[1414,1647,1437],{"class":1424},[1414,1649,1650,1653,1655,1657],{"class":1416,"line":153},[1414,1651,1652],{"class":1433},"    results",[1414,1654,46],{"class":1424},[1414,1656,1462],{"class":1456},[1414,1658,1445],{"class":1424},[1414,1660,1661,1664,1666,1668,1670,1673,1675,1678,1680,1682,1684,1687,1689,1692],{"class":1416,"line":1448},[1414,1662,1663],{"class":1570},"    neighbors",[1414,1665,1425],{"class":1424},[1414,1667,1576],{"class":1433},[1414,1669,46],{"class":1424},[1414,1671,1672],{"class":1456},"reducers",[1414,1674,46],{"class":1424},[1414,1676,1677],{"class":1433},"tuple",[1414,1679,1634],{"class":1424},[1414,1681,1576],{"class":1433},[1414,1683,46],{"class":1424},[1414,1685,1686],{"class":1456},"this",[1414,1688,46],{"class":1424},[1414,1690,1691],{"class":1456},"neighbors_timestamps",[1414,1693,1510],{"class":1424},[1414,1695,1696],{"class":1416,"line":1467},[1414,1697,1592],{"class":1424},[1406,1699,1702],{"className":1700,"code":1701,"language":1601},[1599],"timestamp | neighbors\n------------------------------\n...\n100       | (77, 97, 132)\n...\n------------------------------\n",[100,1703,1701],{"__ignoreMap":152},[22,1705,1706,1707,1710,1711,1714,1715,41,1718,1720,1721,1723,1724,1726],{},"In Pathway, you can obtain the same results directly using ",[100,1708,1709],{},"windowby"," with ",[100,1712,1713],{},"intervals_over",".\nBoth the ",[100,1716,1717],{},"interval_join",[100,1719,1713],{}," approach arrive at the same result. The ",[100,1722,1713],{}," is preferred in this case because of its compact syntax. We will use the ",[100,1725,1713],{}," in this tutorial.",[1406,1728,1730],{"className":1408,"code":1729,"language":1410,"meta":152,"style":152},"results = table_a.windowby(\n    table_a.timestamp,\n    window=pw.temporal.intervals_over(\n        at=table_b.timestamp, lower_bound=-50.0, upper_bound=50.0\n    ),\n).reduce(\n    pw.this._pw_window_location,\n    neighbors=pw.reducers.tuple(pw.this.timestamp),\n)\n",[100,1731,1732,1746,1757,1776,1811,1816,1824,1839,1869],{"__ignoreMap":152},[1414,1733,1734,1736,1738,1740,1742,1744],{"class":1416,"line":1417},[1414,1735,1421],{"class":1420},[1414,1737,1425],{"class":1424},[1414,1739,1428],{"class":1420},[1414,1741,46],{"class":1424},[1414,1743,1709],{"class":1433},[1414,1745,1437],{"class":1424},[1414,1747,1748,1751,1753,1755],{"class":1416,"line":153},[1414,1749,1750],{"class":1433},"    table_a",[1414,1752,46],{"class":1424},[1414,1754,1462],{"class":1456},[1414,1756,1445],{"class":1424},[1414,1758,1759,1762,1764,1766,1768,1770,1772,1774],{"class":1416,"line":1448},[1414,1760,1761],{"class":1570},"    window",[1414,1763,1425],{"class":1424},[1414,1765,1576],{"class":1433},[1414,1767,46],{"class":1424},[1414,1769,1490],{"class":1456},[1414,1771,46],{"class":1424},[1414,1773,1713],{"class":1433},[1414,1775,1437],{"class":1424},[1414,1777,1778,1781,1783,1786,1788,1790,1792,1795,1798,1801,1803,1806,1808],{"class":1416,"line":1467},[1414,1779,1780],{"class":1570},"        at",[1414,1782,1425],{"class":1424},[1414,1784,1785],{"class":1433},"table_b",[1414,1787,46],{"class":1424},[1414,1789,1462],{"class":1456},[1414,1791,1504],{"class":1424},[1414,1793,1794],{"class":1570}," lower_bound",[1414,1796,1797],{"class":1424},"=-",[1414,1799,1800],{"class":1500},"50.0",[1414,1802,1504],{"class":1424},[1414,1804,1805],{"class":1570}," upper_bound",[1414,1807,1425],{"class":1424},[1414,1809,1810],{"class":1500},"50.0\n",[1414,1812,1813],{"class":1416,"line":1483},[1414,1814,1815],{"class":1424},"    ),\n",[1414,1817,1818,1820,1822],{"class":1416,"line":1513},[1414,1819,1544],{"class":1424},[1414,1821,1613],{"class":1433},[1414,1823,1437],{"class":1424},[1414,1825,1826,1828,1830,1832,1834,1837],{"class":1416,"line":1541},[1414,1827,1451],{"class":1433},[1414,1829,46],{"class":1424},[1414,1831,1686],{"class":1456},[1414,1833,46],{"class":1424},[1414,1835,1836],{"class":1456},"_pw_window_location",[1414,1838,1445],{"class":1424},[1414,1840,1841,1843,1845,1847,1849,1851,1853,1855,1857,1859,1861,1863,1865,1867],{"class":1416,"line":1552},[1414,1842,1663],{"class":1570},[1414,1844,1425],{"class":1424},[1414,1846,1576],{"class":1433},[1414,1848,46],{"class":1424},[1414,1850,1672],{"class":1456},[1414,1852,46],{"class":1424},[1414,1854,1677],{"class":1433},[1414,1856,1634],{"class":1424},[1414,1858,1576],{"class":1433},[1414,1860,46],{"class":1424},[1414,1862,1686],{"class":1456},[1414,1864,46],{"class":1424},[1414,1866,1462],{"class":1456},[1414,1868,1510],{"class":1424},[1414,1870,1871],{"class":1416,"line":1567},[1414,1872,1592],{"class":1424},[48,1874,1876],{"id":1875},"_2-apply-gaussian-filtering-with-pathway","2. Apply Gaussian Filtering with Pathway",[22,1878,1879],{},"Now that you know how to aggregate the data, let's do some signal processing!",[22,1881,1882,1883,46],{},"Let's start with a simple Gaussian filter in Pathway.\nA Gaussian filter is a linear filter used to reduce the noise of a signal.\nThe filter works by smoothing the signal using a convolution, applying a Gaussian function to a local segment of the signal.\nI will not delve into details: if you are interested you can check out the ",[29,1884,1887],{"href":1885,"rel":1886},"https://en.wikipedia.org/w/index.php?title=Gaussian_filter&oldid=1161463896",[1373],"Wikipedia article",[22,1889,1890,1891,1895,1896,1898,1899,1901,1902,41,1904,1906],{},"To apply a Gaussian filter, you first need to aggregate the local segments: each point is associated with its surrounding counterparts.\nThis aggregation is done using an ",[1892,1893,1894],"em",{},"interval join"," in data streaming processing (note: an interval join is usually done on two different data streams, but it is possible to do a self-interval join. See our ",[29,1897,312],{"href":1387}," for more info about interval joins).\nYou can easily perform a regular interval join using Pathway with ",[100,1900,1717],{},", but in this case you will need to aggregate the points to perform a convolution, so we will use the ",[100,1903,1709],{},[100,1905,1713],{}," syntax we saw earlier.",[22,1908,1909],{},"Before entering the details, let's generate some data to play with.",[1911,1912,1914],"h3",{"id":1913},"the-data","The data",[22,1916,1917],{},"For this example, you will work on static data.\nHowever, in a real use case, your data source might be a data stream.\nFortunately, you won't need to change the code, as Pathway works transparently with static and streaming data: you will only need to update your input connector to connect to your live data.",[22,1919,1920],{},"The raw data needs to be noisy to see the impact of a Gaussian filter.\nLet's generate noisy data by first generating \"perfect data\": data ranging from 0 to 10000 for the timestamps, and the values are generated by adding to an increasing trend some periodic patterns.\nThen, some normally distributed noise is added to both.",[1406,1922,1924],{"className":1408,"code":1923,"language":1410,"meta":152,"style":152},"import matplotlib.pyplot as plt\n",[100,1925,1926],{"__ignoreMap":152},[1414,1927,1928,1932,1935,1937,1940,1943],{"class":1416,"line":1417},[1414,1929,1931],{"class":1930},"s6cf3","import",[1414,1933,1934],{"class":1420}," matplotlib",[1414,1936,46],{"class":1424},[1414,1938,1939],{"class":1456},"pyplot",[1414,1941,1942],{"class":1930}," as",[1414,1944,1945],{"class":1420}," plt\n",[1406,1947,1949],{"className":1408,"code":1948,"language":1410,"meta":152,"style":152},"import numpy as np\n\nnp.random.seed(10)\nnb_points = 1000\n\n# Generate x values\nx = np.linspace(0, 10000, nb_points)\nx_noise = np.random.normal(0, 2, nb_points)\nx = x + x_noise\nx = np.sort(x)\n\n# Generate y values with a globally increasing trend and periodic patterns\ntrend = 0.01 * x  # Globally increasing trend\nperiodic_pattern = 50 * np.sin(2 * np.pi * x / 1000)  # Periodic pattern\nperiodic_pattern2 = 70 * np.sin(2 * np.pi * (x % 10) / 1000 + np.pi)  # Periodic pattern\nnoise = (0.001 * 10000) * np.random.normal(0, 1, nb_points)\n\n# Combine trend and periodic pattern to create y values\ny = trend + periodic_pattern + periodic_pattern2 + noise\n",[100,1950,1951,1964,1969,1991,2001,2005,2011,2043,2076,2091,2111,2116,2122,2142,2191,2255,2301,2306,2312],{"__ignoreMap":152},[1414,1952,1953,1955,1958,1961],{"class":1416,"line":1417},[1414,1954,1931],{"class":1930},[1414,1956,1957],{"class":1420}," numpy ",[1414,1959,1960],{"class":1930},"as",[1414,1962,1963],{"class":1420}," np\n",[1414,1965,1966],{"class":1416,"line":153},[1414,1967,1968],{"emptyLinePlaceholder":176},"\n",[1414,1970,1971,1974,1976,1979,1981,1984,1986,1989],{"class":1416,"line":1448},[1414,1972,1973],{"class":1420},"np",[1414,1975,46],{"class":1424},[1414,1977,1978],{"class":1456},"random",[1414,1980,46],{"class":1424},[1414,1982,1983],{"class":1433},"seed",[1414,1985,1634],{"class":1424},[1414,1987,1988],{"class":1500},"10",[1414,1990,1592],{"class":1424},[1414,1992,1993,1996,1998],{"class":1416,"line":1467},[1414,1994,1995],{"class":1420},"nb_points ",[1414,1997,1425],{"class":1424},[1414,1999,2000],{"class":1500}," 1000\n",[1414,2002,2003],{"class":1416,"line":1483},[1414,2004,1968],{"emptyLinePlaceholder":176},[1414,2006,2007],{"class":1416,"line":1513},[1414,2008,2010],{"class":2009},"saEQR","# Generate x values\n",[1414,2012,2013,2016,2018,2021,2023,2026,2028,2031,2033,2036,2038,2041],{"class":1416,"line":1541},[1414,2014,2015],{"class":1420},"x ",[1414,2017,1425],{"class":1424},[1414,2019,2020],{"class":1420}," np",[1414,2022,46],{"class":1424},[1414,2024,2025],{"class":1433},"linspace",[1414,2027,1634],{"class":1424},[1414,2029,2030],{"class":1500},"0",[1414,2032,1504],{"class":1424},[1414,2034,2035],{"class":1500}," 10000",[1414,2037,1504],{"class":1424},[1414,2039,2040],{"class":1433}," nb_points",[1414,2042,1592],{"class":1424},[1414,2044,2045,2048,2050,2052,2054,2056,2058,2061,2063,2065,2067,2070,2072,2074],{"class":1416,"line":1552},[1414,2046,2047],{"class":1420},"x_noise ",[1414,2049,1425],{"class":1424},[1414,2051,2020],{"class":1420},[1414,2053,46],{"class":1424},[1414,2055,1978],{"class":1456},[1414,2057,46],{"class":1424},[1414,2059,2060],{"class":1433},"normal",[1414,2062,1634],{"class":1424},[1414,2064,2030],{"class":1500},[1414,2066,1504],{"class":1424},[1414,2068,2069],{"class":1500}," 2",[1414,2071,1504],{"class":1424},[1414,2073,2040],{"class":1433},[1414,2075,1592],{"class":1424},[1414,2077,2078,2080,2082,2085,2088],{"class":1416,"line":1567},[1414,2079,2015],{"class":1420},[1414,2081,1425],{"class":1424},[1414,2083,2084],{"class":1420}," x ",[1414,2086,2087],{"class":1424},"+",[1414,2089,2090],{"class":1420}," x_noise\n",[1414,2092,2093,2095,2097,2099,2101,2104,2106,2109],{"class":1416,"line":1589},[1414,2094,2015],{"class":1420},[1414,2096,1425],{"class":1424},[1414,2098,2020],{"class":1420},[1414,2100,46],{"class":1424},[1414,2102,2103],{"class":1433},"sort",[1414,2105,1634],{"class":1424},[1414,2107,2108],{"class":1433},"x",[1414,2110,1592],{"class":1424},[1414,2112,2114],{"class":1416,"line":2113},11,[1414,2115,1968],{"emptyLinePlaceholder":176},[1414,2117,2119],{"class":1416,"line":2118},12,[1414,2120,2121],{"class":2009},"# Generate y values with a globally increasing trend and periodic patterns\n",[1414,2123,2125,2128,2130,2133,2136,2139],{"class":1416,"line":2124},13,[1414,2126,2127],{"class":1420},"trend ",[1414,2129,1425],{"class":1424},[1414,2131,2132],{"class":1500}," 0.01",[1414,2134,2135],{"class":1424}," *",[1414,2137,2138],{"class":1420}," x  ",[1414,2140,2141],{"class":2009},"# Globally increasing trend\n",[1414,2143,2145,2148,2150,2152,2154,2156,2158,2161,2163,2166,2168,2170,2172,2175,2177,2179,2182,2185,2188],{"class":1416,"line":2144},14,[1414,2146,2147],{"class":1420},"periodic_pattern ",[1414,2149,1425],{"class":1424},[1414,2151,1507],{"class":1500},[1414,2153,2135],{"class":1424},[1414,2155,2020],{"class":1420},[1414,2157,46],{"class":1424},[1414,2159,2160],{"class":1433},"sin",[1414,2162,1634],{"class":1424},[1414,2164,2165],{"class":1500},"2",[1414,2167,2135],{"class":1424},[1414,2169,2020],{"class":1433},[1414,2171,46],{"class":1424},[1414,2173,2174],{"class":1456},"pi",[1414,2176,2135],{"class":1424},[1414,2178,2084],{"class":1433},[1414,2180,2181],{"class":1424},"/",[1414,2183,2184],{"class":1500}," 1000",[1414,2186,2187],{"class":1424},")",[1414,2189,2190],{"class":2009},"  # Periodic pattern\n",[1414,2192,2194,2197,2199,2202,2204,2206,2208,2210,2212,2214,2216,2218,2220,2222,2224,2227,2229,2232,2235,2237,2240,2242,2245,2247,2249,2251,2253],{"class":1416,"line":2193},15,[1414,2195,2196],{"class":1420},"periodic_pattern2 ",[1414,2198,1425],{"class":1424},[1414,2200,2201],{"class":1500}," 70",[1414,2203,2135],{"class":1424},[1414,2205,2020],{"class":1420},[1414,2207,46],{"class":1424},[1414,2209,2160],{"class":1433},[1414,2211,1634],{"class":1424},[1414,2213,2165],{"class":1500},[1414,2215,2135],{"class":1424},[1414,2217,2020],{"class":1433},[1414,2219,46],{"class":1424},[1414,2221,2174],{"class":1456},[1414,2223,2135],{"class":1424},[1414,2225,2226],{"class":1424}," (",[1414,2228,2015],{"class":1433},[1414,2230,2231],{"class":1424},"%",[1414,2233,2234],{"class":1500}," 10",[1414,2236,2187],{"class":1424},[1414,2238,2239],{"class":1424}," /",[1414,2241,2184],{"class":1500},[1414,2243,2244],{"class":1424}," +",[1414,2246,2020],{"class":1433},[1414,2248,46],{"class":1424},[1414,2250,2174],{"class":1456},[1414,2252,2187],{"class":1424},[1414,2254,2190],{"class":2009},[1414,2256,2258,2261,2263,2265,2268,2270,2272,2274,2276,2278,2280,2282,2284,2286,2288,2290,2292,2295,2297,2299],{"class":1416,"line":2257},16,[1414,2259,2260],{"class":1420},"noise ",[1414,2262,1425],{"class":1424},[1414,2264,2226],{"class":1424},[1414,2266,2267],{"class":1500},"0.001",[1414,2269,2135],{"class":1424},[1414,2271,2035],{"class":1500},[1414,2273,2187],{"class":1424},[1414,2275,2135],{"class":1424},[1414,2277,2020],{"class":1420},[1414,2279,46],{"class":1424},[1414,2281,1978],{"class":1456},[1414,2283,46],{"class":1424},[1414,2285,2060],{"class":1433},[1414,2287,1634],{"class":1424},[1414,2289,2030],{"class":1500},[1414,2291,1504],{"class":1424},[1414,2293,2294],{"class":1500}," 1",[1414,2296,1504],{"class":1424},[1414,2298,2040],{"class":1433},[1414,2300,1592],{"class":1424},[1414,2302,2304],{"class":1416,"line":2303},17,[1414,2305,1968],{"emptyLinePlaceholder":176},[1414,2307,2309],{"class":1416,"line":2308},18,[1414,2310,2311],{"class":2009},"# Combine trend and periodic pattern to create y values\n",[1414,2313,2315,2318,2320,2323,2325,2328,2330,2333,2335],{"class":1416,"line":2314},19,[1414,2316,2317],{"class":1420},"y ",[1414,2319,1425],{"class":1424},[1414,2321,2322],{"class":1420}," trend ",[1414,2324,2087],{"class":1424},[1414,2326,2327],{"class":1420}," periodic_pattern ",[1414,2329,2087],{"class":1424},[1414,2331,2332],{"class":1420}," periodic_pattern2 ",[1414,2334,2087],{"class":1424},[1414,2336,2337],{"class":1420}," noise\n",[22,2339,2340],{},"Let's plot the signal:",[1406,2342,2344],{"className":1408,"code":2343,"language":1410,"meta":152,"style":152},"def set_params_plot():\n    plt.xlabel(\"time\")\n    plt.ylabel(\"value\")\n    plt.xticks([], [])\n    plt.yticks([], [])\n    plt.title(\"\")\n\n\n# Plot the data points\nplt.subplot(2, 1, 1)\nset_params_plot()\nplt.plot(x, y)\n\nplt.show()\n",[100,2345,2346,2358,2381,2401,2416,2429,2445,2449,2453,2458,2482,2490,2510,2514],{"__ignoreMap":152},[1414,2347,2348,2352,2355],{"class":1416,"line":1417},[1414,2349,2351],{"class":2350},"sJ14y","def",[1414,2353,2354],{"class":1433}," set_params_plot",[1414,2356,2357],{"class":1424},"():\n",[1414,2359,2360,2363,2365,2368,2370,2373,2377,2379],{"class":1416,"line":153},[1414,2361,2362],{"class":1420},"    plt",[1414,2364,46],{"class":1424},[1414,2366,2367],{"class":1433},"xlabel",[1414,2369,1634],{"class":1424},[1414,2371,2372],{"class":1424},"\"",[1414,2374,2376],{"class":2375},"sfyAc","time",[1414,2378,2372],{"class":1424},[1414,2380,1592],{"class":1424},[1414,2382,2383,2385,2387,2390,2392,2394,2397,2399],{"class":1416,"line":1448},[1414,2384,2362],{"class":1420},[1414,2386,46],{"class":1424},[1414,2388,2389],{"class":1433},"ylabel",[1414,2391,1634],{"class":1424},[1414,2393,2372],{"class":1424},[1414,2395,2396],{"class":2375},"value",[1414,2398,2372],{"class":1424},[1414,2400,1592],{"class":1424},[1414,2402,2403,2405,2407,2410,2413],{"class":1416,"line":1467},[1414,2404,2362],{"class":1420},[1414,2406,46],{"class":1424},[1414,2408,2409],{"class":1433},"xticks",[1414,2411,2412],{"class":1424},"([],",[1414,2414,2415],{"class":1424}," [])\n",[1414,2417,2418,2420,2422,2425,2427],{"class":1416,"line":1483},[1414,2419,2362],{"class":1420},[1414,2421,46],{"class":1424},[1414,2423,2424],{"class":1433},"yticks",[1414,2426,2412],{"class":1424},[1414,2428,2415],{"class":1424},[1414,2430,2431,2433,2435,2438,2440,2443],{"class":1416,"line":1513},[1414,2432,2362],{"class":1420},[1414,2434,46],{"class":1424},[1414,2436,2437],{"class":1433},"title",[1414,2439,1634],{"class":1424},[1414,2441,2442],{"class":1424},"\"\"",[1414,2444,1592],{"class":1424},[1414,2446,2447],{"class":1416,"line":1541},[1414,2448,1968],{"emptyLinePlaceholder":176},[1414,2450,2451],{"class":1416,"line":1552},[1414,2452,1968],{"emptyLinePlaceholder":176},[1414,2454,2455],{"class":1416,"line":1567},[1414,2456,2457],{"class":2009},"# Plot the data points\n",[1414,2459,2460,2463,2465,2468,2470,2472,2474,2476,2478,2480],{"class":1416,"line":1589},[1414,2461,2462],{"class":1420},"plt",[1414,2464,46],{"class":1424},[1414,2466,2467],{"class":1433},"subplot",[1414,2469,1634],{"class":1424},[1414,2471,2165],{"class":1500},[1414,2473,1504],{"class":1424},[1414,2475,2294],{"class":1500},[1414,2477,1504],{"class":1424},[1414,2479,2294],{"class":1500},[1414,2481,1592],{"class":1424},[1414,2483,2484,2487],{"class":1416,"line":2113},[1414,2485,2486],{"class":1433},"set_params_plot",[1414,2488,2489],{"class":1424},"()\n",[1414,2491,2492,2494,2496,2499,2501,2503,2505,2508],{"class":1416,"line":2118},[1414,2493,2462],{"class":1420},[1414,2495,46],{"class":1424},[1414,2497,2498],{"class":1433},"plot",[1414,2500,1634],{"class":1424},[1414,2502,2108],{"class":1433},[1414,2504,1504],{"class":1424},[1414,2506,2507],{"class":1433}," y",[1414,2509,1592],{"class":1424},[1414,2511,2512],{"class":1416,"line":2124},[1414,2513,1968],{"emptyLinePlaceholder":176},[1414,2515,2516,2518,2520,2523],{"class":1416,"line":2144},[1414,2517,2462],{"class":1420},[1414,2519,46],{"class":1424},[1414,2521,2522],{"class":1433},"show",[1414,2524,2489],{"class":1424},[22,2526,2527],{},[1398,2528],{"alt":2529,"src":2530},"png","/assets/autogenerated/../pathway/docs/2.developers/7.templates/ETL/.interval_over_gaussian_filter/article_5_0.png",[22,2532,2533],{},"Now you can convert this data to Pandas and load it to Pathway",[1406,2535,2537],{"className":1408,"code":2536,"language":1410,"meta":152,"style":152},"import pandas as pd\n",[100,2538,2539],{"__ignoreMap":152},[1414,2540,2541,2543,2546,2548],{"class":1416,"line":1417},[1414,2542,1931],{"class":1930},[1414,2544,2545],{"class":1420}," pandas ",[1414,2547,1960],{"class":1930},[1414,2549,2550],{"class":1420}," pd\n",[1406,2552,2554],{"className":1408,"code":2553,"language":1410,"meta":152,"style":152},"import pathway as pw\n\n\n\ndef load_to_pathway(x, y):\n    time_series_pd = pd.DataFrame({\"x\": x, \"y\": y})\n    return pw.debug.table_from_pandas(time_series_pd)\n\n\ntime_series = load_to_pathway(x, y)\n",[100,2555,2556,2568,2572,2576,2580,2598,2645,2669,2673,2677],{"__ignoreMap":152},[1414,2557,2558,2560,2563,2565],{"class":1416,"line":1417},[1414,2559,1931],{"class":1930},[1414,2561,2562],{"class":1420}," pathway ",[1414,2564,1960],{"class":1930},[1414,2566,2567],{"class":1420}," pw\n",[1414,2569,2570],{"class":1416,"line":153},[1414,2571,1968],{"emptyLinePlaceholder":176},[1414,2573,2574],{"class":1416,"line":1448},[1414,2575,1968],{"emptyLinePlaceholder":176},[1414,2577,2578],{"class":1416,"line":1467},[1414,2579,1968],{"emptyLinePlaceholder":176},[1414,2581,2582,2584,2587,2589,2591,2593,2595],{"class":1416,"line":1483},[1414,2583,2351],{"class":2350},[1414,2585,2586],{"class":1433}," load_to_pathway",[1414,2588,1634],{"class":1424},[1414,2590,2108],{"class":1570},[1414,2592,1504],{"class":1424},[1414,2594,2507],{"class":1570},[1414,2596,2597],{"class":1424},"):\n",[1414,2599,2600,2603,2605,2608,2610,2613,2616,2618,2620,2622,2625,2628,2630,2633,2636,2638,2640,2642],{"class":1416,"line":1513},[1414,2601,2602],{"class":1420},"    time_series_pd ",[1414,2604,1425],{"class":1424},[1414,2606,2607],{"class":1420}," pd",[1414,2609,46],{"class":1424},[1414,2611,2612],{"class":1433},"DataFrame",[1414,2614,2615],{"class":1424},"({",[1414,2617,2372],{"class":1424},[1414,2619,2108],{"class":2375},[1414,2621,2372],{"class":1424},[1414,2623,2624],{"class":1424},":",[1414,2626,2627],{"class":1433}," x",[1414,2629,1504],{"class":1424},[1414,2631,2632],{"class":1424}," \"",[1414,2634,2635],{"class":2375},"y",[1414,2637,2372],{"class":1424},[1414,2639,2624],{"class":1424},[1414,2641,2507],{"class":1433},[1414,2643,2644],{"class":1424},"})\n",[1414,2646,2647,2650,2652,2654,2657,2659,2662,2664,2667],{"class":1416,"line":1541},[1414,2648,2649],{"class":1930},"    return",[1414,2651,1529],{"class":1420},[1414,2653,46],{"class":1424},[1414,2655,2656],{"class":1456},"debug",[1414,2658,46],{"class":1424},[1414,2660,2661],{"class":1433},"table_from_pandas",[1414,2663,1634],{"class":1424},[1414,2665,2666],{"class":1433},"time_series_pd",[1414,2668,1592],{"class":1424},[1414,2670,2671],{"class":1416,"line":1552},[1414,2672,1968],{"emptyLinePlaceholder":176},[1414,2674,2675],{"class":1416,"line":1567},[1414,2676,1968],{"emptyLinePlaceholder":176},[1414,2678,2679,2682,2684,2686,2688,2690,2692,2694],{"class":1416,"line":1589},[1414,2680,2681],{"class":1420},"time_series ",[1414,2683,1425],{"class":1424},[1414,2685,2586],{"class":1433},[1414,2687,1634],{"class":1424},[1414,2689,2108],{"class":1433},[1414,2691,1504],{"class":1424},[1414,2693,2507],{"class":1433},[1414,2695,1592],{"class":1424},[1406,2697,2699],{"className":1408,"code":2698,"language":1410,"meta":152,"style":152},"time_series.typehints()\n",[100,2700,2701],{"__ignoreMap":152},[1414,2702,2703,2706,2708,2711],{"class":1416,"line":1417},[1414,2704,2705],{"class":1420},"time_series",[1414,2707,46],{"class":1424},[1414,2709,2710],{"class":1433},"typehints",[1414,2712,2489],{"class":1424},[1406,2714,2717],{"className":2715,"code":2716,"language":1601},[1599],"mappingproxy({'x': float, 'y': float})\n",[100,2718,2716],{"__ignoreMap":152},[22,2720,2721],{},"And that's it! Now, let's see how to build and apply a Gaussian filter in Pathway.",[1911,2723,2725,2726],{"id":2724},"computing-aggregations-with-windowby","Computing aggregations with ",[100,2727,1709],{},[22,2729,2730],{},"You want to obtain an aggregation of the closest points for each point.\nWith streaming data, you do not have any assumption on the distribution of surrounding points.\nYou can ask for a fixed number of previous/next points, but you'd take the risk to obtain points with very different timestamps.\nSo the solution is to ask for the data points in an interval around your point.",[22,2732,2733,2734,2736],{},"To do so, you can use ",[100,2735,1709],{},", that takes three arguments:",[139,2738,2739,2744,2750],{},[66,2740,2741,2742,46],{},"The column used for windowing: the timestamp in the column ",[100,2743,2108],{},[66,2745,2746,2747,2749],{},"The window: in your case you will use an ",[100,2748,1713],{}," window.",[66,2751,2752],{},"The optional sharding: not relevant here.",[22,2754,2755,2756,2758,2759,2762,2763,2766,2767,2770],{},"For ",[100,2757,1713],{},", you need to set the parameter ",[100,2760,2761],{},"at",", defining the column of times for which windows are to be created and the ",[100,2764,2765],{},"lower_bound"," and the ",[100,2768,2769],{},"upper_bound"," of the interval.\nLet's try with an interval of length 100:",[1406,2772,2774],{"className":1408,"code":2773,"language":1410,"meta":152,"style":152},"points_within_50 = time_series.windowby(\n    time_series.x,\n    window=pw.temporal.intervals_over(\n        at=time_series.x, lower_bound=-50.0, upper_bound=50.0, is_outer=False\n    ),\n).reduce(\n    pw.this._pw_window_location,\n    xs=pw.reducers.ndarray(pw.this.x),\n    ys=pw.reducers.ndarray(pw.this.y),\n)\n",[100,2775,2776,2792,2803,2821,2857,2861,2869,2883,2915,2946],{"__ignoreMap":152},[1414,2777,2778,2781,2783,2786,2788,2790],{"class":1416,"line":1417},[1414,2779,2780],{"class":1420},"points_within_50 ",[1414,2782,1425],{"class":1424},[1414,2784,2785],{"class":1420}," time_series",[1414,2787,46],{"class":1424},[1414,2789,1709],{"class":1433},[1414,2791,1437],{"class":1424},[1414,2793,2794,2797,2799,2801],{"class":1416,"line":153},[1414,2795,2796],{"class":1433},"    time_series",[1414,2798,46],{"class":1424},[1414,2800,2108],{"class":1456},[1414,2802,1445],{"class":1424},[1414,2804,2805,2807,2809,2811,2813,2815,2817,2819],{"class":1416,"line":1448},[1414,2806,1761],{"class":1570},[1414,2808,1425],{"class":1424},[1414,2810,1576],{"class":1433},[1414,2812,46],{"class":1424},[1414,2814,1490],{"class":1456},[1414,2816,46],{"class":1424},[1414,2818,1713],{"class":1433},[1414,2820,1437],{"class":1424},[1414,2822,2823,2825,2827,2829,2831,2833,2835,2837,2839,2841,2843,2845,2847,2849,2851,2854],{"class":1416,"line":1467},[1414,2824,1780],{"class":1570},[1414,2826,1425],{"class":1424},[1414,2828,2705],{"class":1433},[1414,2830,46],{"class":1424},[1414,2832,2108],{"class":1456},[1414,2834,1504],{"class":1424},[1414,2836,1794],{"class":1570},[1414,2838,1797],{"class":1424},[1414,2840,1800],{"class":1500},[1414,2842,1504],{"class":1424},[1414,2844,1805],{"class":1570},[1414,2846,1425],{"class":1424},[1414,2848,1800],{"class":1500},[1414,2850,1504],{"class":1424},[1414,2852,2853],{"class":1570}," is_outer",[1414,2855,2856],{"class":1424},"=False\n",[1414,2858,2859],{"class":1416,"line":1483},[1414,2860,1815],{"class":1424},[1414,2862,2863,2865,2867],{"class":1416,"line":1513},[1414,2864,1544],{"class":1424},[1414,2866,1613],{"class":1433},[1414,2868,1437],{"class":1424},[1414,2870,2871,2873,2875,2877,2879,2881],{"class":1416,"line":1541},[1414,2872,1451],{"class":1433},[1414,2874,46],{"class":1424},[1414,2876,1686],{"class":1456},[1414,2878,46],{"class":1424},[1414,2880,1836],{"class":1456},[1414,2882,1445],{"class":1424},[1414,2884,2885,2888,2890,2892,2894,2896,2898,2901,2903,2905,2907,2909,2911,2913],{"class":1416,"line":1552},[1414,2886,2887],{"class":1570},"    xs",[1414,2889,1425],{"class":1424},[1414,2891,1576],{"class":1433},[1414,2893,46],{"class":1424},[1414,2895,1672],{"class":1456},[1414,2897,46],{"class":1424},[1414,2899,2900],{"class":1433},"ndarray",[1414,2902,1634],{"class":1424},[1414,2904,1576],{"class":1433},[1414,2906,46],{"class":1424},[1414,2908,1686],{"class":1456},[1414,2910,46],{"class":1424},[1414,2912,2108],{"class":1456},[1414,2914,1510],{"class":1424},[1414,2916,2917,2920,2922,2924,2926,2928,2930,2932,2934,2936,2938,2940,2942,2944],{"class":1416,"line":1567},[1414,2918,2919],{"class":1570},"    ys",[1414,2921,1425],{"class":1424},[1414,2923,1576],{"class":1433},[1414,2925,46],{"class":1424},[1414,2927,1672],{"class":1456},[1414,2929,46],{"class":1424},[1414,2931,2900],{"class":1433},[1414,2933,1634],{"class":1424},[1414,2935,1576],{"class":1433},[1414,2937,46],{"class":1424},[1414,2939,1686],{"class":1456},[1414,2941,46],{"class":1424},[1414,2943,2635],{"class":1456},[1414,2945,1510],{"class":1424},[1414,2947,2948],{"class":1416,"line":1589},[1414,2949,1592],{"class":1424},[1911,2951,2953],{"id":2952},"gaussian-filtering-in-python-as-a-udf","Gaussian Filtering in Python as a UDF",[22,2955,2956,2957,2960,2961,2963,2964,2967],{},"Now that you have the intervals, you need the Gaussian filter.\nIn Pathway, you can easily define a User-Defined Function (UDF) with the ",[100,2958,2959],{},"@pw.udf"," annotation.\nTo perform a Gaussian filter, you need the data points gathered by the ",[100,2962,1709],{}," and the associated time (called ",[100,2965,2966],{},"window_location",").\nThe times and the values are kept in separate columns, so we takes two different arrays as input:",[1406,2969,2971],{"className":1408,"code":2970,"language":1410,"meta":152,"style":152},"import scipy.stats\n\n\n# UDF for Gaussian filter\n@pw.udf\ndef gaussian_filter(\n    times: np.ndarray,\n    values: np.ndarray,\n    window_location,\n) -> float:\n    stdev = (max(times) - min(times)) / 2\n    gaussian_distribution = scipy.stats.norm(window_location, stdev)\n\n    coefficients = gaussian_distribution.pdf(times)\n    normalized_coefficients = coefficients / sum(coefficients)\n    return np.dot(values, normalized_coefficients)\n",[100,2972,2973,2985,2989,2993,2998,3010,3019,3034,3049,3056,3070,3107,3137,3141,3162,3184],{"__ignoreMap":152},[1414,2974,2975,2977,2980,2982],{"class":1416,"line":1417},[1414,2976,1931],{"class":1930},[1414,2978,2979],{"class":1420}," scipy",[1414,2981,46],{"class":1424},[1414,2983,2984],{"class":1456},"stats\n",[1414,2986,2987],{"class":1416,"line":153},[1414,2988,1968],{"emptyLinePlaceholder":176},[1414,2990,2991],{"class":1416,"line":1448},[1414,2992,1968],{"emptyLinePlaceholder":176},[1414,2994,2995],{"class":1416,"line":1467},[1414,2996,2997],{"class":2009},"# UDF for Gaussian filter\n",[1414,2999,3000,3003,3005,3007],{"class":1416,"line":1483},[1414,3001,3002],{"class":1424},"@",[1414,3004,1576],{"class":1433},[1414,3006,46],{"class":1424},[1414,3008,3009],{"class":1433},"udf\n",[1414,3011,3012,3014,3017],{"class":1416,"line":1513},[1414,3013,2351],{"class":2350},[1414,3015,3016],{"class":1433}," gaussian_filter",[1414,3018,1437],{"class":1424},[1414,3020,3021,3024,3026,3028,3030,3032],{"class":1416,"line":1541},[1414,3022,3023],{"class":1570},"    times",[1414,3025,2624],{"class":1424},[1414,3027,2020],{"class":1420},[1414,3029,46],{"class":1424},[1414,3031,2900],{"class":1456},[1414,3033,1445],{"class":1424},[1414,3035,3036,3039,3041,3043,3045,3047],{"class":1416,"line":1552},[1414,3037,3038],{"class":1570},"    values",[1414,3040,2624],{"class":1424},[1414,3042,2020],{"class":1420},[1414,3044,46],{"class":1424},[1414,3046,2900],{"class":1456},[1414,3048,1445],{"class":1424},[1414,3050,3051,3054],{"class":1416,"line":1567},[1414,3052,3053],{"class":1570},"    window_location",[1414,3055,1445],{"class":1424},[1414,3057,3058,3060,3063,3067],{"class":1416,"line":1589},[1414,3059,2187],{"class":1424},[1414,3061,3062],{"class":1424}," ->",[1414,3064,3066],{"class":3065},"s5Dmg"," float",[1414,3068,3069],{"class":1424},":\n",[1414,3071,3072,3075,3077,3079,3082,3084,3087,3089,3092,3095,3097,3099,3102,3104],{"class":1416,"line":2113},[1414,3073,3074],{"class":1420},"    stdev ",[1414,3076,1425],{"class":1424},[1414,3078,2226],{"class":1424},[1414,3080,3081],{"class":1433},"max",[1414,3083,1634],{"class":1424},[1414,3085,3086],{"class":1433},"times",[1414,3088,2187],{"class":1424},[1414,3090,3091],{"class":1424}," -",[1414,3093,3094],{"class":1433}," min",[1414,3096,1634],{"class":1424},[1414,3098,3086],{"class":1433},[1414,3100,3101],{"class":1424},"))",[1414,3103,2239],{"class":1424},[1414,3105,3106],{"class":1500}," 2\n",[1414,3108,3109,3112,3114,3116,3118,3121,3123,3126,3128,3130,3132,3135],{"class":1416,"line":2118},[1414,3110,3111],{"class":1420},"    gaussian_distribution ",[1414,3113,1425],{"class":1424},[1414,3115,2979],{"class":1420},[1414,3117,46],{"class":1424},[1414,3119,3120],{"class":1456},"stats",[1414,3122,46],{"class":1424},[1414,3124,3125],{"class":1433},"norm",[1414,3127,1634],{"class":1424},[1414,3129,2966],{"class":1433},[1414,3131,1504],{"class":1424},[1414,3133,3134],{"class":1433}," stdev",[1414,3136,1592],{"class":1424},[1414,3138,3139],{"class":1416,"line":2124},[1414,3140,1968],{"emptyLinePlaceholder":176},[1414,3142,3143,3146,3148,3151,3153,3156,3158,3160],{"class":1416,"line":2144},[1414,3144,3145],{"class":1420},"    coefficients ",[1414,3147,1425],{"class":1424},[1414,3149,3150],{"class":1420}," gaussian_distribution",[1414,3152,46],{"class":1424},[1414,3154,3155],{"class":1433},"pdf",[1414,3157,1634],{"class":1424},[1414,3159,3086],{"class":1433},[1414,3161,1592],{"class":1424},[1414,3163,3164,3167,3169,3172,3174,3177,3179,3182],{"class":1416,"line":2193},[1414,3165,3166],{"class":1420},"    normalized_coefficients ",[1414,3168,1425],{"class":1424},[1414,3170,3171],{"class":1420}," coefficients ",[1414,3173,2181],{"class":1424},[1414,3175,3176],{"class":1433}," sum",[1414,3178,1634],{"class":1424},[1414,3180,3181],{"class":1433},"coefficients",[1414,3183,1592],{"class":1424},[1414,3185,3186,3188,3190,3192,3195,3197,3200,3202,3205],{"class":1416,"line":2257},[1414,3187,2649],{"class":1930},[1414,3189,2020],{"class":1420},[1414,3191,46],{"class":1424},[1414,3193,3194],{"class":1433},"dot",[1414,3196,1634],{"class":1424},[1414,3198,3199],{"class":1433},"values",[1414,3201,1504],{"class":1424},[1414,3203,3204],{"class":1433}," normalized_coefficients",[1414,3206,1592],{"class":1424},[22,3208,3209,3210,2624],{},"With the UDF, you can directly apply the filter to each interval, without using ",[100,3211,3212],{},"pw.apply",[1406,3214,3216],{"className":1408,"code":3215,"language":1410,"meta":152,"style":152},"# Applying the Gaussian filter\ntime_series_filtered = points_within_50.select(\n    x=pw.this._pw_window_location,\n    y=gaussian_filter(pw.this.xs, pw.this.ys, pw.this._pw_window_location),\n)\n",[100,3217,3218,3223,3239,3258,3308],{"__ignoreMap":152},[1414,3219,3220],{"class":1416,"line":1417},[1414,3221,3222],{"class":2009},"# Applying the Gaussian filter\n",[1414,3224,3225,3228,3230,3233,3235,3237],{"class":1416,"line":153},[1414,3226,3227],{"class":1420},"time_series_filtered ",[1414,3229,1425],{"class":1424},[1414,3231,3232],{"class":1420}," points_within_50",[1414,3234,46],{"class":1424},[1414,3236,1547],{"class":1433},[1414,3238,1437],{"class":1424},[1414,3240,3241,3244,3246,3248,3250,3252,3254,3256],{"class":1416,"line":1448},[1414,3242,3243],{"class":1570},"    x",[1414,3245,1425],{"class":1424},[1414,3247,1576],{"class":1433},[1414,3249,46],{"class":1424},[1414,3251,1686],{"class":1456},[1414,3253,46],{"class":1424},[1414,3255,1836],{"class":1456},[1414,3257,1445],{"class":1424},[1414,3259,3260,3263,3265,3268,3270,3272,3274,3276,3278,3281,3283,3285,3287,3289,3291,3294,3296,3298,3300,3302,3304,3306],{"class":1416,"line":1467},[1414,3261,3262],{"class":1570},"    y",[1414,3264,1425],{"class":1424},[1414,3266,3267],{"class":1433},"gaussian_filter",[1414,3269,1634],{"class":1424},[1414,3271,1576],{"class":1433},[1414,3273,46],{"class":1424},[1414,3275,1686],{"class":1456},[1414,3277,46],{"class":1424},[1414,3279,3280],{"class":1456},"xs",[1414,3282,1504],{"class":1424},[1414,3284,1529],{"class":1433},[1414,3286,46],{"class":1424},[1414,3288,1686],{"class":1456},[1414,3290,46],{"class":1424},[1414,3292,3293],{"class":1456},"ys",[1414,3295,1504],{"class":1424},[1414,3297,1529],{"class":1433},[1414,3299,46],{"class":1424},[1414,3301,1686],{"class":1456},[1414,3303,46],{"class":1424},[1414,3305,1836],{"class":1456},[1414,3307,1510],{"class":1424},[1414,3309,3310],{"class":1416,"line":1483},[1414,3311,1592],{"class":1424},[1911,3313,3315],{"id":3314},"displaying-the-output","Displaying the output",[22,3317,3318,3319,3323],{},"Let's plot the results.\nLet's first output the data in Pandas and then plot the timeseries.\nIn practice you'd use a ",[29,3320,3322],{"href":3321},"/developers/user-guide/connect/pathway-connectors","output connector"," to output the results to Kafka or Postgres.",[1406,3325,3327],{"className":1408,"code":3326,"language":1410,"meta":152,"style":152},"time_series_filtered_pd = pw.debug.table_to_pandas(time_series_filtered)\ntime_series_filtered_pd = time_series_filtered_pd.sort_values(by=[\"x\"])\n",[100,3328,3329,3354],{"__ignoreMap":152},[1414,3330,3331,3334,3336,3338,3340,3342,3344,3347,3349,3352],{"class":1416,"line":1417},[1414,3332,3333],{"class":1420},"time_series_filtered_pd ",[1414,3335,1425],{"class":1424},[1414,3337,1529],{"class":1420},[1414,3339,46],{"class":1424},[1414,3341,2656],{"class":1456},[1414,3343,46],{"class":1424},[1414,3345,3346],{"class":1433},"table_to_pandas",[1414,3348,1634],{"class":1424},[1414,3350,3351],{"class":1433},"time_series_filtered",[1414,3353,1592],{"class":1424},[1414,3355,3356,3358,3360,3363,3365,3368,3370,3373,3376,3378,3380,3382],{"class":1416,"line":153},[1414,3357,3333],{"class":1420},[1414,3359,1425],{"class":1424},[1414,3361,3362],{"class":1420}," time_series_filtered_pd",[1414,3364,46],{"class":1424},[1414,3366,3367],{"class":1433},"sort_values",[1414,3369,1634],{"class":1424},[1414,3371,3372],{"class":1570},"by",[1414,3374,3375],{"class":1424},"=[",[1414,3377,2372],{"class":1424},[1414,3379,2108],{"class":2375},[1414,3381,2372],{"class":1424},[1414,3383,3384],{"class":1424},"])\n",[1406,3386,3388],{"className":1408,"code":3387,"language":1410,"meta":152,"style":152},"plt.subplot(2, 1, 1)\nplt.plot(x, y)\nplt.plot(time_series_filtered_pd.x, time_series_filtered_pd.y)\nset_params_plot()\n\nplt.subplot(2, 1, 2)\nplt.plot(time_series_filtered_pd.x, time_series_filtered_pd.y, color=\"#ff7f0e\")\nset_params_plot()\n\nplt.tight_layout()\nplt.show()\n",[100,3389,3390,3412,3430,3457,3463,3467,3489,3529,3535,3539,3550],{"__ignoreMap":152},[1414,3391,3392,3394,3396,3398,3400,3402,3404,3406,3408,3410],{"class":1416,"line":1417},[1414,3393,2462],{"class":1420},[1414,3395,46],{"class":1424},[1414,3397,2467],{"class":1433},[1414,3399,1634],{"class":1424},[1414,3401,2165],{"class":1500},[1414,3403,1504],{"class":1424},[1414,3405,2294],{"class":1500},[1414,3407,1504],{"class":1424},[1414,3409,2294],{"class":1500},[1414,3411,1592],{"class":1424},[1414,3413,3414,3416,3418,3420,3422,3424,3426,3428],{"class":1416,"line":153},[1414,3415,2462],{"class":1420},[1414,3417,46],{"class":1424},[1414,3419,2498],{"class":1433},[1414,3421,1634],{"class":1424},[1414,3423,2108],{"class":1433},[1414,3425,1504],{"class":1424},[1414,3427,2507],{"class":1433},[1414,3429,1592],{"class":1424},[1414,3431,3432,3434,3436,3438,3440,3443,3445,3447,3449,3451,3453,3455],{"class":1416,"line":1448},[1414,3433,2462],{"class":1420},[1414,3435,46],{"class":1424},[1414,3437,2498],{"class":1433},[1414,3439,1634],{"class":1424},[1414,3441,3442],{"class":1433},"time_series_filtered_pd",[1414,3444,46],{"class":1424},[1414,3446,2108],{"class":1456},[1414,3448,1504],{"class":1424},[1414,3450,3362],{"class":1433},[1414,3452,46],{"class":1424},[1414,3454,2635],{"class":1456},[1414,3456,1592],{"class":1424},[1414,3458,3459,3461],{"class":1416,"line":1467},[1414,3460,2486],{"class":1433},[1414,3462,2489],{"class":1424},[1414,3464,3465],{"class":1416,"line":1483},[1414,3466,1968],{"emptyLinePlaceholder":176},[1414,3468,3469,3471,3473,3475,3477,3479,3481,3483,3485,3487],{"class":1416,"line":1513},[1414,3470,2462],{"class":1420},[1414,3472,46],{"class":1424},[1414,3474,2467],{"class":1433},[1414,3476,1634],{"class":1424},[1414,3478,2165],{"class":1500},[1414,3480,1504],{"class":1424},[1414,3482,2294],{"class":1500},[1414,3484,1504],{"class":1424},[1414,3486,2069],{"class":1500},[1414,3488,1592],{"class":1424},[1414,3490,3491,3493,3495,3497,3499,3501,3503,3505,3507,3509,3511,3513,3515,3518,3520,3522,3525,3527],{"class":1416,"line":1541},[1414,3492,2462],{"class":1420},[1414,3494,46],{"class":1424},[1414,3496,2498],{"class":1433},[1414,3498,1634],{"class":1424},[1414,3500,3442],{"class":1433},[1414,3502,46],{"class":1424},[1414,3504,2108],{"class":1456},[1414,3506,1504],{"class":1424},[1414,3508,3362],{"class":1433},[1414,3510,46],{"class":1424},[1414,3512,2635],{"class":1456},[1414,3514,1504],{"class":1424},[1414,3516,3517],{"class":1570}," color",[1414,3519,1425],{"class":1424},[1414,3521,2372],{"class":1424},[1414,3523,3524],{"class":2375},"#ff7f0e",[1414,3526,2372],{"class":1424},[1414,3528,1592],{"class":1424},[1414,3530,3531,3533],{"class":1416,"line":1552},[1414,3532,2486],{"class":1433},[1414,3534,2489],{"class":1424},[1414,3536,3537],{"class":1416,"line":1567},[1414,3538,1968],{"emptyLinePlaceholder":176},[1414,3540,3541,3543,3545,3548],{"class":1416,"line":1589},[1414,3542,2462],{"class":1420},[1414,3544,46],{"class":1424},[1414,3546,3547],{"class":1433},"tight_layout",[1414,3549,2489],{"class":1424},[1414,3551,3552,3554,3556,3558],{"class":1416,"line":2113},[1414,3553,2462],{"class":1420},[1414,3555,46],{"class":1424},[1414,3557,2522],{"class":1433},[1414,3559,2489],{"class":1424},[22,3561,3562],{},[1398,3563],{"alt":2529,"src":3564},"/assets/autogenerated/../pathway/docs/2.developers/7.templates/ETL/.interval_over_gaussian_filter/article_18_0.png",[22,3566,3567],{},"See how smooth the new plot is compared to the noisy raw data!",[48,3569,3571],{"id":3570},"gaussian-filtering-in-python-handling-late-and-out-of-order-data-points","Gaussian Filtering in Python: Handling Late and Out-of-Order data points",[22,3573,3574],{},"Now that your pipeline is ready, Pathway will manage late and out-of-order points by updating its results whenever new (or late, in this case) data points come into the system.",[22,3576,3577],{},"But what does it mean exactly? What happens when late points arrive?",[22,3579,3580,3581],{},"As a reminder, a standard DSP system will retain the data until all the data is ready to apply the Gaussian filter.\n",[69,3582,3583],{},"In a streaming system, you cannot afford to wait for data points for an undetermined time: some data points may be lost and never arrive!\nPathway computes an output with whatever data is available then and revises the result whenever new data points arrive.",[22,3585,3586],{},"Let's see the impact on late data points in our example by delaying the arrival of several points.",[22,3588,3589],{},"For clarity, let's focus on a small sample of the data:",[1406,3591,3593],{"className":1408,"code":3592,"language":1410,"meta":152,"style":152},"sample_size = 20\nx_sample = x[0:sample_size]\ny_sample = y[0:sample_size]\ntime_series_filtered_x_sample = time_series_filtered_pd.x.tolist()[0:sample_size]\ntime_series_filtered_y_sample = time_series_filtered_pd.y.tolist()[0:sample_size]\n\n# Plot the data points\nplt.subplot(2, 1, 1)\nset_params_plot()\nplt.plot(x_sample, y_sample, \"x-\", label=\"Data sample\")\nplt.plot(\n    time_series_filtered_x_sample,\n    time_series_filtered_y_sample,\n    \"x-\",\n    label=\"Smoothed signal\",\n)\nplt.legend()\n\nplt.show()\n",[100,3594,3595,3605,3627,3646,3675,3702,3706,3710,3732,3738,3781,3791,3798,3805,3816,3832,3836,3847,3851],{"__ignoreMap":152},[1414,3596,3597,3600,3602],{"class":1416,"line":1417},[1414,3598,3599],{"class":1420},"sample_size ",[1414,3601,1425],{"class":1424},[1414,3603,3604],{"class":1500}," 20\n",[1414,3606,3607,3610,3612,3614,3617,3619,3621,3624],{"class":1416,"line":153},[1414,3608,3609],{"class":1420},"x_sample ",[1414,3611,1425],{"class":1424},[1414,3613,2627],{"class":1420},[1414,3615,3616],{"class":1424},"[",[1414,3618,2030],{"class":1500},[1414,3620,2624],{"class":1424},[1414,3622,3623],{"class":1420},"sample_size",[1414,3625,3626],{"class":1424},"]\n",[1414,3628,3629,3632,3634,3636,3638,3640,3642,3644],{"class":1416,"line":1448},[1414,3630,3631],{"class":1420},"y_sample ",[1414,3633,1425],{"class":1424},[1414,3635,2507],{"class":1420},[1414,3637,3616],{"class":1424},[1414,3639,2030],{"class":1500},[1414,3641,2624],{"class":1424},[1414,3643,3623],{"class":1420},[1414,3645,3626],{"class":1424},[1414,3647,3648,3651,3653,3655,3657,3659,3661,3664,3667,3669,3671,3673],{"class":1416,"line":1467},[1414,3649,3650],{"class":1420},"time_series_filtered_x_sample ",[1414,3652,1425],{"class":1424},[1414,3654,3362],{"class":1420},[1414,3656,46],{"class":1424},[1414,3658,2108],{"class":1456},[1414,3660,46],{"class":1424},[1414,3662,3663],{"class":1433},"tolist",[1414,3665,3666],{"class":1424},"()[",[1414,3668,2030],{"class":1500},[1414,3670,2624],{"class":1424},[1414,3672,3623],{"class":1420},[1414,3674,3626],{"class":1424},[1414,3676,3677,3680,3682,3684,3686,3688,3690,3692,3694,3696,3698,3700],{"class":1416,"line":1483},[1414,3678,3679],{"class":1420},"time_series_filtered_y_sample ",[1414,3681,1425],{"class":1424},[1414,3683,3362],{"class":1420},[1414,3685,46],{"class":1424},[1414,3687,2635],{"class":1456},[1414,3689,46],{"class":1424},[1414,3691,3663],{"class":1433},[1414,3693,3666],{"class":1424},[1414,3695,2030],{"class":1500},[1414,3697,2624],{"class":1424},[1414,3699,3623],{"class":1420},[1414,3701,3626],{"class":1424},[1414,3703,3704],{"class":1416,"line":1513},[1414,3705,1968],{"emptyLinePlaceholder":176},[1414,3707,3708],{"class":1416,"line":1541},[1414,3709,2457],{"class":2009},[1414,3711,3712,3714,3716,3718,3720,3722,3724,3726,3728,3730],{"class":1416,"line":1552},[1414,3713,2462],{"class":1420},[1414,3715,46],{"class":1424},[1414,3717,2467],{"class":1433},[1414,3719,1634],{"class":1424},[1414,3721,2165],{"class":1500},[1414,3723,1504],{"class":1424},[1414,3725,2294],{"class":1500},[1414,3727,1504],{"class":1424},[1414,3729,2294],{"class":1500},[1414,3731,1592],{"class":1424},[1414,3733,3734,3736],{"class":1416,"line":1567},[1414,3735,2486],{"class":1433},[1414,3737,2489],{"class":1424},[1414,3739,3740,3742,3744,3746,3748,3751,3753,3756,3758,3760,3763,3765,3767,3770,3772,3774,3777,3779],{"class":1416,"line":1589},[1414,3741,2462],{"class":1420},[1414,3743,46],{"class":1424},[1414,3745,2498],{"class":1433},[1414,3747,1634],{"class":1424},[1414,3749,3750],{"class":1433},"x_sample",[1414,3752,1504],{"class":1424},[1414,3754,3755],{"class":1433}," y_sample",[1414,3757,1504],{"class":1424},[1414,3759,2632],{"class":1424},[1414,3761,3762],{"class":2375},"x-",[1414,3764,2372],{"class":1424},[1414,3766,1504],{"class":1424},[1414,3768,3769],{"class":1570}," label",[1414,3771,1425],{"class":1424},[1414,3773,2372],{"class":1424},[1414,3775,3776],{"class":2375},"Data sample",[1414,3778,2372],{"class":1424},[1414,3780,1592],{"class":1424},[1414,3782,3783,3785,3787,3789],{"class":1416,"line":2113},[1414,3784,2462],{"class":1420},[1414,3786,46],{"class":1424},[1414,3788,2498],{"class":1433},[1414,3790,1437],{"class":1424},[1414,3792,3793,3796],{"class":1416,"line":2118},[1414,3794,3795],{"class":1433},"    time_series_filtered_x_sample",[1414,3797,1445],{"class":1424},[1414,3799,3800,3803],{"class":1416,"line":2124},[1414,3801,3802],{"class":1433},"    time_series_filtered_y_sample",[1414,3804,1445],{"class":1424},[1414,3806,3807,3810,3812,3814],{"class":1416,"line":2144},[1414,3808,3809],{"class":1424},"    \"",[1414,3811,3762],{"class":2375},[1414,3813,2372],{"class":1424},[1414,3815,1445],{"class":1424},[1414,3817,3818,3821,3823,3825,3828,3830],{"class":1416,"line":2193},[1414,3819,3820],{"class":1570},"    label",[1414,3822,1425],{"class":1424},[1414,3824,2372],{"class":1424},[1414,3826,3827],{"class":2375},"Smoothed signal",[1414,3829,2372],{"class":1424},[1414,3831,1445],{"class":1424},[1414,3833,3834],{"class":1416,"line":2257},[1414,3835,1592],{"class":1424},[1414,3837,3838,3840,3842,3845],{"class":1416,"line":2303},[1414,3839,2462],{"class":1420},[1414,3841,46],{"class":1424},[1414,3843,3844],{"class":1433},"legend",[1414,3846,2489],{"class":1424},[1414,3848,3849],{"class":1416,"line":2308},[1414,3850,1968],{"emptyLinePlaceholder":176},[1414,3852,3853,3855,3857,3859],{"class":1416,"line":2314},[1414,3854,2462],{"class":1420},[1414,3856,46],{"class":1424},[1414,3858,2522],{"class":1433},[1414,3860,2489],{"class":1424},[22,3862,3863],{},[1398,3864],{"alt":2529,"src":3865},"/assets/autogenerated/../pathway/docs/2.developers/7.templates/ETL/.interval_over_gaussian_filter/article_21_0.png",[22,3867,3868],{},"Until now, the data was considered static and ingested all at once in a single batch: let's switch to streaming data by adding timestamp to the data!",[22,3870,3871],{},"You will create two batches of points: one with all the \"on-time\" data points arriving at time 0, and another one with \"late\" data points arriving at a later time.\nTo make the computation interesting, the late data points should be in the middle of the signal:",[1406,3873,3875],{"className":1408,"code":3874,"language":1410,"meta":152,"style":152},"init_late = 6\nlength_late = 7\n\n# Extracting the late points\nx_sample_late = x_sample[init_late : init_late + length_late]\ny_sample_late = y_sample[init_late : init_late + length_late]\n\n# Removing the late points from the original sample\nx_sample_ontime = x_sample.copy().tolist()\ndel x_sample_ontime[init_late : init_late + length_late]\ny_sample_ontime = y_sample.copy().tolist()\ndel y_sample_ontime[init_late : init_late + length_late]\n\n# The on-time data points have a timestamp of 0, the other ones have a timestamp of 1\ntime = [0 for i in range(len(x_sample_ontime))]\nfor i in range(len(x_sample_late)):\n    time.append(1)\n\nout_of_order_x_sample = x_sample_ontime + x_sample_late.tolist()\nout_of_order_y_sample = y_sample_ontime + y_sample_late.tolist()\n\nout_of_order_time_series_pd = pd.DataFrame(\n    {\"x\": out_of_order_x_sample, \"y\": out_of_order_y_sample, \"time\": time}\n)\nout_of_order_time_series_pd.to_csv(\"./ooo_ts.csv\", index=False)\n",[100,3876,3877,3887,3897,3901,3906,3932,3955,3959,3964,3985,4007,4026,4047,4051,4056,4093,4116,4133,4137,4158,4180,4185,4201,4247,4252],{"__ignoreMap":152},[1414,3878,3879,3882,3884],{"class":1416,"line":1417},[1414,3880,3881],{"class":1420},"init_late ",[1414,3883,1425],{"class":1424},[1414,3885,3886],{"class":1500}," 6\n",[1414,3888,3889,3892,3894],{"class":1416,"line":153},[1414,3890,3891],{"class":1420},"length_late ",[1414,3893,1425],{"class":1424},[1414,3895,3896],{"class":1500}," 7\n",[1414,3898,3899],{"class":1416,"line":1448},[1414,3900,1968],{"emptyLinePlaceholder":176},[1414,3902,3903],{"class":1416,"line":1467},[1414,3904,3905],{"class":2009},"# Extracting the late points\n",[1414,3907,3908,3911,3913,3916,3918,3920,3922,3925,3927,3930],{"class":1416,"line":1483},[1414,3909,3910],{"class":1420},"x_sample_late ",[1414,3912,1425],{"class":1424},[1414,3914,3915],{"class":1420}," x_sample",[1414,3917,3616],{"class":1424},[1414,3919,3881],{"class":1420},[1414,3921,2624],{"class":1424},[1414,3923,3924],{"class":1420}," init_late ",[1414,3926,2087],{"class":1424},[1414,3928,3929],{"class":1420}," length_late",[1414,3931,3626],{"class":1424},[1414,3933,3934,3937,3939,3941,3943,3945,3947,3949,3951,3953],{"class":1416,"line":1513},[1414,3935,3936],{"class":1420},"y_sample_late ",[1414,3938,1425],{"class":1424},[1414,3940,3755],{"class":1420},[1414,3942,3616],{"class":1424},[1414,3944,3881],{"class":1420},[1414,3946,2624],{"class":1424},[1414,3948,3924],{"class":1420},[1414,3950,2087],{"class":1424},[1414,3952,3929],{"class":1420},[1414,3954,3626],{"class":1424},[1414,3956,3957],{"class":1416,"line":1541},[1414,3958,1968],{"emptyLinePlaceholder":176},[1414,3960,3961],{"class":1416,"line":1552},[1414,3962,3963],{"class":2009},"# Removing the late points from the original sample\n",[1414,3965,3966,3969,3971,3973,3975,3978,3981,3983],{"class":1416,"line":1567},[1414,3967,3968],{"class":1420},"x_sample_ontime ",[1414,3970,1425],{"class":1424},[1414,3972,3915],{"class":1420},[1414,3974,46],{"class":1424},[1414,3976,3977],{"class":1433},"copy",[1414,3979,3980],{"class":1424},"().",[1414,3982,3663],{"class":1433},[1414,3984,2489],{"class":1424},[1414,3986,3987,3990,3993,3995,3997,3999,4001,4003,4005],{"class":1416,"line":1589},[1414,3988,3989],{"class":1930},"del",[1414,3991,3992],{"class":1420}," x_sample_ontime",[1414,3994,3616],{"class":1424},[1414,3996,3881],{"class":1420},[1414,3998,2624],{"class":1424},[1414,4000,3924],{"class":1420},[1414,4002,2087],{"class":1424},[1414,4004,3929],{"class":1420},[1414,4006,3626],{"class":1424},[1414,4008,4009,4012,4014,4016,4018,4020,4022,4024],{"class":1416,"line":2113},[1414,4010,4011],{"class":1420},"y_sample_ontime ",[1414,4013,1425],{"class":1424},[1414,4015,3755],{"class":1420},[1414,4017,46],{"class":1424},[1414,4019,3977],{"class":1433},[1414,4021,3980],{"class":1424},[1414,4023,3663],{"class":1433},[1414,4025,2489],{"class":1424},[1414,4027,4028,4030,4033,4035,4037,4039,4041,4043,4045],{"class":1416,"line":2118},[1414,4029,3989],{"class":1930},[1414,4031,4032],{"class":1420}," y_sample_ontime",[1414,4034,3616],{"class":1424},[1414,4036,3881],{"class":1420},[1414,4038,2624],{"class":1424},[1414,4040,3924],{"class":1420},[1414,4042,2087],{"class":1424},[1414,4044,3929],{"class":1420},[1414,4046,3626],{"class":1424},[1414,4048,4049],{"class":1416,"line":2124},[1414,4050,1968],{"emptyLinePlaceholder":176},[1414,4052,4053],{"class":1416,"line":2144},[1414,4054,4055],{"class":2009},"# The on-time data points have a timestamp of 0, the other ones have a timestamp of 1\n",[1414,4057,4058,4061,4063,4066,4068,4071,4074,4077,4080,4082,4085,4087,4090],{"class":1416,"line":2193},[1414,4059,4060],{"class":1420},"time ",[1414,4062,1425],{"class":1424},[1414,4064,4065],{"class":1424}," [",[1414,4067,2030],{"class":1500},[1414,4069,4070],{"class":1930}," for",[1414,4072,4073],{"class":1420}," i ",[1414,4075,4076],{"class":1930},"in",[1414,4078,4079],{"class":1433}," range",[1414,4081,1634],{"class":1424},[1414,4083,4084],{"class":1433},"len",[1414,4086,1634],{"class":1424},[1414,4088,4089],{"class":1433},"x_sample_ontime",[1414,4091,4092],{"class":1424},"))]\n",[1414,4094,4095,4098,4100,4102,4104,4106,4108,4110,4113],{"class":1416,"line":2257},[1414,4096,4097],{"class":1930},"for",[1414,4099,4073],{"class":1420},[1414,4101,4076],{"class":1930},[1414,4103,4079],{"class":1433},[1414,4105,1634],{"class":1424},[1414,4107,4084],{"class":1433},[1414,4109,1634],{"class":1424},[1414,4111,4112],{"class":1433},"x_sample_late",[1414,4114,4115],{"class":1424},")):\n",[1414,4117,4118,4121,4123,4126,4128,4131],{"class":1416,"line":2303},[1414,4119,4120],{"class":1420},"    time",[1414,4122,46],{"class":1424},[1414,4124,4125],{"class":1433},"append",[1414,4127,1634],{"class":1424},[1414,4129,4130],{"class":1500},"1",[1414,4132,1592],{"class":1424},[1414,4134,4135],{"class":1416,"line":2308},[1414,4136,1968],{"emptyLinePlaceholder":176},[1414,4138,4139,4142,4144,4147,4149,4152,4154,4156],{"class":1416,"line":2314},[1414,4140,4141],{"class":1420},"out_of_order_x_sample ",[1414,4143,1425],{"class":1424},[1414,4145,4146],{"class":1420}," x_sample_ontime ",[1414,4148,2087],{"class":1424},[1414,4150,4151],{"class":1420}," x_sample_late",[1414,4153,46],{"class":1424},[1414,4155,3663],{"class":1433},[1414,4157,2489],{"class":1424},[1414,4159,4161,4164,4166,4169,4171,4174,4176,4178],{"class":1416,"line":4160},20,[1414,4162,4163],{"class":1420},"out_of_order_y_sample ",[1414,4165,1425],{"class":1424},[1414,4167,4168],{"class":1420}," y_sample_ontime ",[1414,4170,2087],{"class":1424},[1414,4172,4173],{"class":1420}," y_sample_late",[1414,4175,46],{"class":1424},[1414,4177,3663],{"class":1433},[1414,4179,2489],{"class":1424},[1414,4181,4183],{"class":1416,"line":4182},21,[1414,4184,1968],{"emptyLinePlaceholder":176},[1414,4186,4188,4191,4193,4195,4197,4199],{"class":1416,"line":4187},22,[1414,4189,4190],{"class":1420},"out_of_order_time_series_pd ",[1414,4192,1425],{"class":1424},[1414,4194,2607],{"class":1420},[1414,4196,46],{"class":1424},[1414,4198,2612],{"class":1433},[1414,4200,1437],{"class":1424},[1414,4202,4204,4207,4209,4211,4213,4215,4218,4220,4222,4224,4226,4228,4231,4233,4235,4237,4239,4241,4244],{"class":1416,"line":4203},23,[1414,4205,4206],{"class":1424},"    {",[1414,4208,2372],{"class":1424},[1414,4210,2108],{"class":2375},[1414,4212,2372],{"class":1424},[1414,4214,2624],{"class":1424},[1414,4216,4217],{"class":1433}," out_of_order_x_sample",[1414,4219,1504],{"class":1424},[1414,4221,2632],{"class":1424},[1414,4223,2635],{"class":2375},[1414,4225,2372],{"class":1424},[1414,4227,2624],{"class":1424},[1414,4229,4230],{"class":1433}," out_of_order_y_sample",[1414,4232,1504],{"class":1424},[1414,4234,2632],{"class":1424},[1414,4236,2376],{"class":2375},[1414,4238,2372],{"class":1424},[1414,4240,2624],{"class":1424},[1414,4242,4243],{"class":1433}," time",[1414,4245,4246],{"class":1424},"}\n",[1414,4248,4250],{"class":1416,"line":4249},24,[1414,4251,1592],{"class":1424},[1414,4253,4255,4258,4260,4263,4265,4267,4270,4272,4274,4277],{"class":1416,"line":4254},25,[1414,4256,4257],{"class":1420},"out_of_order_time_series_pd",[1414,4259,46],{"class":1424},[1414,4261,4262],{"class":1433},"to_csv",[1414,4264,1634],{"class":1424},[1414,4266,2372],{"class":1424},[1414,4268,4269],{"class":2375},"./ooo_ts.csv",[1414,4271,2372],{"class":1424},[1414,4273,1504],{"class":1424},[1414,4275,4276],{"class":1570}," index",[1414,4278,4279],{"class":1424},"=False)\n",[22,4281,4282],{},"Let's plot the resulting data sample: the late points, in red, have been removed from the original time series:",[1406,4284,4286],{"className":1408,"code":4285,"language":1410,"meta":152,"style":152},"plt.subplot(2, 1, 1)\nset_params_plot()\nplt.plot(x_sample_ontime, y_sample_ontime, \"x-\", label=\"On-time data points\")\nplt.plot(x_sample_late, y_sample_late, \"x\", color=\"red\", label=\"Late data points\")\nplt.legend()\n\nplt.show()\n",[100,4287,4288,4310,4316,4355,4407,4417,4421],{"__ignoreMap":152},[1414,4289,4290,4292,4294,4296,4298,4300,4302,4304,4306,4308],{"class":1416,"line":1417},[1414,4291,2462],{"class":1420},[1414,4293,46],{"class":1424},[1414,4295,2467],{"class":1433},[1414,4297,1634],{"class":1424},[1414,4299,2165],{"class":1500},[1414,4301,1504],{"class":1424},[1414,4303,2294],{"class":1500},[1414,4305,1504],{"class":1424},[1414,4307,2294],{"class":1500},[1414,4309,1592],{"class":1424},[1414,4311,4312,4314],{"class":1416,"line":153},[1414,4313,2486],{"class":1433},[1414,4315,2489],{"class":1424},[1414,4317,4318,4320,4322,4324,4326,4328,4330,4332,4334,4336,4338,4340,4342,4344,4346,4348,4351,4353],{"class":1416,"line":1448},[1414,4319,2462],{"class":1420},[1414,4321,46],{"class":1424},[1414,4323,2498],{"class":1433},[1414,4325,1634],{"class":1424},[1414,4327,4089],{"class":1433},[1414,4329,1504],{"class":1424},[1414,4331,4032],{"class":1433},[1414,4333,1504],{"class":1424},[1414,4335,2632],{"class":1424},[1414,4337,3762],{"class":2375},[1414,4339,2372],{"class":1424},[1414,4341,1504],{"class":1424},[1414,4343,3769],{"class":1570},[1414,4345,1425],{"class":1424},[1414,4347,2372],{"class":1424},[1414,4349,4350],{"class":2375},"On-time data points",[1414,4352,2372],{"class":1424},[1414,4354,1592],{"class":1424},[1414,4356,4357,4359,4361,4363,4365,4367,4369,4371,4373,4375,4377,4379,4381,4383,4385,4387,4390,4392,4394,4396,4398,4400,4403,4405],{"class":1416,"line":1467},[1414,4358,2462],{"class":1420},[1414,4360,46],{"class":1424},[1414,4362,2498],{"class":1433},[1414,4364,1634],{"class":1424},[1414,4366,4112],{"class":1433},[1414,4368,1504],{"class":1424},[1414,4370,4173],{"class":1433},[1414,4372,1504],{"class":1424},[1414,4374,2632],{"class":1424},[1414,4376,2108],{"class":2375},[1414,4378,2372],{"class":1424},[1414,4380,1504],{"class":1424},[1414,4382,3517],{"class":1570},[1414,4384,1425],{"class":1424},[1414,4386,2372],{"class":1424},[1414,4388,4389],{"class":2375},"red",[1414,4391,2372],{"class":1424},[1414,4393,1504],{"class":1424},[1414,4395,3769],{"class":1570},[1414,4397,1425],{"class":1424},[1414,4399,2372],{"class":1424},[1414,4401,4402],{"class":2375},"Late data points",[1414,4404,2372],{"class":1424},[1414,4406,1592],{"class":1424},[1414,4408,4409,4411,4413,4415],{"class":1416,"line":1483},[1414,4410,2462],{"class":1420},[1414,4412,46],{"class":1424},[1414,4414,3844],{"class":1433},[1414,4416,2489],{"class":1424},[1414,4418,4419],{"class":1416,"line":1513},[1414,4420,1968],{"emptyLinePlaceholder":176},[1414,4422,4423,4425,4427,4429],{"class":1416,"line":1541},[1414,4424,2462],{"class":1420},[1414,4426,46],{"class":1424},[1414,4428,2522],{"class":1433},[1414,4430,2489],{"class":1424},[22,4432,4433],{},[1398,4434],{"alt":2529,"src":4435},"/assets/autogenerated/../pathway/docs/2.developers/7.templates/ETL/.interval_over_gaussian_filter/article_25_0.png",[22,4437,4438],{},"Now, you will apply the filter to this data source.\nYou first need to create a pipeline to apply the Gaussian filter, as done in the previous sections:",[1406,4440,4442],{"className":1408,"code":4441,"language":1410,"meta":152,"style":152},"def smooth_table(table):\n    points_within_50 = table.windowby(\n        table.x,\n        window=pw.temporal.intervals_over(\n            at=table.x, lower_bound=-50.0, upper_bound=50.0, is_outer=False\n        ),\n    ).reduce(\n        pw.this._pw_window_location,\n        xs=pw.reducers.ndarray(pw.this.x),\n        ys=pw.reducers.ndarray(pw.this.y),\n    )\n    # Applying the Gaussian filter\n    time_series_filtered = points_within_50.select(\n        x=pw.this._pw_window_location,\n        y=gaussian_filter(pw.this.xs, pw.this.ys, pw.this._pw_window_location),\n    )\n    return time_series_filtered\n",[100,4443,4444,4458,4474,4485,4504,4539,4544,4553,4568,4599,4630,4635,4640,4655,4674,4721,4725],{"__ignoreMap":152},[1414,4445,4446,4448,4451,4453,4456],{"class":1416,"line":1417},[1414,4447,2351],{"class":2350},[1414,4449,4450],{"class":1433}," smooth_table",[1414,4452,1634],{"class":1424},[1414,4454,4455],{"class":1570},"table",[1414,4457,2597],{"class":1424},[1414,4459,4460,4463,4465,4468,4470,4472],{"class":1416,"line":153},[1414,4461,4462],{"class":1420},"    points_within_50 ",[1414,4464,1425],{"class":1424},[1414,4466,4467],{"class":1420}," table",[1414,4469,46],{"class":1424},[1414,4471,1709],{"class":1433},[1414,4473,1437],{"class":1424},[1414,4475,4476,4479,4481,4483],{"class":1416,"line":1448},[1414,4477,4478],{"class":1433},"        table",[1414,4480,46],{"class":1424},[1414,4482,2108],{"class":1456},[1414,4484,1445],{"class":1424},[1414,4486,4487,4490,4492,4494,4496,4498,4500,4502],{"class":1416,"line":1467},[1414,4488,4489],{"class":1570},"        window",[1414,4491,1425],{"class":1424},[1414,4493,1576],{"class":1433},[1414,4495,46],{"class":1424},[1414,4497,1490],{"class":1456},[1414,4499,46],{"class":1424},[1414,4501,1713],{"class":1433},[1414,4503,1437],{"class":1424},[1414,4505,4506,4509,4511,4513,4515,4517,4519,4521,4523,4525,4527,4529,4531,4533,4535,4537],{"class":1416,"line":1483},[1414,4507,4508],{"class":1570},"            at",[1414,4510,1425],{"class":1424},[1414,4512,4455],{"class":1433},[1414,4514,46],{"class":1424},[1414,4516,2108],{"class":1456},[1414,4518,1504],{"class":1424},[1414,4520,1794],{"class":1570},[1414,4522,1797],{"class":1424},[1414,4524,1800],{"class":1500},[1414,4526,1504],{"class":1424},[1414,4528,1805],{"class":1570},[1414,4530,1425],{"class":1424},[1414,4532,1800],{"class":1500},[1414,4534,1504],{"class":1424},[1414,4536,2853],{"class":1570},[1414,4538,2856],{"class":1424},[1414,4540,4541],{"class":1416,"line":1513},[1414,4542,4543],{"class":1424},"        ),\n",[1414,4545,4546,4549,4551],{"class":1416,"line":1541},[1414,4547,4548],{"class":1424},"    ).",[1414,4550,1613],{"class":1433},[1414,4552,1437],{"class":1424},[1414,4554,4555,4558,4560,4562,4564,4566],{"class":1416,"line":1552},[1414,4556,4557],{"class":1433},"        pw",[1414,4559,46],{"class":1424},[1414,4561,1686],{"class":1456},[1414,4563,46],{"class":1424},[1414,4565,1836],{"class":1456},[1414,4567,1445],{"class":1424},[1414,4569,4570,4573,4575,4577,4579,4581,4583,4585,4587,4589,4591,4593,4595,4597],{"class":1416,"line":1567},[1414,4571,4572],{"class":1570},"        xs",[1414,4574,1425],{"class":1424},[1414,4576,1576],{"class":1433},[1414,4578,46],{"class":1424},[1414,4580,1672],{"class":1456},[1414,4582,46],{"class":1424},[1414,4584,2900],{"class":1433},[1414,4586,1634],{"class":1424},[1414,4588,1576],{"class":1433},[1414,4590,46],{"class":1424},[1414,4592,1686],{"class":1456},[1414,4594,46],{"class":1424},[1414,4596,2108],{"class":1456},[1414,4598,1510],{"class":1424},[1414,4600,4601,4604,4606,4608,4610,4612,4614,4616,4618,4620,4622,4624,4626,4628],{"class":1416,"line":1589},[1414,4602,4603],{"class":1570},"        ys",[1414,4605,1425],{"class":1424},[1414,4607,1576],{"class":1433},[1414,4609,46],{"class":1424},[1414,4611,1672],{"class":1456},[1414,4613,46],{"class":1424},[1414,4615,2900],{"class":1433},[1414,4617,1634],{"class":1424},[1414,4619,1576],{"class":1433},[1414,4621,46],{"class":1424},[1414,4623,1686],{"class":1456},[1414,4625,46],{"class":1424},[1414,4627,2635],{"class":1456},[1414,4629,1510],{"class":1424},[1414,4631,4632],{"class":1416,"line":2113},[1414,4633,4634],{"class":1424},"    )\n",[1414,4636,4637],{"class":1416,"line":2118},[1414,4638,4639],{"class":2009},"    # Applying the Gaussian filter\n",[1414,4641,4642,4645,4647,4649,4651,4653],{"class":1416,"line":2124},[1414,4643,4644],{"class":1420},"    time_series_filtered ",[1414,4646,1425],{"class":1424},[1414,4648,3232],{"class":1420},[1414,4650,46],{"class":1424},[1414,4652,1547],{"class":1433},[1414,4654,1437],{"class":1424},[1414,4656,4657,4660,4662,4664,4666,4668,4670,4672],{"class":1416,"line":2144},[1414,4658,4659],{"class":1570},"        x",[1414,4661,1425],{"class":1424},[1414,4663,1576],{"class":1433},[1414,4665,46],{"class":1424},[1414,4667,1686],{"class":1456},[1414,4669,46],{"class":1424},[1414,4671,1836],{"class":1456},[1414,4673,1445],{"class":1424},[1414,4675,4676,4679,4681,4683,4685,4687,4689,4691,4693,4695,4697,4699,4701,4703,4705,4707,4709,4711,4713,4715,4717,4719],{"class":1416,"line":2193},[1414,4677,4678],{"class":1570},"        y",[1414,4680,1425],{"class":1424},[1414,4682,3267],{"class":1433},[1414,4684,1634],{"class":1424},[1414,4686,1576],{"class":1433},[1414,4688,46],{"class":1424},[1414,4690,1686],{"class":1456},[1414,4692,46],{"class":1424},[1414,4694,3280],{"class":1456},[1414,4696,1504],{"class":1424},[1414,4698,1529],{"class":1433},[1414,4700,46],{"class":1424},[1414,4702,1686],{"class":1456},[1414,4704,46],{"class":1424},[1414,4706,3293],{"class":1456},[1414,4708,1504],{"class":1424},[1414,4710,1529],{"class":1433},[1414,4712,46],{"class":1424},[1414,4714,1686],{"class":1456},[1414,4716,46],{"class":1424},[1414,4718,1836],{"class":1456},[1414,4720,1510],{"class":1424},[1414,4722,4723],{"class":1416,"line":2257},[1414,4724,4634],{"class":1424},[1414,4726,4727,4729],{"class":1416,"line":2303},[1414,4728,2649],{"class":1930},[1414,4730,4731],{"class":1420}," time_series_filtered\n",[22,4733,4734,4735,4738,4739,4743,4744,2624],{},"Then, you need to read the input source.\nYou are not working with a static data source anymore, but a streaming one.\nYou can use ",[100,4736,4737],{},"pw.demo.replay_csv_with_time"," to replay a data source in a CSV file respecting the time between updates based on a timestamp columns (more information about this function in our dedicated ",[29,4740,4742],{"href":4741},"/developers/user-guide/connect/artificial-streams#replaying-static-csv-files-with-replay_csv-and-replay_csv_with_time","article","). Here the time is contained in the column ",[100,4745,2376],{},[1406,4747,4749],{"className":1408,"code":4748,"language":1410,"meta":152,"style":152},"# For streaming data, you need to define the schema\nclass InputSchema(pw.Schema):\n    x: float\n    y: float\n    time: int\n\n\nooo_input_table = pw.demo.replay_csv_with_time(\n    path=\"ooo_ts.csv\",\n    schema=InputSchema,\n    time_column=\"time\",\n    unit=\"s\",\n    autocommit_ms=100,\n)\nooo_smoothed_table = smooth_table(ooo_input_table)\npw.io.csv.write(ooo_smoothed_table, \"ooo_results.csv\")\npw.run()\n",[100,4750,4751,4756,4775,4784,4792,4801,4805,4809,4830,4846,4858,4873,4889,4901,4905,4921,4956],{"__ignoreMap":152},[1414,4752,4753],{"class":1416,"line":1417},[1414,4754,4755],{"class":2009},"# For streaming data, you need to define the schema\n",[1414,4757,4758,4761,4764,4766,4768,4770,4773],{"class":1416,"line":153},[1414,4759,4760],{"class":2350},"class",[1414,4762,4763],{"class":3065}," InputSchema",[1414,4765,1634],{"class":1424},[1414,4767,1576],{"class":3065},[1414,4769,46],{"class":1424},[1414,4771,4772],{"class":3065},"Schema",[1414,4774,2597],{"class":1424},[1414,4776,4777,4779,4781],{"class":1416,"line":1448},[1414,4778,3243],{"class":1420},[1414,4780,2624],{"class":1424},[1414,4782,4783],{"class":3065}," float\n",[1414,4785,4786,4788,4790],{"class":1416,"line":1467},[1414,4787,3262],{"class":1420},[1414,4789,2624],{"class":1424},[1414,4791,4783],{"class":3065},[1414,4793,4794,4796,4798],{"class":1416,"line":1483},[1414,4795,4120],{"class":1420},[1414,4797,2624],{"class":1424},[1414,4799,4800],{"class":3065}," int\n",[1414,4802,4803],{"class":1416,"line":1513},[1414,4804,1968],{"emptyLinePlaceholder":176},[1414,4806,4807],{"class":1416,"line":1541},[1414,4808,1968],{"emptyLinePlaceholder":176},[1414,4810,4811,4814,4816,4818,4820,4823,4825,4828],{"class":1416,"line":1552},[1414,4812,4813],{"class":1420},"ooo_input_table ",[1414,4815,1425],{"class":1424},[1414,4817,1529],{"class":1420},[1414,4819,46],{"class":1424},[1414,4821,4822],{"class":1456},"demo",[1414,4824,46],{"class":1424},[1414,4826,4827],{"class":1433},"replay_csv_with_time",[1414,4829,1437],{"class":1424},[1414,4831,4832,4835,4837,4839,4842,4844],{"class":1416,"line":1567},[1414,4833,4834],{"class":1570},"    path",[1414,4836,1425],{"class":1424},[1414,4838,2372],{"class":1424},[1414,4840,4841],{"class":2375},"ooo_ts.csv",[1414,4843,2372],{"class":1424},[1414,4845,1445],{"class":1424},[1414,4847,4848,4851,4853,4856],{"class":1416,"line":1589},[1414,4849,4850],{"class":1570},"    schema",[1414,4852,1425],{"class":1424},[1414,4854,4855],{"class":1433},"InputSchema",[1414,4857,1445],{"class":1424},[1414,4859,4860,4863,4865,4867,4869,4871],{"class":1416,"line":2113},[1414,4861,4862],{"class":1570},"    time_column",[1414,4864,1425],{"class":1424},[1414,4866,2372],{"class":1424},[1414,4868,2376],{"class":2375},[1414,4870,2372],{"class":1424},[1414,4872,1445],{"class":1424},[1414,4874,4875,4878,4880,4882,4885,4887],{"class":1416,"line":2118},[1414,4876,4877],{"class":1570},"    unit",[1414,4879,1425],{"class":1424},[1414,4881,2372],{"class":1424},[1414,4883,4884],{"class":2375},"s",[1414,4886,2372],{"class":1424},[1414,4888,1445],{"class":1424},[1414,4890,4891,4894,4896,4899],{"class":1416,"line":2124},[1414,4892,4893],{"class":1570},"    autocommit_ms",[1414,4895,1425],{"class":1424},[1414,4897,4898],{"class":1500},"100",[1414,4900,1445],{"class":1424},[1414,4902,4903],{"class":1416,"line":2144},[1414,4904,1592],{"class":1424},[1414,4906,4907,4910,4912,4914,4916,4919],{"class":1416,"line":2193},[1414,4908,4909],{"class":1420},"ooo_smoothed_table ",[1414,4911,1425],{"class":1424},[1414,4913,4450],{"class":1433},[1414,4915,1634],{"class":1424},[1414,4917,4918],{"class":1433},"ooo_input_table",[1414,4920,1592],{"class":1424},[1414,4922,4923,4925,4927,4930,4932,4935,4937,4940,4942,4945,4947,4949,4952,4954],{"class":1416,"line":2257},[1414,4924,1576],{"class":1420},[1414,4926,46],{"class":1424},[1414,4928,4929],{"class":1456},"io",[1414,4931,46],{"class":1424},[1414,4933,4934],{"class":1456},"csv",[1414,4936,46],{"class":1424},[1414,4938,4939],{"class":1433},"write",[1414,4941,1634],{"class":1424},[1414,4943,4944],{"class":1433},"ooo_smoothed_table",[1414,4946,1504],{"class":1424},[1414,4948,2632],{"class":1424},[1414,4950,4951],{"class":2375},"ooo_results.csv",[1414,4953,2372],{"class":1424},[1414,4955,1592],{"class":1424},[1414,4957,4958,4960,4962,4965],{"class":1416,"line":2303},[1414,4959,1576],{"class":1420},[1414,4961,46],{"class":1424},[1414,4963,4964],{"class":1433},"run",[1414,4966,2489],{"class":1424},[1406,4968,4971],{"className":4969,"code":4970,"language":1601},[1599],"Output()\n",[100,4972,4970],{"__ignoreMap":152},[1406,4974],{"style":4975},"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace",[22,4977,4978],{},"Let's take a look at the results:",[1406,4980,4982],{"className":1408,"code":4981,"language":1410,"meta":152,"style":152},"results = pd.read_csv(\"./ooo_results.csv\")\nresults = results.sort_values(by=[\"x\"])\nresults.head()\n",[100,4983,4984,5008,5034],{"__ignoreMap":152},[1414,4985,4986,4988,4990,4992,4994,4997,4999,5001,5004,5006],{"class":1416,"line":1417},[1414,4987,1421],{"class":1420},[1414,4989,1425],{"class":1424},[1414,4991,2607],{"class":1420},[1414,4993,46],{"class":1424},[1414,4995,4996],{"class":1433},"read_csv",[1414,4998,1634],{"class":1424},[1414,5000,2372],{"class":1424},[1414,5002,5003],{"class":2375},"./ooo_results.csv",[1414,5005,2372],{"class":1424},[1414,5007,1592],{"class":1424},[1414,5009,5010,5012,5014,5016,5018,5020,5022,5024,5026,5028,5030,5032],{"class":1416,"line":153},[1414,5011,1421],{"class":1420},[1414,5013,1425],{"class":1424},[1414,5015,1627],{"class":1420},[1414,5017,46],{"class":1424},[1414,5019,3367],{"class":1433},[1414,5021,1634],{"class":1424},[1414,5023,3372],{"class":1570},[1414,5025,3375],{"class":1424},[1414,5027,2372],{"class":1424},[1414,5029,2108],{"class":2375},[1414,5031,2372],{"class":1424},[1414,5033,3384],{"class":1424},[1414,5035,5036,5038,5040,5043],{"class":1416,"line":1448},[1414,5037,1637],{"class":1420},[1414,5039,46],{"class":1424},[1414,5041,5042],{"class":1433},"head",[1414,5044,2489],{"class":1424},[5046,5047,5048,5052],"div",{},[5049,5050,5051],"style",{"scoped":176},"\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n.dataframe tbody tr th {\n    vertical-align: top;\n}\n\n.dataframe thead th {\n    text-align: right;\n}\n",[4455,5053,5056,5057,5056,5078],{"border":1417,"className":5054},[5055],"dataframe","\n  ",[5058,5059,5060,5061,5056],"thead",{},"\n    ",[5062,5063,5065,5066,5065,5069,5065,5071,5065,5073,5065,5075,5060],"tr",{"style":5064},"text-align: right;","\n      ",[5067,5068],"th",{},[5067,5070,2108],{},[5067,5072,2635],{},[5067,5074,2376],{},[5067,5076,5077],{},"diff",[5079,5080,5060,5081,5060,5098,5060,5114,5060,5129,5060,5142,5056],"tbody",{},[5062,5082,5065,5083,5065,5086,5065,5090,5065,5093,5065,5096,5060],{},[5067,5084,5085],{},"5",[5087,5088,5089],"td",{},"2.663173",[5087,5091,5092],{},"9.995850",[5087,5094,5095],{},"1722589159134",[5087,5097,4130],{},[5062,5099,5065,5100,5065,5103,5065,5106,5065,5109,5065,5112,5060],{},[5067,5101,5102],{},"23",[5087,5104,5105],{},"11.440568",[5087,5107,5108],{},"11.651747",[5087,5110,5111],{},"1722589160450",[5087,5113,4130],{},[5062,5115,5065,5116,5065,5119,5065,5121,5065,5124,5065,5126,5060],{},[5067,5117,5118],{},"21",[5087,5120,5105],{},[5087,5122,5123],{},"10.975576",[5087,5125,5111],{},[5087,5127,5128],{},"-1",[5062,5130,5065,5131,5065,5134,5065,5136,5065,5138,5065,5140,5060],{},[5067,5132,5133],{},"4",[5087,5135,5105],{},[5087,5137,5123],{},[5087,5139,5095],{},[5087,5141,4130],{},[5062,5143,5065,5144,5065,5147,5065,5150,5065,5153,5065,5155,5060],{},[5067,5145,5146],{},"18",[5087,5148,5149],{},"16.929219",[5087,5151,5152],{},"11.780476",[5087,5154,5111],{},[5087,5156,5128],{},[22,5158,5159,5160,41,5162,5164,5165,5167,5168,5170,5171,5174,5175,1544],{},"There are two new columns: ",[100,5161,2376],{},[100,5163,5077],{},".\nThe column ",[100,5166,2376],{}," corresponds to the processing time at which Pathway generated the output, and ",[100,5169,5077],{}," represents whether the output was added or removed.\nAn update is decomposed in two simultaneous atomic operations: the removal of the last value (",[100,5172,5173],{},"diff==-1",") and the addition of the new one (",[100,5176,5177],{},"diff==1",[22,5179,5180,5181,46],{},"Here we have only two batches, so there are only two different values in the column ",[100,5182,2376],{},[22,5184,5185],{},"Let's see the results of the first batch when the \"on-time\" data points were added to the system:",[1406,5187,5189],{"className":1408,"code":5188,"language":1410,"meta":152,"style":152},"# The first batch has only additions, no need to filter on `diff` (all values are equal to 1).\nfirst_results = results[results[\"time\"] == min(results[\"time\"])]\n",[100,5190,5191,5196],{"__ignoreMap":152},[1414,5192,5193],{"class":1416,"line":1417},[1414,5194,5195],{"class":2009},"# The first batch has only additions, no need to filter on `diff` (all values are equal to 1).\n",[1414,5197,5198,5201,5203,5205,5207,5209,5211,5213,5215,5217,5220,5223,5225,5227,5229,5231,5233,5235,5237],{"class":1416,"line":153},[1414,5199,5200],{"class":1420},"first_results ",[1414,5202,1425],{"class":1424},[1414,5204,1627],{"class":1420},[1414,5206,3616],{"class":1424},[1414,5208,1637],{"class":1420},[1414,5210,3616],{"class":1424},[1414,5212,2372],{"class":1424},[1414,5214,2376],{"class":2375},[1414,5216,2372],{"class":1424},[1414,5218,5219],{"class":1424},"]",[1414,5221,5222],{"class":1424}," ==",[1414,5224,3094],{"class":1433},[1414,5226,1634],{"class":1424},[1414,5228,1637],{"class":1433},[1414,5230,3616],{"class":1424},[1414,5232,2372],{"class":1424},[1414,5234,2376],{"class":2375},[1414,5236,2372],{"class":1424},[1414,5238,5239],{"class":1424},"])]\n",[1406,5241,5243],{"className":1408,"code":5242,"language":1410,"meta":152,"style":152},"plt.subplot(2, 1, 1)\nplt.plot(x_sample_ontime, y_sample_ontime, \"x-\", label=\"On-time data points\")\nplt.plot(first_results.x, first_results.y, \"x-\", label=\"Smoothed time series\")\nplt.legend()\nset_params_plot()\n",[100,5244,5245,5267,5305,5354,5364],{"__ignoreMap":152},[1414,5246,5247,5249,5251,5253,5255,5257,5259,5261,5263,5265],{"class":1416,"line":1417},[1414,5248,2462],{"class":1420},[1414,5250,46],{"class":1424},[1414,5252,2467],{"class":1433},[1414,5254,1634],{"class":1424},[1414,5256,2165],{"class":1500},[1414,5258,1504],{"class":1424},[1414,5260,2294],{"class":1500},[1414,5262,1504],{"class":1424},[1414,5264,2294],{"class":1500},[1414,5266,1592],{"class":1424},[1414,5268,5269,5271,5273,5275,5277,5279,5281,5283,5285,5287,5289,5291,5293,5295,5297,5299,5301,5303],{"class":1416,"line":153},[1414,5270,2462],{"class":1420},[1414,5272,46],{"class":1424},[1414,5274,2498],{"class":1433},[1414,5276,1634],{"class":1424},[1414,5278,4089],{"class":1433},[1414,5280,1504],{"class":1424},[1414,5282,4032],{"class":1433},[1414,5284,1504],{"class":1424},[1414,5286,2632],{"class":1424},[1414,5288,3762],{"class":2375},[1414,5290,2372],{"class":1424},[1414,5292,1504],{"class":1424},[1414,5294,3769],{"class":1570},[1414,5296,1425],{"class":1424},[1414,5298,2372],{"class":1424},[1414,5300,4350],{"class":2375},[1414,5302,2372],{"class":1424},[1414,5304,1592],{"class":1424},[1414,5306,5307,5309,5311,5313,5315,5318,5320,5322,5324,5327,5329,5331,5333,5335,5337,5339,5341,5343,5345,5347,5350,5352],{"class":1416,"line":1448},[1414,5308,2462],{"class":1420},[1414,5310,46],{"class":1424},[1414,5312,2498],{"class":1433},[1414,5314,1634],{"class":1424},[1414,5316,5317],{"class":1433},"first_results",[1414,5319,46],{"class":1424},[1414,5321,2108],{"class":1456},[1414,5323,1504],{"class":1424},[1414,5325,5326],{"class":1433}," first_results",[1414,5328,46],{"class":1424},[1414,5330,2635],{"class":1456},[1414,5332,1504],{"class":1424},[1414,5334,2632],{"class":1424},[1414,5336,3762],{"class":2375},[1414,5338,2372],{"class":1424},[1414,5340,1504],{"class":1424},[1414,5342,3769],{"class":1570},[1414,5344,1425],{"class":1424},[1414,5346,2372],{"class":1424},[1414,5348,5349],{"class":2375},"Smoothed time series",[1414,5351,2372],{"class":1424},[1414,5353,1592],{"class":1424},[1414,5355,5356,5358,5360,5362],{"class":1416,"line":1467},[1414,5357,2462],{"class":1420},[1414,5359,46],{"class":1424},[1414,5361,3844],{"class":1433},[1414,5363,2489],{"class":1424},[1414,5365,5366,5368],{"class":1416,"line":1483},[1414,5367,2486],{"class":1433},[1414,5369,2489],{"class":1424},[22,5371,5372],{},[1398,5373],{"alt":2529,"src":5374},"/assets/autogenerated/../pathway/docs/2.developers/7.templates/ETL/.interval_over_gaussian_filter/article_34_0.png",[22,5376,5377],{},"As you can see, there is a gap in the middle of the time series as some points are missing.\nHowever, the Gaussian filter did its job by smoothing the signal with the data points it had at its disposal.",[22,5379,5380],{},"Now, let's see what was the output when the second batch of points was ingested:",[1406,5382,5384],{"className":1408,"code":5383,"language":1410,"meta":152,"style":152},"late_results = results[results[\"time\"] == max(results[\"time\"])]\n# We only focus on the new values, i.e., with diff==1\nlate_results = late_results[late_results[\"diff\"] == 1]\n",[100,5385,5386,5428,5433],{"__ignoreMap":152},[1414,5387,5388,5391,5393,5395,5397,5399,5401,5403,5405,5407,5409,5411,5414,5416,5418,5420,5422,5424,5426],{"class":1416,"line":1417},[1414,5389,5390],{"class":1420},"late_results ",[1414,5392,1425],{"class":1424},[1414,5394,1627],{"class":1420},[1414,5396,3616],{"class":1424},[1414,5398,1637],{"class":1420},[1414,5400,3616],{"class":1424},[1414,5402,2372],{"class":1424},[1414,5404,2376],{"class":2375},[1414,5406,2372],{"class":1424},[1414,5408,5219],{"class":1424},[1414,5410,5222],{"class":1424},[1414,5412,5413],{"class":1433}," max",[1414,5415,1634],{"class":1424},[1414,5417,1637],{"class":1433},[1414,5419,3616],{"class":1424},[1414,5421,2372],{"class":1424},[1414,5423,2376],{"class":2375},[1414,5425,2372],{"class":1424},[1414,5427,5239],{"class":1424},[1414,5429,5430],{"class":1416,"line":153},[1414,5431,5432],{"class":2009},"# We only focus on the new values, i.e., with diff==1\n",[1414,5434,5435,5437,5439,5442,5444,5447,5449,5451,5453,5455,5457,5459,5461],{"class":1416,"line":1448},[1414,5436,5390],{"class":1420},[1414,5438,1425],{"class":1424},[1414,5440,5441],{"class":1420}," late_results",[1414,5443,3616],{"class":1424},[1414,5445,5446],{"class":1420},"late_results",[1414,5448,3616],{"class":1424},[1414,5450,2372],{"class":1424},[1414,5452,5077],{"class":2375},[1414,5454,2372],{"class":1424},[1414,5456,5219],{"class":1424},[1414,5458,5222],{"class":1424},[1414,5460,2294],{"class":1500},[1414,5462,3626],{"class":1424},[1406,5464,5466],{"className":1408,"code":5465,"language":1410,"meta":152,"style":152},"plt.subplot(2, 1, 1)\nplt.plot(x_sample, y_sample, \"x-\", label=\"Total data points\")\nplt.plot(\n    late_results.x, late_results.y, \"x-\", color=\"red\", label=\"Updates of the smoothing\"\n)\nplt.legend()\nset_params_plot()\n",[100,5467,5468,5490,5529,5539,5590,5594,5604],{"__ignoreMap":152},[1414,5469,5470,5472,5474,5476,5478,5480,5482,5484,5486,5488],{"class":1416,"line":1417},[1414,5471,2462],{"class":1420},[1414,5473,46],{"class":1424},[1414,5475,2467],{"class":1433},[1414,5477,1634],{"class":1424},[1414,5479,2165],{"class":1500},[1414,5481,1504],{"class":1424},[1414,5483,2294],{"class":1500},[1414,5485,1504],{"class":1424},[1414,5487,2294],{"class":1500},[1414,5489,1592],{"class":1424},[1414,5491,5492,5494,5496,5498,5500,5502,5504,5506,5508,5510,5512,5514,5516,5518,5520,5522,5525,5527],{"class":1416,"line":153},[1414,5493,2462],{"class":1420},[1414,5495,46],{"class":1424},[1414,5497,2498],{"class":1433},[1414,5499,1634],{"class":1424},[1414,5501,3750],{"class":1433},[1414,5503,1504],{"class":1424},[1414,5505,3755],{"class":1433},[1414,5507,1504],{"class":1424},[1414,5509,2632],{"class":1424},[1414,5511,3762],{"class":2375},[1414,5513,2372],{"class":1424},[1414,5515,1504],{"class":1424},[1414,5517,3769],{"class":1570},[1414,5519,1425],{"class":1424},[1414,5521,2372],{"class":1424},[1414,5523,5524],{"class":2375},"Total data points",[1414,5526,2372],{"class":1424},[1414,5528,1592],{"class":1424},[1414,5530,5531,5533,5535,5537],{"class":1416,"line":1448},[1414,5532,2462],{"class":1420},[1414,5534,46],{"class":1424},[1414,5536,2498],{"class":1433},[1414,5538,1437],{"class":1424},[1414,5540,5541,5544,5546,5548,5550,5552,5554,5556,5558,5560,5562,5564,5566,5568,5570,5572,5574,5576,5578,5580,5582,5584,5587],{"class":1416,"line":1467},[1414,5542,5543],{"class":1433},"    late_results",[1414,5545,46],{"class":1424},[1414,5547,2108],{"class":1456},[1414,5549,1504],{"class":1424},[1414,5551,5441],{"class":1433},[1414,5553,46],{"class":1424},[1414,5555,2635],{"class":1456},[1414,5557,1504],{"class":1424},[1414,5559,2632],{"class":1424},[1414,5561,3762],{"class":2375},[1414,5563,2372],{"class":1424},[1414,5565,1504],{"class":1424},[1414,5567,3517],{"class":1570},[1414,5569,1425],{"class":1424},[1414,5571,2372],{"class":1424},[1414,5573,4389],{"class":2375},[1414,5575,2372],{"class":1424},[1414,5577,1504],{"class":1424},[1414,5579,3769],{"class":1570},[1414,5581,1425],{"class":1424},[1414,5583,2372],{"class":1424},[1414,5585,5586],{"class":2375},"Updates of the smoothing",[1414,5588,5589],{"class":1424},"\"\n",[1414,5591,5592],{"class":1416,"line":1483},[1414,5593,1592],{"class":1424},[1414,5595,5596,5598,5600,5602],{"class":1416,"line":1513},[1414,5597,2462],{"class":1420},[1414,5599,46],{"class":1424},[1414,5601,3844],{"class":1433},[1414,5603,2489],{"class":1424},[1414,5605,5606,5608],{"class":1416,"line":1541},[1414,5607,2486],{"class":1433},[1414,5609,2489],{"class":1424},[22,5611,5612],{},[1398,5613],{"alt":2529,"src":5614},"/assets/autogenerated/../pathway/docs/2.developers/7.templates/ETL/.interval_over_gaussian_filter/article_37_0.png",[22,5616,5617],{},"You can see that the gap has disappeared, replaced by the late data points and the associated smoothed signal.",[22,5619,5620],{},"Let's compare both outputs:",[1406,5622,5624],{"className":1408,"code":5623,"language":1410,"meta":152,"style":152},"plt.subplot(2, 1, 1)\nplt.plot(x_sample, y_sample, label=\"Total data points\")\nplt.plot(\n    first_results.x, first_results.y, \"x-\", color=\"#ff7f0e\", label=\"Original smoothing\"\n)\nplt.plot(late_results.x, late_results.y, \"x-\", color=\"red\", label=\"Updated smoothing\")\nplt.legend()\nset_params_plot()\n\n",[100,5625,5626,5648,5678,5688,5738,5742,5801,5811],{"__ignoreMap":152},[1414,5627,5628,5630,5632,5634,5636,5638,5640,5642,5644,5646],{"class":1416,"line":1417},[1414,5629,2462],{"class":1420},[1414,5631,46],{"class":1424},[1414,5633,2467],{"class":1433},[1414,5635,1634],{"class":1424},[1414,5637,2165],{"class":1500},[1414,5639,1504],{"class":1424},[1414,5641,2294],{"class":1500},[1414,5643,1504],{"class":1424},[1414,5645,2294],{"class":1500},[1414,5647,1592],{"class":1424},[1414,5649,5650,5652,5654,5656,5658,5660,5662,5664,5666,5668,5670,5672,5674,5676],{"class":1416,"line":153},[1414,5651,2462],{"class":1420},[1414,5653,46],{"class":1424},[1414,5655,2498],{"class":1433},[1414,5657,1634],{"class":1424},[1414,5659,3750],{"class":1433},[1414,5661,1504],{"class":1424},[1414,5663,3755],{"class":1433},[1414,5665,1504],{"class":1424},[1414,5667,3769],{"class":1570},[1414,5669,1425],{"class":1424},[1414,5671,2372],{"class":1424},[1414,5673,5524],{"class":2375},[1414,5675,2372],{"class":1424},[1414,5677,1592],{"class":1424},[1414,5679,5680,5682,5684,5686],{"class":1416,"line":1448},[1414,5681,2462],{"class":1420},[1414,5683,46],{"class":1424},[1414,5685,2498],{"class":1433},[1414,5687,1437],{"class":1424},[1414,5689,5690,5693,5695,5697,5699,5701,5703,5705,5707,5709,5711,5713,5715,5717,5719,5721,5723,5725,5727,5729,5731,5733,5736],{"class":1416,"line":1467},[1414,5691,5692],{"class":1433},"    first_results",[1414,5694,46],{"class":1424},[1414,5696,2108],{"class":1456},[1414,5698,1504],{"class":1424},[1414,5700,5326],{"class":1433},[1414,5702,46],{"class":1424},[1414,5704,2635],{"class":1456},[1414,5706,1504],{"class":1424},[1414,5708,2632],{"class":1424},[1414,5710,3762],{"class":2375},[1414,5712,2372],{"class":1424},[1414,5714,1504],{"class":1424},[1414,5716,3517],{"class":1570},[1414,5718,1425],{"class":1424},[1414,5720,2372],{"class":1424},[1414,5722,3524],{"class":2375},[1414,5724,2372],{"class":1424},[1414,5726,1504],{"class":1424},[1414,5728,3769],{"class":1570},[1414,5730,1425],{"class":1424},[1414,5732,2372],{"class":1424},[1414,5734,5735],{"class":2375},"Original smoothing",[1414,5737,5589],{"class":1424},[1414,5739,5740],{"class":1416,"line":1483},[1414,5741,1592],{"class":1424},[1414,5743,5744,5746,5748,5750,5752,5754,5756,5758,5760,5762,5764,5766,5768,5770,5772,5774,5776,5778,5780,5782,5784,5786,5788,5790,5792,5794,5797,5799],{"class":1416,"line":1513},[1414,5745,2462],{"class":1420},[1414,5747,46],{"class":1424},[1414,5749,2498],{"class":1433},[1414,5751,1634],{"class":1424},[1414,5753,5446],{"class":1433},[1414,5755,46],{"class":1424},[1414,5757,2108],{"class":1456},[1414,5759,1504],{"class":1424},[1414,5761,5441],{"class":1433},[1414,5763,46],{"class":1424},[1414,5765,2635],{"class":1456},[1414,5767,1504],{"class":1424},[1414,5769,2632],{"class":1424},[1414,5771,3762],{"class":2375},[1414,5773,2372],{"class":1424},[1414,5775,1504],{"class":1424},[1414,5777,3517],{"class":1570},[1414,5779,1425],{"class":1424},[1414,5781,2372],{"class":1424},[1414,5783,4389],{"class":2375},[1414,5785,2372],{"class":1424},[1414,5787,1504],{"class":1424},[1414,5789,3769],{"class":1570},[1414,5791,1425],{"class":1424},[1414,5793,2372],{"class":1424},[1414,5795,5796],{"class":2375},"Updated smoothing",[1414,5798,2372],{"class":1424},[1414,5800,1592],{"class":1424},[1414,5802,5803,5805,5807,5809],{"class":1416,"line":1541},[1414,5804,2462],{"class":1420},[1414,5806,46],{"class":1424},[1414,5808,3844],{"class":1433},[1414,5810,2489],{"class":1424},[1414,5812,5813,5815],{"class":1416,"line":1552},[1414,5814,2486],{"class":1433},[1414,5816,2489],{"class":1424},[22,5818,5819],{},[1398,5820],{"alt":2529,"src":5821},"/assets/autogenerated/../pathway/docs/2.developers/7.templates/ETL/.interval_over_gaussian_filter/article_39_0.png",[22,5823,5824],{},"The arrival of late data points has triggered an update of the output: not only the gap has been filled, but existing values have been updated to take into account the incoming data points.\nFurthermore, the smoothing has been updated only for data points close to the late points: Pathway did not apply the filter to the whole signal from scratch, but only updated the local parts that were impacted by the arrival of the late points.",[22,5826,5827,5828],{},"This is the power of Pathway: you can build your pipeline as if the data were static, and Pathway will take care of the late and out-of-order data for you.\n",[69,5829,5830],{},"Your results will always be up-to-date compared to the latest available data points.",[48,5832,5834],{"id":5833},"gaussian-filtering-in-python-conclusion","Gaussian Filtering in Python: Conclusion",[22,5836,5837],{},"Signal processing is a fundamental tool for extracting meaningful insights from data, but signal processing on real-world data requires data alignment.\nPathway provides all you need to tackle the challenges of real-world digital signal processing on data streams.\nNow that you understand the differences between idealized signal processing and data stream processing, I'm sure you can leverage Pathway's capabilities to process, filter, and analyze data efficiently, even when dealing with imperfect sampling rates and data imperfections.",[22,5839,5840,5841,5843,5844,5846],{},"If you want to learn more about DSP in Pathway, try our ",[29,5842,312],{"href":44}," about manipulating different data sources by ",[1892,5845,169],{}," a data source to match the sampling rate of given data source.",[5049,5848,5849],{},"html pre.shiki code .s0W1g, html code.shiki .s0W1g{--shiki-default:#BABED8}html pre.shiki code .sAklC, html code.shiki .sAklC{--shiki-default:#89DDFF}html pre.shiki code .sdLwU, html code.shiki .sdLwU{--shiki-default:#82AAFF}html pre.shiki code .s-wAU, html code.shiki .s-wAU{--shiki-default:#F07178}html pre.shiki code .sx098, html code.shiki .sx098{--shiki-default:#F78C6C}html pre.shiki code .s7ZW3, html code.shiki .s7ZW3{--shiki-default:#BABED8;--shiki-default-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s6cf3, html code.shiki .s6cf3{--shiki-default:#89DDFF;--shiki-default-font-style:italic}html pre.shiki code .saEQR, html code.shiki .saEQR{--shiki-default:#676E95;--shiki-default-font-style:italic}html pre.shiki code .sJ14y, html code.shiki .sJ14y{--shiki-default:#C792EA}html pre.shiki code .sfyAc, html code.shiki .sfyAc{--shiki-default:#C3E88D}html pre.shiki code .s5Dmg, html code.shiki .s5Dmg{--shiki-default:#FFCB6B}",{"title":152,"searchDepth":153,"depth":153,"links":5851},[5852,5853,5854,5861,5862],{"id":123,"depth":153,"text":124},{"id":1364,"depth":153,"text":1365},{"id":1875,"depth":153,"text":1876,"children":5855},[5856,5857,5859,5860],{"id":1913,"depth":1448,"text":1914},{"id":2724,"depth":1448,"text":5858},"Computing aggregations with windowby",{"id":2952,"depth":1448,"text":2953},{"id":3314,"depth":1448,"text":3315},{"id":3570,"depth":153,"text":3571},{"id":5833,"depth":153,"text":5834},"Tutorial on signal processing: how to apply a Gaussian filter with Pathway using windowby and intervals_over",{"notebook_export_path":5865,"aside":176,"date":5866,"layout":319,"thumbnail":5867,"tags":5869,"related":5871,"keywords":5872},"notebooks/tutorials/gaussian-filtering-python.ipynb","2023-10-17",{"src":5868},"/assets/content/tutorials/time_series/thumbnail-gaussian.png",[312,5870],"Time Series",[1330,44],[165,5873,5874,5875,1713,5876,237],"Gaussian filter","out-of-order data","irregular sampling","window",{"title":1318,"description":5863},{"loc":39},"2.developers/7.templates/ETL/100.gaussian_filtering_python","2V86AxylDmGIynzEejtz1rdVQChRYy7_gOQfen5w_4Q",[5882,5883],{"title":721,"path":722,"stem":723,"children":-1},{"title":736,"path":737,"stem":738,"children":-1},1775364317387]