This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application.
In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy.
We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.
The document discusses multidimensional databases. It defines multidimensional databases as systems designed to efficiently store and retrieve large volumes of related data that can be viewed from different perspectives or dimensions. It provides an example using automobile sales data that can be analyzed based on dimensions like model, color, dealership, and time. Multidimensional databases allow for interactive analysis of data across multiple dimensions, unlike relational databases which are less flexible and have slower response times for similar analyses.
API-Centric Data Integration for Human Genomics Reference Databases: Achieve...
API-Centric Data Integration for Human Genomics Reference Databases: Achievements, Lessons Learned and Challenges
X-Meeting 2015
Authors: Jamisson Freitas, Marcel Caraciolo, Victor Diniz, Rodrigo Alexandre and João Bosco Oliveira
This project aims to develop an algorithm to detect duplicate records in a dataset. The proposed algorithm will preprocess data, identify potential duplicates using similarity measures, and cluster records into groups for review and removal. It will use Java and existing libraries for analysis. The goal is an accurate and efficient algorithm that can be integrated into data management systems to improve data quality. The document discusses the need to address duplicate records, a literature review of existing techniques, issues with current algorithms, and the proposed approach which is to create a website that takes in strings and uses an MD5 hash-based algorithm to identify duplicates in a MongoDB database.
This document discusses how SAP's in-memory database HANA could help address challenges in pharmaceutical research and development. It outlines data management challenges in pharma R&D, how HANA addresses these challenges through its real-time analytics capabilities, and potential use cases for HANA in areas like genomics, clinical trials, and drug development. The document proposes starting a discussion between pharma R&D and SAP/HPI to explore how HANA could help solve existing problems or future challenges.
Apache Spark is an open-source distributed processing system used for big data workloads. It utilizes in-memory caching and optimized queries for fast analytics of large data. Apache Storm is a distributed real-time processing system designed for high data ingestion rates. Both Spark and Storm support multiple languages and real-time streaming.
Nikhil Sharma has a Master of Science in Data Informatics from USC and a Bachelor of Engineering in Electronics and Communication from M.S. Ramaiah Institute of Technology in India. He has work experience as a Software Engineer Intern at Salesforce where he designed applications for database performance analysis using Python. Previously he was a Senior Systems Engineer at Infosys where he implemented IT infrastructure for banks in various countries. His academic projects involve machine learning, data mining, and information retrieval using technologies like Python, Solr, and Caffe.
Abdul Ahad Abro presented on data science, predictive analytics, machine learning algorithms, regression, classification, Microsoft Azure Machine Learning Studio, and academic publications. The presentation introduced key concepts in data science including machine learning, predictive analytics, regression, classification, and algorithms. It demonstrated regression analysis using Microsoft Azure Machine Learning Studio and Microsoft Excel. The methodology section described using a dataset from Azure for classification and linear regression in both Azure and Excel to compare results.
Bioinformatics may be defined as the field of science
in which biology, computer science, and information
technology merge to form a single discipline. Its ultimate
goal is to enable the discovery of new biological insights as
well as to create a global perspective from which unifying
principles in biology can be discerned by means of
bioinformatics tools for storing, retrieving, organizing and
analyzing biological data. Also most of these tools possess
very distinct features and capabilities making a direct
comparison difficult to be done. In this paper we propose
taxonomy for characterizing bioinformatics tools and briefly
surveys major bioinformatics tools under each categories.
Hopefully this study will stimulate other designers
and
experienced end users understand the details of particular
tool categories/tools, enabling them to make the best choices
for their particular research interests.
Comparison of machine learning methods for breast cancer diagnosis
The document compares machine learning methods SVM and ANN for breast cancer diagnosis. Both algorithms are trained on a dataset containing patient data and class labels. SVM and ANN generate models to predict class labels for new test data. SVM achieved a higher prediction accuracy of 66.66% compared to 38% for ANN, showing SVM is better for this task. Screenshots of a program demonstrate uploading data, training models, predicting results, and comparing algorithm accuracy graphically.
Hadoop clusters can store nearly everything in a cheap and blazingly fast way to your data lake. Answering questions and gaining insights out of this ever growing stream becomes the decisive part for many businesses. Increasingly data has a natural structure as a graph, with vertices linked by edges, and many questions arising about the data involve graph traversals or other complex queries, for which one does not have an a priori given bound on the length of paths.
Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...
The document discusses using a graph database to store and query graph data stored in a Hadoop data lake more efficiently. It describes the limitations of the typical approach of using Spark/GraphFrames on HDFS for graph queries. A graph database allows for faster ad hoc graph queries by leveraging graph traversals. The document proposes using a multi-model database that combines a document store, graph database, and key-value store with a common query language. It suggests this approach could run on a DC/OS cluster for easy deployment and management of resources. Examples show importing data into ArangoDB and running graph queries.
This document compares the performance of various machine learning and classification algorithms, including neural networks, support vector machines, Naive Bayes, decision trees, and decision stumps. It analyzes these algorithms using a dataset of annual and monthly temperature data from India over 1901-2012. The analysis is conducted in RapidMiner and finds that neural networks and support vector machines can effectively model complex nonlinear relationships to predict temperature. Neural networks achieved reasonably accurate predictions of annual temperature compared to the original data values. The document concludes by comparing the performance of the different algorithms.
A WEB REPOSITORY SYSTEM FOR DATA MINING IN DRUG DISCOVERY
This project is to produce a repository database system of drugs, drug features (properties), and drug targets where data can be mined and analyzed. Drug targets are different proteins that drugs try to bind to stop the activities of the protein. Users can utilize the database to mine useful data to predict the specific chemical properties that will have the relative efficacy of a specific target and the coefficient for each chemical property. This database system can be equipped with different data mining approaches/algorithms such as linear, non-linear, and classification types of data modelling. The data models have enhanced with the Genetic Evolution (GE) algorithms. This paper discusses implementation with the linear data models such as Multiple Linear Regression (MLR), Partial Least Square Regression (PLSR), and Support Vector Machine (SVM).
The increased availability of biomedical data, particularly in the public domain, offers the opportunity to better understand human health and to develop effective therapeutics for a wide range of unmet medical needs. However, data scientists remain stymied by the fact that data remain hard to find and to productively reuse because data and their metadata i) are wholly inaccessible, ii) are in non-standard or incompatible representations, iii) do not conform to community standards, and iv) have unclear or highly restricted terms and conditions that preclude legitimate reuse. These limitations require a rethink on data can be made machine and AI-ready - the key motivation behind the FAIR Guiding Principles. Concurrently, while recent efforts have explored the use of deep learning to fuse disparate data into predictive models for a wide range of biomedical applications, these models often fail even when the correct answer is already known, and fail to explain individual predictions in terms that data scientists can appreciate. These limitations suggest that new methods to produce practical artificial intelligence are still needed.
In this talk, I will discuss our work in (1) building an integrative knowledge infrastructure to prepare FAIR and "AI-ready" data and services along with (2) neurosymbolic AI methods to improve the quality of predictions and to generate plausible explanations. Attention is given to standards, platforms, and methods to wrangle knowledge into simple, but effective semantic and latent representations, and to make these available into standards-compliant and discoverable interfaces that can be used in model building, validation, and explanation. Our work, and those of others in the field, creates a baseline for building trustworthy and easy to deploy AI models in biomedicine.
Bio
Dr. Michel Dumontier is the Distinguished Professor of Data Science at Maastricht University, founder and executive director of the Institute of Data Science, and co-founder of the FAIR (Findable, Accessible, Interoperable and Reusable) data principles. His research explores socio-technological approaches for responsible discovery science, which includes collaborative multi-modal knowledge graphs, privacy-preserving distributed data mining, and AI methods for drug discovery and personalized medicine. His work is supported through the Dutch National Research Agenda, the Netherlands Organisation for Scientific Research, Horizon Europe, the European Open Science Cloud, the US National Institutes of Health, and a Marie-Curie Innovative Training Network. He is the editor-in-chief for the journal Data Science and is internationally recognized for his contributions in bioinformatics, biomedical informatics, and semantic technologies including ontologies and linked data.
The document provides an overview of Apache Spark from the perspective of two data scientists, Volkan and Richard. It discusses how Spark has enabled real-time analytics and simplified their work by allowing them to more easily access, analyze, and integrate data from various sources using languages like Python and R. Both data scientists note that Spark has significantly increased their productivity and helped transform their batch processing models into real-time tools.
The document discusses multidimensional databases. It defines multidimensional databases as systems designed to efficiently store and retrieve large volumes of related data that can be viewed from different perspectives or dimensions. It provides an example using automobile sales data that can be analyzed based on dimensions like model, color, dealership, and time. Multidimensional databases allow for interactive analysis of data from multiple angles, unlike relational databases that are slower for such analyses.
The document discusses multidimensional databases. It defines multidimensional databases as systems designed to efficiently store and retrieve large volumes of related data that can be viewed from different perspectives or dimensions. It provides an example using automobile sales data that can be analyzed based on dimensions like model, color, dealership, and time. Multidimensional databases allow for interactive analysis of data across multiple dimensions, unlike relational databases which are less flexible and have slower response times for similar analyses.
API-Centric Data Integration for Human Genomics Reference Databases: Achieve...Genomika Diagnósticos
API-Centric Data Integration for Human Genomics Reference Databases: Achievements, Lessons Learned and Challenges
X-Meeting 2015
Authors: Jamisson Freitas, Marcel Caraciolo, Victor Diniz, Rodrigo Alexandre and João Bosco Oliveira
This project aims to develop an algorithm to detect duplicate records in a dataset. The proposed algorithm will preprocess data, identify potential duplicates using similarity measures, and cluster records into groups for review and removal. It will use Java and existing libraries for analysis. The goal is an accurate and efficient algorithm that can be integrated into data management systems to improve data quality. The document discusses the need to address duplicate records, a literature review of existing techniques, issues with current algorithms, and the proposed approach which is to create a website that takes in strings and uses an MD5 hash-based algorithm to identify duplicates in a MongoDB database.
How SAP HANA can provide value for Pharma R&DMarc Maurer
This document discusses how SAP's in-memory database HANA could help address challenges in pharmaceutical research and development. It outlines data management challenges in pharma R&D, how HANA addresses these challenges through its real-time analytics capabilities, and potential use cases for HANA in areas like genomics, clinical trials, and drug development. The document proposes starting a discussion between pharma R&D and SAP/HPI to explore how HANA could help solve existing problems or future challenges.
Apache Spark is an open-source distributed processing system used for big data workloads. It utilizes in-memory caching and optimized queries for fast analytics of large data. Apache Storm is a distributed real-time processing system designed for high data ingestion rates. Both Spark and Storm support multiple languages and real-time streaming.
Nikhil Sharma has a Master of Science in Data Informatics from USC and a Bachelor of Engineering in Electronics and Communication from M.S. Ramaiah Institute of Technology in India. He has work experience as a Software Engineer Intern at Salesforce where he designed applications for database performance analysis using Python. Previously he was a Senior Systems Engineer at Infosys where he implemented IT infrastructure for banks in various countries. His academic projects involve machine learning, data mining, and information retrieval using technologies like Python, Solr, and Caffe.
Abdul Ahad Abro presented on data science, predictive analytics, machine learning algorithms, regression, classification, Microsoft Azure Machine Learning Studio, and academic publications. The presentation introduced key concepts in data science including machine learning, predictive analytics, regression, classification, and algorithms. It demonstrated regression analysis using Microsoft Azure Machine Learning Studio and Microsoft Excel. The methodology section described using a dataset from Azure for classification and linear regression in both Azure and Excel to compare results.
Bioinformatics may be defined as the field of science
in which biology, computer science, and information
technology merge to form a single discipline. Its ultimate
goal is to enable the discovery of new biological insights as
well as to create a global perspective from which unifying
principles in biology can be discerned by means of
bioinformatics tools for storing, retrieving, organizing and
analyzing biological data. Also most of these tools possess
very distinct features and capabilities making a direct
comparison difficult to be done. In this paper we propose
taxonomy for characterizing bioinformatics tools and briefly
surveys major bioinformatics tools under each categories.
Hopefully this study will stimulate other designers
and
experienced end users understand the details of particular
tool categories/tools, enabling them to make the best choices
for their particular research interests.
Comparison of machine learning methods for breast cancer diagnosisVenkat Projects
The document compares machine learning methods SVM and ANN for breast cancer diagnosis. Both algorithms are trained on a dataset containing patient data and class labels. SVM and ANN generate models to predict class labels for new test data. SVM achieved a higher prediction accuracy of 66.66% compared to 38% for ANN, showing SVM is better for this task. Screenshots of a program demonstrate uploading data, training models, predicting results, and comparing algorithm accuracy graphically.
Hadoop clusters can store nearly everything in a cheap and blazingly fast way to your data lake. Answering questions and gaining insights out of this ever growing stream becomes the decisive part for many businesses. Increasingly data has a natural structure as a graph, with vertices linked by edges, and many questions arising about the data involve graph traversals or other complex queries, for which one does not have an a priori given bound on the length of paths.
Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...Big Data Spain
The document discusses using a graph database to store and query graph data stored in a Hadoop data lake more efficiently. It describes the limitations of the typical approach of using Spark/GraphFrames on HDFS for graph queries. A graph database allows for faster ad hoc graph queries by leveraging graph traversals. The document proposes using a multi-model database that combines a document store, graph database, and key-value store with a common query language. It suggests this approach could run on a DC/OS cluster for easy deployment and management of resources. Examples show importing data into ArangoDB and running graph queries.
This document compares the performance of various machine learning and classification algorithms, including neural networks, support vector machines, Naive Bayes, decision trees, and decision stumps. It analyzes these algorithms using a dataset of annual and monthly temperature data from India over 1901-2012. The analysis is conducted in RapidMiner and finds that neural networks and support vector machines can effectively model complex nonlinear relationships to predict temperature. Neural networks achieved reasonably accurate predictions of annual temperature compared to the original data values. The document concludes by comparing the performance of the different algorithms.
A WEB REPOSITORY SYSTEM FOR DATA MINING IN DRUG DISCOVERYIJDKP
This project is to produce a repository database system of drugs, drug features (properties), and drug targets where data can be mined and analyzed. Drug targets are different proteins that drugs try to bind to stop the activities of the protein. Users can utilize the database to mine useful data to predict the specific chemical properties that will have the relative efficacy of a specific target and the coefficient for each chemical property. This database system can be equipped with different data mining approaches/algorithms such as linear, non-linear, and classification types of data modelling. The data models have enhanced with the Genetic Evolution (GE) algorithms. This paper discusses implementation with the linear data models such as Multiple Linear Regression (MLR), Partial Least Square Regression (PLSR), and Support Vector Machine (SVM).
A WEB REPOSITORY SYSTEM FOR DATA MINING IN DRUG DISCOVERYIJDKP
This project is to produce a repository database system of drugs, drug features (properties), and drug targets where data can be mined and analyzed. Drug targets are different proteins that drugs try to bind to stop the activities of the protein. Users can utilize the database to mine useful data to predict the specific chemical properties that will have the relative efficacy of a specific target and the coefficient for each chemical property. This database system can be equipped with different data mining approaches/algorithms such as linear, non-linear, and classification types of data modelling. The data models have enhanced with the Genetic Evolution (GE) algorithms. This paper discusses implementation with the linear data models such as Multiple Linear Regression (MLR), Partial Least Square Regression (PLSR), and Support Vector Machine (SVM).
Comparing EDA with classical and Bayesian analysis.pptxPremaGanesh1
This document provides an overview of exploratory data analysis (EDA) techniques and commonly used tools. It discusses classical and Bayesian statistical analysis approaches as well as EDA. Popular Python libraries for EDA include NumPy, Pandas, Matplotlib and Seaborn. NumPy allows working with multidimensional arrays and matrices while Pandas facilitates working with structured data. The document also provides examples of creating arrays and dataframes, loading data from files, and analyzing datasets using these tools.
Similar to "NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba (20)
"What does it really mean for your system to be available, or how to define w...Fwdays
We will talk about system monitoring from a few different angles. We will start by covering the basics, then discuss SLOs, how to define them, and why understanding the business well is crucial for success in this exercise.
"Microservices and multitenancy - how to serve thousands of databases in one ...Fwdays
Imagine you are designing a B2B service that will serve millions of businesses. This service will have dozens of different microservices with their own data, which can contain millions of records. How do you design such a database? Why is sharding not always the answer? What other options are there for such an architectural solution?
I'll tell you how we at Uspacy came to serve thousands of small databases instead of a few large ones, what we've encountered and what we plan to face)
"Scaling RAG Applications to serve millions of users", Kevin GoedeckeFwdays
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
"Black Monday: The Story of 5.5 Hours of Downtime", Dmytro DziubenkoFwdays
We will explore the most significant incident in our product's history. We'll discuss the causes that led to the failure, how our team responded, and the measures we took to prevent future incidents. Special attention will be paid to identifying the root cause of the incident and the role of the VACUUM mechanism in PostgreSQL.
"Reaching 3_000_000 HTTP requests per second — conclusions from participation...Fwdays
In this talk, we will get acquainted with TechEmpower Web Framework Benchmarks, consider generalized (programming language-independent) approaches to optimizing a web application and its environment to achieve extreme loads, and most importantly, how some of these things can be applied in practice in your projects.
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...Fwdays
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
"Choosing proper type of scaling", Olena SyrotaFwdays
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
"What I learned through reverse engineering", Yuri ArtiukhFwdays
In recent years, I have gained most of my knowledge through reverse engineering, how I did it and what I learned during this period, I decided to share. All this concerns graphic programming, performance, best practices in the frontend.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
"Micro frontends: Unbelievably true life story", Dmytro PavlovFwdays
A real life story about the experience of using Micro frontends in an existing Enterprise product. Problems and their solutions on the way from the integration of a separate component to an extensible No-code platform.
"Objects validation and comparison using runtime types (io-ts)", Oleksandr SuhakFwdays
A common task in modern JS is parsing, validating and then comparing JSON objects. In this talk I will quickly go through most common ways to parse/validate and compare objects we use today and then focus more on how runtime types (based on io-ts) can help make such tasks easier and quicker to implement.
"JavaScript. Standard evolution, when nobody cares", Roman SavitskyiFwdays
Should we take a look at JavaScript when everyone is writing in TypeScript? What happens to the standard? What did we get last year? What new features can we expect this and next year? And most importantly, when will Observer be standardized?
Let's try to answer all these questions and even a little more, dream about the future, and enjoy that Observer is alive (or not).
"How Preply reduced ML model development time from 1 month to 1 day",Yevhen Y...Fwdays
Case study of how small team in Preply started with inheriting an existing ranking model to being able to produce a model per day. In this talk we'll cover steps to take if you find yourself in a similar situation: what kind of technology and processes can you introduce in order to achieve a great speedup in a development speed.
"GenAI Apps: Our Journey from Ideas to Production Excellence",Danil TopchiiFwdays
In my talk, I will tell about the world of GenAI services beyond GPT-wrappers and how we developed and scaled GenAI-centric applications. I'll share personal experiences about the obstacles, lessons, and strategic tools and methodologies that were key in taking GenAI applications from 0 to 1. I'll talk about the challenges we faced when launching LLM-based and image generative applications and delivering them to end users, and what conclusions and solutions were made.
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
Python engineers are introduced to the transformative potential of Large Language Models (LLMs) in the realm of advanced data analysis and the application of Semantic Kernel techniques. We will talk about how LLMs like ChatGPT can be integrated into Python environments to automate data processing, enhance predictive modeling, and unlock deeper insights from complex datasets. The session will delve into practical strategies for embedding Semantic Kernel methods within Python projects, illustrating how these advanced techniques can refine the accuracy of machine learning models by embedding domain-specific knowledge directly into the analysis process. Attendees will leave with a clear roadmap for leveraging the combined power of LLMs and Semantic Kernels, equipped with actionable knowledge to drive innovation in their data analysis projects and beyond, marking a significant leap forward in the evolution of Python engineering practices.
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
Federated learning. Algorithmic solution to the problem of privacy preserving ML. Pieces involved to support the training with NVIDIA Flare as example. How newest legislation affects federated learning.
"What is a RAG system and how to build it",Dmytro SpodaretsFwdays
Today, large language models are becoming an integral part of almost every IT solution. However, their use is often accompanied by certain limitations, such as the relevance of information or its depth and specificity. One of the ways to overcome these limitations is the method of working with LLMs - RAG (Retrieval Augmented Generation).
In an ideal world, you would write Python code and then it would work perfectly. But unfortunately, it doesn't work in this manner. In my talk, I'll cover how to efficiently debug your programs, especially in cloud environments or inside Kubernetes.
MLOps (Machine Learning Operations) is a recent buzzword, that trends a lot. Let's figure out together how maintaining applications with machine learning components is significantly different from maintaining applications without them.
We will look into MLOps best practices and typical problems and their implementations/solutions in real world production.
Blockchain technology is transforming industries and reshaping the way we conduct business, manage data, and secure transactions. Whether you're new to blockchain or looking to deepen your knowledge, our guidebook, "Blockchain for Dummies", is your ultimate resource.
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfNeo4j
Presented at Gartner Data & Analytics, London Maty 2024. BT Group has used the Neo4j Graph Database to enable impressive digital transformation programs over the last 6 years. By re-imagining their operational support systems to adopt self-serve and data lead principles they have substantially reduced the number of applications and complexity of their operations. The result has been a substantial reduction in risk and costs while improving time to value, innovation, and process automation. Join this session to hear their story, the lessons they learned along the way and how their future innovation plans include the exploration of uses of EKG + Generative AI.
The Rise of Supernetwork Data Intensive ComputingLarry Smarr
Invited Remote Lecture to SC21
The International Conference for High Performance Computing, Networking, Storage, and Analysis
St. Louis, Missouri
November 18, 2021
Quantum Communications Q&A with Gemini LLM. These are based on Shannon's Noisy channel Theorem and offers how the classical theory applies to the quantum world.
Mitigating the Impact of State Management in Cloud Stream Processing SystemsScyllaDB
Stream processing is a crucial component of modern data infrastructure, but constructing an efficient and scalable stream processing system can be challenging. Decoupling compute and storage architecture has emerged as an effective solution to these challenges, but it can introduce high latency issues, especially when dealing with complex continuous queries that necessitate managing extra-large internal states.
In this talk, we focus on addressing the high latency issues associated with S3 storage in stream processing systems that employ a decoupled compute and storage architecture. We delve into the root causes of latency in this context and explore various techniques to minimize the impact of S3 latency on stream processing performance. Our proposed approach is to implement a tiered storage mechanism that leverages a blend of high-performance and low-cost storage tiers to reduce data movement between the compute and storage layers while maintaining efficient processing.
Throughout the talk, we will present experimental results that demonstrate the effectiveness of our approach in mitigating the impact of S3 latency on stream processing. By the end of the talk, attendees will have gained insights into how to optimize their stream processing systems for reduced latency and improved cost-efficiency.
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc
Six months into 2024, and it is clear the privacy ecosystem takes no days off!! Regulators continue to implement and enforce new regulations, businesses strive to meet requirements, and technology advances like AI have privacy professionals scratching their heads about managing risk.
What can we learn about the first six months of data privacy trends and events in 2024? How should this inform your privacy program management for the rest of the year?
Join TrustArc, Goodwin, and Snyk privacy experts as they discuss the changes we’ve seen in the first half of 2024 and gain insight into the concrete, actionable steps you can take to up-level your privacy program in the second half of the year.
This webinar will review:
- Key changes to privacy regulations in 2024
- Key themes in privacy and data governance in 2024
- How to maximize your privacy program in the second half of 2024
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionBert Blevins
Cybersecurity is a major concern in today's connected digital world. Threats to organizations are constantly evolving and have the potential to compromise sensitive information, disrupt operations, and lead to significant financial losses. Traditional cybersecurity techniques often fall short against modern attackers. Therefore, advanced techniques for cyber security analysis and anomaly detection are essential for protecting digital assets. This blog explores these cutting-edge methods, providing a comprehensive overview of their application and importance.
Quality Patents: Patents That Stand the Test of TimeAurora Consulting
Is your patent a vanity piece of paper for your office wall? Or is it a reliable, defendable, assertable, property right? The difference is often quality.
Is your patent simply a transactional cost and a large pile of legal bills for your startup? Or is it a leverageable asset worthy of attracting precious investment dollars, worth its cost in multiples of valuation? The difference is often quality.
Is your patent application only good enough to get through the examination process? Or has it been crafted to stand the tests of time and varied audiences if you later need to assert that document against an infringer, find yourself litigating with it in an Article 3 Court at the hands of a judge and jury, God forbid, end up having to defend its validity at the PTAB, or even needing to use it to block pirated imports at the International Trade Commission? The difference is often quality.
Quality will be our focus for a good chunk of the remainder of this season. What goes into a quality patent, and where possible, how do you get it without breaking the bank?
** Episode Overview **
In this first episode of our quality series, Kristen Hansen and the panel discuss:
⦿ What do we mean when we say patent quality?
⦿ Why is patent quality important?
⦿ How to balance quality and budget
⦿ The importance of searching, continuations, and draftsperson domain expertise
⦿ Very practical tips, tricks, examples, and Kristen’s Musts for drafting quality applications
https://www.aurorapatents.com/patently-strategic-podcast.html
INDIAN AIR FORCE FIGHTER PLANES LIST.pdfjackson110191
These fighter aircraft have uses outside of traditional combat situations. They are essential in defending India's territorial integrity, averting dangers, and delivering aid to those in need during natural calamities. Additionally, the IAF improves its interoperability and fortifies international military alliances by working together and conducting joint exercises with other air forces.
Best Practices for Effectively Running dbt in Airflow.pdfTatiana Al-Chueyr
As a popular open-source library for analytics engineering, dbt is often used in combination with Airflow. Orchestrating and executing dbt models as DAGs ensures an additional layer of control over tasks, observability, and provides a reliable, scalable environment to run dbt models.
This webinar will cover a step-by-step guide to Cosmos, an open source package from Astronomer that helps you easily run your dbt Core projects as Airflow DAGs and Task Groups, all with just a few lines of code. We’ll walk through:
- Standard ways of running dbt (and when to utilize other methods)
- How Cosmos can be used to run and visualize your dbt projects in Airflow
- Common challenges and how to address them, including performance, dependency conflicts, and more
- How running dbt projects in Airflow helps with cost optimization
Webinar given on 9 July 2024
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...Erasmo Purificato
Slide of the tutorial entitled "Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Emerging Trends" held at UMAP'24: 32nd ACM Conference on User Modeling, Adaptation and Personalization (July 1, 2024 | Cagliari, Italy)
Are you interested in dipping your toes in the cloud native observability waters, but as an engineer you are not sure where to get started with tracing problems through your microservices and application landscapes on Kubernetes? Then this is the session for you, where we take you on your first steps in an active open-source project that offers a buffet of languages, challenges, and opportunities for getting started with telemetry data.
The project is called openTelemetry, but before diving into the specifics, we’ll start with de-mystifying key concepts and terms such as observability, telemetry, instrumentation, cardinality, percentile to lay a foundation. After understanding the nuts and bolts of observability and distributed traces, we’ll explore the openTelemetry community; its Special Interest Groups (SIGs), repositories, and how to become not only an end-user, but possibly a contributor.We will wrap up with an overview of the components in this project, such as the Collector, the OpenTelemetry protocol (OTLP), its APIs, and its SDKs.
Attendees will leave with an understanding of key observability concepts, become grounded in distributed tracing terminology, be aware of the components of openTelemetry, and know how to take their first steps to an open-source contribution!
Key Takeaways: Open source, vendor neutral instrumentation is an exciting new reality as the industry standardizes on openTelemetry for observability. OpenTelemetry is on a mission to enable effective observability by making high-quality, portable telemetry ubiquitous. The world of observability and monitoring today has a steep learning curve and in order to achieve ubiquity, the project would benefit from growing our contributor community.
論文紹介：A Systematic Survey of Prompt Engineering on Vision-Language Foundation ...Toru Tamaki
Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr "A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models" arXiv2023
https://arxiv.org/abs/2307.12980
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Chris Swan
Have you noticed the OpenSSF Scorecard badges on the official Dart and Flutter repos? It's Google's way of showing that they care about security. Practices such as pinning dependencies, branch protection, required reviews, continuous integration tests etc. are measured to provide a score and accompanying badge.
You can do the same for your projects, and this presentation will show you how, with an emphasis on the unique challenges that come up when working with Dart and Flutter.
The session will provide a walkthrough of the steps involved in securing a first repository, and then what it takes to repeat that process across an organization with multiple repos. It will also look at the ongoing maintenance involved once scorecards have been implemented, and how aspects of that maintenance can be better automated to minimize toil.
Measuring the Impact of Network Latency at TwitterScyllaDB
Widya Salim and Victor Ma will outline the causal impact analysis, framework, and key learnings used to quantify the impact of reducing Twitter's network latency.
Measuring the Impact of Network Latency at Twitter
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
1. ADVANCING DRUG SEARCH
WITH POSTGRESQL AND
AZURE AI
Taras Kloba
Senior Solutions Architect, Data & AI at
Microsoft
POSETTE - 2024
2. PANACEA
Taras Kloba
Senior Solutions Architect, Data & AI at Microsoft
• Founded a volunteering IT group for Ukrainian army
systems, winner of the TIDE NATO Hackathon and
Ukraine Defence Hackathon.
• Co-leader of PostgreSQL Ukraine and Big Data
Community
• More than 14 years of experience in Data
Engineering
• Ph.D. in Economics
• Winner of the Ukrainian IT Awards 2019 in Software
Architecture
• Certified Cloud Architect & Data Engineer on Google
Cloud, Microsoft, and Amazon Web Services.
Microsoft Certified Trainer
• Father of three daughters
4. PANACEA
Medication Match-Up.
When a new war starts, NATO troops
from different countries unite, each
with their medical supplies. Our
challenge was to make an app that
helps find the same medicine
across different country brands.
4
The Suwalki Gap: A Hypothetical Flashpoint
PANACEA
6. PANACEA
Inna Olkhova.
The diagram shows our app's ability to
match different medicine brands
through their active ingredient,
Acetaminophen. Inna’s idea helps find
the same medicine, whether it's Tylenol
in the US, Panadol in the UK,
Paracetamol in the EU, or Dafalgan in
France.
known as the "Bee Paramedic"
6
7. PANACEA
Tata Kepler.
From Artistry to Aid: Shaping Accessible Care in the
Field
7
Tata showed us how medicine sorting works,
especially when volunteers might not know medical
terms. She introduced us to using the ATC system in
our app, making it easier for everyone to find and
organize medicines correctly.
8. PANACEA
Daryna Smolnikova.
Daryna shared the idea of using NFC,
QR codes, and short links to store and
share information about the wounded.
She emphasized that sometimes,
sharing this information, even with the
enemy, is crucial to saving lives.
Immediate Action: Key to Battlefield Survival
8
13. PANACEA 13
ILIKE.
The ILIKE operator in PostgreSQL is used for
case-insensitive pattern matching. It
functions similarly to the LIKE operator but
ignores the case of the character
SELECT *
FROM drugs
WHERE drug_name ILIKE
'tylen%';
SELECT *
FROM drugs
WHERE drug_name ILIKE
'%tylen%';
14. PANACEA 14
TRIGRAMS.
A trigram is a group of three consecutive
characters in a string. By comparing these
trigrams, PostgreSQL can measure the
similarity between strings. Useful for finding
similar strings despite minor spelling errors
and variations.
CREATE EXTENSION pg_trgm;
SELECT drug_name
FROM drugs
WHERE drug_name % 'tylelon'
ORDER BY similarity(drug_name, 'tylelon')
DESC;
15. PANACEA 15
VECTOR SEARCH.
Vector search uses vector representations of text to
find similar items. By comparing these vectors,
PostgreSQL can measure the similarity between text
descriptions.
CREATE EXTENSION azure_ai;
ALTER TABLE drugs
ADD COLUMN vector VECTOR GENERATED ALWAYS AS
(azure_ai.vectorize(drug_name)) STORED;
SELECT drug_name
FROM drugs
ORDER BY vector <=> azure_ai.vectorize('pill for
pain management')
LIMIT 5;
16. PANACEA 16
VECTOR SEARCH.
Vector search uses vector representations of text to
find similar items. By comparing these vectors,
PostgreSQL can measure the similarity between text
descriptions.
CREATE EXTENSION azure_ai;
ALTER TABLE drugs
ADD COLUMN vector VECTOR GENERATED ALWAYS AS
(azure_ai.vectorize(drug_name)) STORED;
SELECT drug_name
FROM drugs
ORDER BY vector <=> azure_ai.vectorize('red pill for
headaches')
LIMIT 5;
17. PANACEA 17
DRUGGPT (AI RAG ASSSISTANT).
RAG (Retrieval-Augmented Generation) is a hybrid
approach that combines retrieval and generation
techniques to enhance the quality of AI responses. It
first retrieves relevant information from a knowledge
base or database and then uses this information to
generate more accurate and contextually relevant
answers.
18. PANACEA 18
Search.
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7
Machine learning models
for semantic
representation, ranking
for the rest of results.
Exact and pattern text
search (ilike%), trigram
similarity calculation,
and ranking for best
results.
Similarity search with
threshold 0.7, ranking for
best results.
Machine learning models
for semantic
representation, with a
threshold of 0.9, ranking
for best results.
Pattern text search
(%ilike%), trigram
similarity calculation,
ranking for best results.
Full-text search in
summary, description, and
indication columns.
Similarity search with
threshold 0.5, ranking for
best results.
Yes
No
Is result
found?
Yes
No
Is result
found?
Yes
No
Is result
found?
Yes
No
Is result
found?
Yes
No
Is result
found?
Yes
Is result
found?
No
