DBOS Network Sensing: A Web Services Approach to Collaborative Awareness

2025-09-11

AI Summary

This paper addresses the lack of network-aware collaborative perception in Database Operating Systems (DBOS) deployed in distributed environments. To resolve this, we propose a lightweight, GraphBLAS-based hyper-sparse matrix mechanism for network awareness. Our approach enables real-time shared environmental awareness across multiple DBOS instances via web services—marking the first integration of collaborative perception with DBOS architecture. The system leverages Python-GraphBLAS and OneSparse PostgreSQL to construct parallelized network traffic matrices. Evaluated on the MIT SuperCloud 64-node cluster, it achieves >10⁵ QPS per DBOS instance, with negligible overhead from the perception module. Both Python-GraphBLAS and OneSparse variants demonstrate near-linear scalability—up to 64 and 32 nodes, respectively. The solution significantly enhances collective security, resilience, and real-time responsiveness in large-scale DBOS deployments.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

DBOS (DataBase Operating System) is a novel capability that integrates web services, operating system functions, and database features to significantly reduce web-deployment effort while increasing resilience. Integration of high performance network sensing enables DBOS web services to collaboratively create a shared awareness of their network environments to enhance their collective resilience and security. Network sensing is added to DBOS using GraphBLAS hypersparse traffic matrices via two approaches: (1) Python-GraphBLAS and (2) OneSparse PostgreSQL. These capabilities are demonstrated using the workflow and analytics from the IEEE/MIT/Amazon Anonymized Network Sensing Graph Challenge. The system was parallelized using pPython and benchmarked using 64 compute nodes on the MIT SuperCloud. The web request rate sustained by a single DBOS instance was ${>}10^5$, well above the required maximum, indicating that network sensing can be added to DBOS with negligible overhead. For collaborative awareness, many DBOS instances were connected to a single DBOS aggregator. The Python-GraphBLAS and OneSparse PostgreSQL implementations scaled linearly up to 64 and 32 nodes respectively. These results suggest that DBOS collaborative network awareness can be achieved with a negligible increase in computing resources.

Problem

• Integrating network sensing into DBOS for collaborative awareness
• Enhancing collective resilience and security via web services
• Scaling network sensing with GraphBLAS and PostgreSQL

Innovation

• Integrates web services with OS and database
• Uses GraphBLAS hypersparse traffic matrices
• Scalable Python and PostgreSQL implementations