Content
# Project Synapse 🧠
> **🌟 Multi-Branch Repository**: This repository showcases Project Synapse in two complementary implementations:
> - 🎓 **[`educational-simulation`](https://github.com/YanCotta/project-synapse/tree/educational-simulation)**: A comprehensive educational simulation with detailed documentation, perfect for understanding multi-agent architecture concepts without running live services.
> - 🚀 **[`working-app`](https://github.com/YanCotta/project-synapse/tree/working-app)**: A fully functional, production-ready implementation with Docker, Kubernetes, and monitoring. **You are currently viewing this branch.**
**A production-ready multi-agent system showcasing Agent Communication Protocol (ACP) and Model Context Protocol (MCP) capabilities through a collaborative research workflow.**
> *"Where artificial intelligence meets production-grade architecture"*
## 🎯 Project Overview
Project Synapse is a comprehensive multi-agent system built with modern async Python, featuring specialized agents that work together to investigate complex research questions. The system demonstrates advanced patterns in agent communication, secure tool integration, and production deployment practices.
### Key Highlights
- **🚀 Production-Ready**: Docker containerization with resource management and health monitoring
- **⚡ High Performance**: Async/await architecture with measured performance metrics
- **🔒 Security-First**: MCP Roots implementation with filesystem access controls
- **📊 Observable**: Comprehensive logging and real-time monitoring
- **🏗️ Scalable**: RabbitMQ message bus with connection pooling
## 🌟 Architecture Features
### Production Infrastructure
| Component | Technology | Performance | Status |
|-----------|------------|-------------|---------|
| **HTTP Servers** | FastAPI v0.104.1 | 1,447 RPS file ops | ✅ Optimized |
| **Message Bus** | RabbitMQ 3.13.7 | 557 RPS API calls | ✅ High Availability |
| **Containerization** | Docker Compose | <131MB per service | ✅ Resource Managed |
| **Agent Coordination** | Async Python | 19.8 RPS search ops | ✅ Production Ready |
### Agent Ecosystem
```text
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Orchestrator │◄──►│ Search Agent │◄──►│ Extraction Agent│
│ (Coordinator) │ │ (Discovery) │ │ (Processing) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Fact Checker │ │ Synthesis │ │ File Save │
│ (Validation) │ │ (Generation) │ │ (Storage) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
▼
┌─────────────────┐
│ Logger Agent │
│ (Monitoring) │
└─────────────────┘
```
## � Quick Start
### Prerequisites
- Docker and Docker Compose
- Git
### Production Deployment
```bash
# Clone the repository
git clone https://github.com/YanCotta/project-synapse.git
cd project-synapse
# Deploy with optimized configuration (includes monitoring)
docker-compose -f docker-compose.optimized.yml up --build
# Monitor system performance
python scripts/monitor_system.py
```
### Kubernetes Deployment
```bash
# Deploy to Kubernetes cluster
./k8s/deploy.sh
# Access monitoring
kubectl port-forward svc/grafana 3000:3000
kubectl port-forward svc/prometheus 9090:9090
```
### Development Setup
```bash
# Alternative: Local development
pip install -r requirements.txt
python async_main.py
```
## 📊 Performance Metrics
### Measured Performance (Production Testing)
| Metric | Primary Server | Filesystem Server | RabbitMQ |
|--------|---------------|-------------------|----------|
| **Response Time** | 0.9ms avg | 0.8ms avg | 1.4ms avg |
| **P95 Latency** | 3.5ms | 2.1ms | 2.6ms |
| **Throughput** | 668.4 RPS | 882.7 RPS | 557 RPS |
| **Success Rate** | 100% | 100% | 100% |
### System Resources
- **Memory Usage**: ~131MB per service (optimized)
- **Container Startup**: 20-30% faster with optimization
- **Connection Efficiency**: 40-60% improvement with pooling
- **Network Isolation**: Custom bridge network (172.20.0.0/16)
## 🏗️ System Architecture
### Production Infrastructure Stack
#### Container Orchestration
- **Docker Compose**: Multi-service deployment with health checks
- **Resource Limits**: CPU and memory constraints for production stability
- **Health Monitoring**: Automated health checks with retry mechanisms
- **Network Isolation**: Secure service communication
#### Message Bus (RabbitMQ)
- **High Availability**: Production-grade message broker
- **Connection Pooling**: Efficient connection reuse
- **Performance Tuning**: Memory watermarks and optimization
- **Authentication**: Secure credential management
#### HTTP Infrastructure
- **FastAPI Servers**: Async HTTP servers with streaming SSE
- **Connection Pooling**: 50 total connections, 10 per host
- **Error Handling**: Comprehensive error responses
- **Health Endpoints**: Service status monitoring
### Security Architecture
#### MCP Roots Implementation
- **Filesystem Boundaries**: Restricted access to approved directories
- **Path Validation**: Comprehensive security checks
- **Access Logging**: Security event monitoring
- **Error Handling**: Secure error responses
## � Configuration Management
### Docker Compose Production Configuration
```yaml
version: '3.8'
services:
rabbitmq:
image: rabbitmq:3.13.7-management
environment:
RABBITMQ_VM_MEMORY_HIGH_WATERMARK: 0.8
deploy:
resources:
limits: { memory: 512M, cpus: "0.5" }
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "ping"]
interval: 30s
timeout: 10s
retries: 3
```
### Performance Optimization
- **Connection Pooling**: 30-second keep-alive with automatic cleanup
- **Resource Limits**: Memory and CPU constraints
- **Health Checks**: Automated service monitoring
- **Network Optimization**: Custom bridge network for isolation
## 📁 Project Structure
```text
project-synapse/
├── src/
│ ├── agents/ # 7 specialized async agents
│ │ ├── async_orchestrator.py # Central coordinator
│ │ ├── async_search_agent.py # Web search capabilities
│ │ ├── async_extraction_agent.py # Content extraction
│ │ ├── async_fact_checker_agent.py # Validation services
│ │ ├── async_synthesis_agent.py # Report generation
│ │ ├── async_file_save_agent.py # Secure file operations
│ │ ├── async_logger_agent.py # System monitoring
│ │ └── async_base_agent.py # Common agent functionality
│ ├── mcp_servers/ # Production MCP servers
│ │ ├── fastapi_primary_server.py # Web tools with progress
│ │ └── fastapi_filesystem_server.py # Secure file operations
│ ├── message_bus/ # RabbitMQ message bus implementation
│ │ └── rabbitmq_bus.py # Async message routing
│ └── protocols/ # Communication schemas
│ ├── acp_schema.py # Agent Communication Protocol
│ └── mcp_schemas.py # Model Context Protocol
├── k8s/ # Kubernetes deployment manifests
│ ├── configmap.yaml # Environment configuration
│ ├── rabbitmq-deployment.yaml # RabbitMQ message broker
│ ├── primary-server-deployment.yaml # Primary MCP server
│ ├── filesystem-server-deployment.yaml # Filesystem MCP server
│ ├── agents-deployment.yaml # Agent application
│ └── deploy.sh # Deployment automation script
├── monitoring/ # Observability and monitoring
│ ├── prometheus.yml # Prometheus configuration
│ ├── grafana_dashboard.json # Pre-built Grafana dashboard
│ ├── grafana-datasources.yml # Grafana data source config
│ └── grafana-dashboards.yml # Dashboard provisioning config
├── scripts/ # Performance and monitoring tools
│ ├── performance_test.py # Load testing framework
│ ├── optimize_performance.py # Performance optimization
│ ├── monitor_system.py # Real-time monitoring
│ ├── health_check.py # System health validation
│ └── integration_test.py # End-to-end testing
├── docs/ # Comprehensive documentation
│ ├── ARCHITECTURE.md # System architecture guide
│ ├── ACP_SPEC.md # ACP protocol specification
│ └── MCP_IN_DEPTH.md # MCP implementation guide
├── docker-compose.optimized.yml # Production deployment
├── async_main.py # System entry point
└── requirements.txt # Python dependencies
```
## 🔬 Technical Documentation
### Protocol Specifications
- **[Agent Communication Protocol (ACP)](docs/ACP_SPEC.md)**: Custom messaging protocol with type safety
- **[Model Context Protocol (MCP)](docs/MCP_IN_DEPTH.md)**: Tool integration with progress notifications
- **[System Architecture](docs/ARCHITECTURE.md)**: Complete architectural documentation
### Production Considerations
- **Performance Testing**: Load testing with baseline metrics
- **Resource Management**: Memory and CPU optimization
- **Health Monitoring**: Automated service health checks
- **Security**: MCP Roots and access control implementation
## �️ Development Workflow
### Performance Testing
```bash
# Run performance baseline tests
python scripts/performance_test.py
# Monitor system resources
python scripts/monitor_system.py
# Optimize performance settings
python scripts/optimize_performance.py
```
### Health Monitoring
The system includes comprehensive health monitoring:
- **Service Health**: HTTP health endpoints for all services
- **Message Bus Status**: RabbitMQ connection and queue monitoring
- **Agent Status**: Real-time agent activity tracking
- **Resource Usage**: Memory and CPU utilization monitoring
## 🎯 Use Cases
### Research Workflow Automation
The system demonstrates automated research workflows:
1. **Query Processing**: Complex research question analysis
2. **Web Search**: Distributed search across multiple sources
3. **Content Extraction**: Intelligent content processing
4. **Fact Checking**: Automated claim validation
5. **Report Synthesis**: Comprehensive report generation
6. **Secure Storage**: MCP Roots-protected file operations
### Agent Coordination Patterns
- **Command & Control**: Orchestrator coordinating specialized agents
- **Peer Review**: Fact-checking and validation workflows
- **Publish-Subscribe**: System-wide event monitoring
- **Request-Response**: Sophisticated inter-agent communication
## 🔮 Production Deployment
### Scalability Features
- **Horizontal Scaling**: Multiple worker processes per service
- **Resource Isolation**: Container-based deployment
- **Health Monitoring**: Automated failure detection
- **Connection Pooling**: Efficient resource utilization
### Monitoring and Observability
- **Real-time Metrics**: Performance monitoring with baseline comparison
- **Health Dashboards**: Service status and resource utilization
- **Error Tracking**: Comprehensive error logging and reporting
- **Performance Analytics**: Response time and throughput analysis
## 📜 License
MIT License - see [LICENSE](LICENSE) for details.
## 🎯 Key Features
✅ **Production-Ready**: Docker containerization with Kubernetes deployment
✅ **High Performance**: Measured metrics with optimization framework
✅ **Secure Architecture**: MCP Roots implementation with access controls
✅ **Observable Systems**: Prometheus metrics and Grafana dashboards
✅ **Scalable Design**: Async architecture with connection pooling
✅ **Complete Documentation**: Architecture guides and implementation details
✅ **Production Monitoring**: Real-time metrics and alerting capabilities
---
**Deploy a production-grade multi-agent system:**
```bash
# Docker Compose (recommended for development/testing)
docker-compose -f docker-compose.optimized.yml up --build
# Kubernetes (recommended for production)
./k8s/deploy.sh
```
*Experience the power of async agent coordination with comprehensive monitoring and production-grade deployment.* 🧠✨
## 🎉 Project Completion Status
**🚀 PRODUCTION READY - FULLY VALIDATED** ✅
Project Synapse has successfully completed comprehensive development and QA validation:
### ✅ Development Milestones Achieved
- **Multi-Agent Architecture**: 7 specialized async agents with orchestration
- **Production Infrastructure**: Docker containerization with Kubernetes support
- **Message Bus Integration**: High-performance RabbitMQ async communication
- **MCP Protocol Implementation**: Secure tool integration with progress tracking
- **MLOps Monitoring Stack**: Prometheus + Grafana observability
- **Performance Optimization**: Resource limits and connection pooling
- **Security Implementation**: MCP Roots filesystem access controls
- **Comprehensive Documentation**: Complete technical specifications
### 🔬 Quality Assurance Validation
- **Repository Cleanup**: ✅ No legacy files or dead code
- **System Integration**: ✅ All 6 containers healthy and operational
- **End-to-End Workflows**: ✅ Live research workflow completed successfully
- **Monitoring Stack**: ✅ Prometheus metrics and Grafana dashboards active
- **Performance Metrics**: ✅ Production-grade response times verified
### 📊 Final System Metrics
- **Workflow Execution**: 4-second end-to-end research processing
- **Container Performance**: <131MB memory per optimized service
- **API Response Times**: 0.8-0.9ms average latency
- **Message Throughput**: 557+ RPS through RabbitMQ
- **Success Rate**: 100% operational reliability
**View the complete validation results in our [System Validation Report](SYSTEM_VALIDATION_REPORT.md)**
---
## 📚 Documentation Links
### Core Documentation
- **[🏛️ System Architecture](docs/ARCHITECTURE.md)**: Complete architectural overview with production infrastructure details
- **[📡 ACP Protocol Specification](docs/ACP_SPEC.md)**: Agent Communication Protocol technical specification
- **[🔧 MCP Implementation Guide](docs/MCP_IN_DEPTH.md)**: Model Context Protocol detailed implementation guide
- **[📋 Implementation Status](docs/IMPLEMENTATION_COMPLETE.md)**: Comprehensive development completion report
- **[✅ System Validation Report](SYSTEM_VALIDATION_REPORT.md)**: Final QA validation and production readiness certification
### Branch Information
- **[🎓 Educational Simulation Branch](https://github.com/YanCotta/project-synapse/tree/educational-simulation)**: Step-by-step learning implementation with detailed explanations
- **[🚀 Production Implementation (Current)](https://github.com/YanCotta/project-synapse/tree/working-app)**: Fully functional, production-ready system with monitoring
