SanctumOS

The Modular, Self-Hosted Agentic Operating System

Troubleshooting Guide

This guide helps you diagnose and resolve common issues with SanctumOS installations and configurations.

🚨 Common Issues

Installation Issues

Port Conflicts

Problem: Services fail to start due to port conflicts.

Symptoms:

  • Error messages about ports being in use
  • Services fail to bind to specified ports

Solutions:

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Check what's using the ports

lsof -i :8000 # Sanctum Letta MCP lsof -i :8080 # Sanctum Web Chat

Kill conflicting processes

kill -9

Use different ports

python smcp/mcp_server.py --port 9000 php -S localhost:8081


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Permission Issues

Problem: File permission errors during installation.

Symptoms:

  • Permission denied errors
  • Cannot create directories or files

Solutions:

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Fix file permissions

chmod -R 755 ~/sanctum chown -R $USER:$USER ~/sanctum

Check directory permissions

ls -la ~/sanctum


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Dependency Issues

Problem: Python package installation failures.

Symptoms:

  • pip install failures
  • Import errors
  • Missing modules

Solutions:

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Update pip

pip install --upgrade pip

Reinstall dependencies

pip install -r requirements.txt --force-reinstall

Use virtual environment

python -m venv venv source venv/bin/activate pip install -r requirements.txt


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Configuration Issues

Agent Connection Problems

Problem: Broca-2 cannot connect to AI agents.

Symptoms:

  • Connection timeout errors
  • Authentication failures
  • Agent not responding

Solutions:

  1. Check agent endpoint:
   # Verify agent is running
   curl -f https://your-agent-api.com/health

Verify agent is running

curl -f https://your-agent-api.com/health


2. Verify API key:
   bash
   # Check .env file
   cat .env | grep AGENT_API_KEY
  1. Verify API key:
   # Verify agent is running
   curl -f https://your-agent-api.com/health

Check .env file

cat .env | grep AGENT_API_KEY


2. Verify API key:
   bash
   # Check .env file
   cat .env | grep AGENT_API_KEY
  1. Test with echo mode:
   # Verify agent is running
   curl -f https://your-agent-api.com/health

Use echo mode for testing

echo 'MESSAGE_MODE=echo' >> .env


2. Verify API key:
   bash
   # Check .env file
   cat .env | grep AGENT_API_KEY

Database Issues

Problem: Database connection or corruption issues.

Symptoms:

  • Database locked errors
  • Corrupted database files
  • Migration failures

Solutions:

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Check database file permissions

ls -la sanctum.db

Backup and recreate database

cp sanctum.db sanctum.db.backup rm sanctum.db python -c "from database.operations.shared import initialize_database; initialize_database()"

Check database integrity

sqlite3 sanctum.db "PRAGMA integrity_check;"


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Runtime Issues

Message Queue Problems

Problem: Messages not being processed or stuck in queue.

Symptoms:

  • Messages accumulating in queue
  • No responses from agents
  • Queue processing errors

Solutions:

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Check queue status

python -m cli.btool queue list

Flush stuck messages

python -m cli.btool queue flush

Check queue processing logs

tail -f logs/broca.log


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Plugin Loading Issues

Problem: Plugins not loading or functioning correctly.

Symptoms:

  • Plugin discovery failures
  • Tool execution errors
  • Plugin crashes

Solutions:

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Check plugin directory

ls -la smcp/plugins/

Test plugin manually

python smcp/plugins/your_plugin/cli.py --help

Check plugin permissions

chmod +x smcp/plugins/your_plugin/cli.py


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Web Chat Issues

Problem: Web chat interface not working or API errors.

Symptoms:

  • Web interface not loading
  • API authentication failures
  • Rate limiting errors

Solutions:

  1. Check API keys:
   # Verify agent is running
   curl -f https://your-agent-api.com/health

Verify API key configuration

echo $WEB_CHAT_API_KEY


2. Verify API key:
   bash
   # Check .env file
   cat .env | grep AGENT_API_KEY
  1. Test API endpoints:
   # Verify agent is running
   curl -f https://your-agent-api.com/health

Test health endpoint

curl http://localhost:8080/api/v1/?action=health


2. Verify API key:
   bash
   # Check .env file
   cat .env | grep AGENT_API_KEY
  1. Check rate limits:
   # Verify agent is running
   curl -f https://your-agent-api.com/health

Check rate limit logs

tail -f logs/api.log


2. Verify API key:
   bash
   # Check .env file
   cat .env | grep AGENT_API_KEY

🔍 Diagnostic Tools

Health Checks

System Health

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Check Sanctum Letta MCP

curl -X POST http://localhost:8000/messages/ \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"health","arguments":{}}}'

Check Broca-2

python -m cli.btool health check

Check Sanctum Web Chat

curl http://localhost:8080/api/v1/?action=health


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Service Status

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Check running processes

ps aux | grep python ps aux | grep php

Check port usage

netstat -tlnp | grep :8000 netstat -tlnp | grep :8080

Check system resources

top df -h free -h


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Log Analysis

Log Locations

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Sanctum Letta MCP logs

tail -f mcp.log

Broca-2 logs

tail -f logs/broca.log

Sanctum Web Chat logs

tail -f logs/api.log


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Log Analysis

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Search for errors

grep -i error logs/.log

Search for specific issues

grep -i "connection" logs/.log grep -i "timeout" logs/.log grep -i "permission" logs/.log


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

🛠ī¸ Debugging

Enable Debug Mode

Sanctum Letta MCP

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Enable debug logging

export MCP_DEBUG=true python smcp/mcp_server.py


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Broca-2

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Enable debug mode

echo 'DEBUG_MODE=true' >> .env python main.py


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Sanctum Web Chat

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Enable debug mode

export WEB_CHAT_DEBUG=true php -S localhost:8080


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Verbose Output

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Run with verbose output

python smcp/mcp_server.py --verbose python main.py --debug php -S localhost:8080 -t public


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Network Debugging

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Test network connectivity

ping your-agent-api.com telnet your-agent-api.com 443

Check DNS resolution

nslookup your-agent-api.com dig your-agent-api.com


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

🔧 Configuration Validation

Validate Configuration Files

Environment Variables

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Check required environment variables

echo "AGENT_ENDPOINT: $AGENT_ENDPOINT" echo "AGENT_API_KEY: $AGENT_API_KEY" echo "WEB_CHAT_API_KEY: $WEB_CHAT_API_KEY"


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Configuration Files

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Validate JSON configuration

python -m json.tool settings.json

Check .env file format

cat .env | grep -v "^#" | grep -v "^$"


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Database Schema

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Check database schema

sqlite3 sanctum.db ".schema"

Verify table structure

sqlite3 sanctum.db "SELECT name FROM sqlite_master WHERE type='table';"


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

🚨 Emergency Recovery

Service Recovery

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Restart all services

pkill -f "python.mcp_server.py" pkill -f "python.main.py" pkill -f "php.-S"

Restart services

python smcp/mcp_server.py & python main.py & php -S localhost:8080 &


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Data Recovery

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Backup current state

tar -czf sanctum-backup-$(date +%Y%m%d).tar.gz ~/sanctum

Restore from backup

tar -xzf sanctum-backup-YYYYMMDD.tar.gz


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

Configuration Reset

# Check what's using the ports
lsof -i :8000  # Sanctum Letta MCP
lsof -i :8080  # Sanctum Web Chat

# Kill conflicting processes
kill -9 <PID>

# Use different ports
python smcp/mcp_server.py --port 9000
php -S localhost:8081

Reset to default configuration

cp .env.example .env cp settings.json.example settings.json

Reinitialize database

rm sanctum.db python -c "from database.operations.shared import initialize_database; initialize_database()"


#### Permission Issues
Problem: File permission errors during installation.

Symptoms:
- Permission denied errors
- Cannot create directories or files

Solutions:
`bash
# Fix file permissions
chmod -R 755 ~/sanctum
chown -R $USER:$USER ~/sanctum

# Check directory permissions
ls -la ~/sanctum

📞 Getting Help

Self-Help Resources

  1. Check Documentation: Review module-specific documentation
  2. Search Issues: Check GitHub issues for known problems
  3. Community Forums: Join community discussions
  4. Log Analysis: Analyze logs for error patterns

Reporting Issues

When reporting issues, include:

  1. System Information:
  2. Operating system and version
  3. Python version
  4. SanctumOS version
  1. Error Details:
  2. Complete error messages
  3. Relevant log entries
  4. Steps to reproduce
  1. Configuration:
  2. Relevant configuration files (sanitized)
  3. Environment variables (sanitized)
  4. Network configuration

Support Channels

  • GitHub Issues: https://github.com/sanctumos/sanctumos/issues
  • Documentation: https://docs.sanctumos.org
  • Community: https://community.sanctumos.org

🔄 Maintenance

Regular Maintenance

Daily

  • Check service health
  • Monitor log files
  • Verify agent connectivity

Weekly

  • Review error logs
  • Check disk space
  • Update dependencies

Monthly

  • Backup configurations
  • Review security settings
  • Update documentation

Preventive Measures

  1. Regular Backups: Automated backup of configurations and data
  2. Monitoring: Set up monitoring and alerting
  3. Updates: Keep dependencies and system updated
  4. Security: Regular security audits and updates

SanctumOS Troubleshooting Guide - Resolve issues and keep your AI agent communication platform running smoothly.*