There's a lot here...

So, let's get into it.
First, we are straying from the in-memory ideation of storing all our pcap chunks in the memory
I fear that we might bottle up the ram if we depend solely on it.
So, i've made things default to sqlite flatfile, with the option for both remote and in memory
This way, dependant on the users needs, they can use a different implementation.

That's why the creation of a great deal of files and an entire new directory space
I've almost finished implementing the database system into itself, but I want the DB system
to be largely independent of the rest of the system. Because, you know me,
I am a fan of encapsulation. and I like making software coded to be independent.

Yes, if the rest of the system isn't working, the database has nothing to docker
but at least it makes the system more robust, if even a major system can collapse,
and the system as a whole remains functional.

Philisophically, I find this is largely lacking in modern software, and I believe
that these practices will see the protohandler performing much better than we would expect
Since this system as a whole will be independent of the main NETRAVE system,
it opens the door for further expanding, or perhaps even completely repurposing
this protocol going forward.

docker/netrave-protohandler/config_manager.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+require 'json'
+require 'singleton'
+
+module ProtoConfig
+  class ConfigManager
+    include Singleton
+
+    attr_reader :recently_connected_time, :listen_ip, :listen_port, :db_type, :db_params, :db_username, :db_password
+
+    def initialize
+      # Initialize with default settings
+      @recently_connected_time = 120 # default value
+      @listen_ip = '0.0.0.0' # default value
+      @listen_port = 3080 # default value
+      @db_type = 'sqlite'
+      @db_params = { username: nil, password: nil } # default value
+
+      # Check if the config file exists in the current working directory
+      config_path = 'config.json'
+      create_default_config(config_path) unless File.exist?(config_path)
+
+      load_config(config_path)
+    end
+
+    def create_default_config(config_path)
+      default_config = {
+        'recently_connected_time' => @recently_connected_time,
+        'listen_ip' => @listen_ip,
+        'listen_port' => @listen_port,
+        'db_type' => @db_type,
+        'db_params' => @db_params,
+        'db_username' => @db_username,
+        'db_password' => @db_password
+      }
+      File.write(config_path, JSON.pretty_generate(default_config))
+    end
+
+    def load_config(config_path)
+      # Load configuration from a JSON file
+      config_data = JSON.parse(File.read(config_path))
+
+      @recently_connected_time = config_data['recently_connected_time'] if config_data['recently_connected_time']
+      @listen_ip = config_data['listen_ip'] if config_data['listen_ip']
+      @listen_port = config_data['listen_port'] if config_data['listen_port']
+      @db_type = config_data['db_type'] if config_data['db_type']
+      @db_params = config_data['db_params'] if config_data['db_params']
+    rescue StandardError => e
+      puts "Failed to load configuration: #{e.message}"
+    end
+  end
+end

docker/netrave-protohandler/db/database_manager_factory.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+# The 'factory' that constructs the constructors for our various database managers
+class DatabaseManagerFactory
+  def self.create(db_type, username: nil, password: nil)
+    db_manager = Object.new
+    case db_type
+    when 'sqlite_memory'
+      db_manager.extend(SQLiteMemoryManager)
+    when 'sqlite_flatfile'
+      db_manager.extend(SQLiteFlatFileManager)
+    when 'mysql'
+      db_manager.extend(MySQLManager)
+    when 'postgres'
+      db_manager.extend(PostgresManager)
+    when 'mongodb'
+      db_manager.extend(MongoDBManager)
+    else
+      raise "Unknown database type: #{db_type}"
+    end
+
+    # Initialize the database manager with username and password if they exist
+    db_manager.initialize(username, password) if db_manager.respond_to?(:initialize)
+
+    db_manager
+  end
+end

docker/netrave-protohandler/db/dbmanager_module.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+# CRUD operations for Database modularity
+module DatabaseManager
+  def initialize_db(username = nil, password = nil); end
+
+  def insert(table, data)
+    raise NotImplementedError, 'This method is defined in a mixin and must be overridden'
+  end
+
+  def query(table, condition)
+    raise NotImplementedError, 'This method is defined in a mixin and must be overridden'
+  end
+
+  def update(table, query, update)
+    raise NotImplementedError, 'This method is defined in a mixin and must be overridden'
+  end
+
+  def delete(table, query)
+    raise NotImplementedError, 'This method is defined in a mixin and must be overridden'
+  end
+end

docker/netrave-protohandler/db/mongodb_module.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require 'mongo'
+require_relative 'dbmanager_module'
+
+module MongoDBManager
+  include DatabaseManager
+
+  def initialize_db(username = nil, password = nil)
+    credentials = username && password ? { user: username, password: password } : {}
+    @client = Mongo::Client.new(['127.0.0.1:27017'], { database: 'my_db' }.merge(credentials))
+  end
+
+  def insert(table, data)
+    @client[table].insert_one(data)
+  end
+
+  def query(table, condition)
+    @client[table].find(condition).to_a
+  end
+
+  def update(table, query, update)
+    @client[table].find_one_and_update(query, { '$set' => update })
+  end
+
+  def delete(table, query)
+    @client[table].find_one_and_delete(query)
+  end
+end

docker/netrave-protohandler/db/mysqldb_module.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require 'sequel'
+
+module MySQLManager
+  include DatabaseManager
+
+  def initialize_db
+    @db = Sequel.connect(adapter: 'mysql2', host: 'localhost', database: 'my_db', user: 'user', password: 'password')
+    # ... setup tables
+  end
+
+  # ... implement CRUD operations
+end

docker/netrave-protohandler/db/postgre_module.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require 'sequel'
+
+module PostgresManager
+  include DatabaseManager
+
+  def initialize_db
+    @db = Sequel.connect(adapter: 'postgres', host: 'localhost', database: 'my_db', user: 'user', password: 'password')
+    # ... setup tables
+  end
+
+  # ... implement CRUD operations
+end

docker/netrave-protohandler/db/protohandler_dbmanager.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+require 'sequel'
+
+class ProtohandlerDBManager
+  attr_reader :db
+
+  def initialize
+    setup_database
+  end
+
+  def setup_database
+    @db = Sequel.sqlite # In-memory database
+
+    create_processors_table unless table_exists?(:processors)
+    create_orchestrators_table unless table_exists?(:orchestrators)
+    create_blacklist_table unless table_exists?(:blacklist)
+  end
+
+  def table_exists?(table_name)
+    @db.table_exists?(table_name)
+  end
+
+  def create_processors_table
+    @db.create_table :processors do
+      primary_key :proto_handler_id
+      String :uuid
+      String :domain
+      Integer :port
+    end
+  end
+
+  def create_orchestrators_table
+    @db.create_table :orchestrators do
+      primary_key :id
+      String :domain
+      Integer :port
+    end
+  end
+
+  def create_blacklist_table
+    @db.create_table :blacklist do
+      primary_key :id
+      String :uuid
+    end
+  end
+end

docker/netrave-protohandler/db/sqliteflatfile_module.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require 'sequel'
+
+module SQLiteFlatFileManager
+  include DatabaseManager
+
+  def initialize_db
+    @db = Sequel.sqlite('my_database.db')
+    # ... setup tables
+  end
+
+  # ... implement CRUD operations
+end

docker/netrave-protohandler/db/sqlitememory_module.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require 'sequel'
+
+module SQLiteMemoryManager
+  include DatabaseManager
+
+  def initialize_db
+    @db = Sequel.sqlite
+    # ... setup tables
+  end
+
+  # ... implement CRUD operations
+end

docker/netrave-protohandler/netrave_protohandler.rb

@@ -2,42 +2,30 @@
 
 require 'socket'
 require 'async'
-require 'sequel'
 require 'openssl'
 require 'securerandom'
+require_relative 'db/protohandler_dbmanager'
 
-# Set up the database
+# main class for the protocol handler server
-DB = Sequel.sqlite # In-memory database
+class ProtoServer
+  attr_reader :listen_ip, :listen_port, :db_manager
 
-# Create processors table
+  def initialize(listen_ip, listen_port)
-DB.create_table :processors do
+    @listen_ip = listen_ip
-  primary_key :proto_handler_id
+    @listen_port = listen_port
-  String :uuid
+    @db_manager = ProtohandlerDBManager.new
-  String :domain
+  end
-  Integer :port
 end
 
-# Create orchestrators table
+# Initialize server with config
-DB.create_table :orchestrators do
+config = ServerConfig.new('0.0.0.0', 3080)
-  primary_key :id
+server = TCPServer.new(config.listen_ip, config.listen_port)
-  String :domain
-  Integer :port
-end
 
-# Create blacklist table
+# Initialize database tables
-DB.create_table :blacklist do
+db_manager = config.db_manager
-  primary_key :id
+processors = db_manager.db[:processors]
-  String :uuid
+orchestrators = db_manager.db[:orchestrators]
-end
+blacklist = db_manager.db[:blacklist]
-
-processors = DB[:processors]
-orchestrators = DB[:orchestrators]
-blacklist = DB[:blacklist]
-
-listen_ip = ENV['LISTEN_IP'] || '0.0.0.0'
-listen_port = ENV['LISTEN_PORT'] || 3080
-
-server = TCPServer.new(listen_ip, listen_port)
 
 # This hash will store the recently connected UUIDs with their last validation timestamp
 recently_connected = {}
@@ -58,21 +46,37 @@ def create_socket(ip, port) # rubocop:disable Metrics/MethodLength
   end
 end
 
-def handle_input(line, processors, orchestrators, blacklist, recently_connected) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
+# Check if the UUID is blacklisted
+def blacklisted?(uuid, blacklist)
+  blacklist.where(uuid:).count.positive?
+end
+
+# Validate if the UUID has recently connected
+def recently_connected?(uuid, recently_connected)
+  recently_connected[uuid] && Time.now - recently_connected[uuid] < 120
+end
+
+# Handle unregistered UUIDs
+def handle_unregistered_uuid(uuid, processors, blacklist)
+  if uuid.nil? || processors.where(uuid:).count.zero?
+    blacklist.insert(uuid:)
+    return 'ERROR Unrecognized UUID'
+  end
+  nil
+end
+
+def handle_input(line, processors, orchestrators, blacklist, recently_connected)
   command, *args = line.split
   uuid = args.shift if command != 'REGISTER'
 
-  # Check if the UUID is blacklisted
+  return 'TERMINATE' if blacklisted?(uuid, blacklist)
-  return 'TERMINATE' if blacklist.where(uuid:).count.positive?
 
-  # Check if the UUID is in the recently connected cache
+  if recently_connected?(uuid, recently_connected)
-  if recently_connected[uuid] && Time.now - recently_connected[uuid] < 120
     # UUID is recently connected and within 2 minutes, no need to re-validate
-  elsif command != 'REGISTER' && (uuid.nil? || processors.where(uuid:).count.zero?)
-    # UUID is not recently connected or is invalid, add to blacklist
-    blacklist.insert(uuid:)
-    return 'ERROR Unrecognized UUID'
   else
+    unregistered_response = handle_unregistered_uuid(uuid, processors, blacklist)
+    return unregistered_response if unregistered_response
+
     # UUID is valid, update the recently connected cache
     recently_connected[uuid] = Time.now
   end
@@ -108,6 +112,7 @@ def register_orchestrator(line, orchestrators)
   puts "Orchestrator registered with domain: #{domain}, port: #{port}"
 end
 
+# Main Async loop
 Async do
   loop do
     Async::Task.new do
@@ -120,7 +125,7 @@ Async do
           register_orchestrator(line, orchestrators)
         else
           # Here we handle each line of input from the client
-          handle_input(line, processors, blacklist, recently_connected)
+          handle_input(line, processors, orchestrators, blacklist, recently_connected)
         end
       ensure
         # This code will be executed when the fiber is finished, regardless of whether an exception was raised

docker/netrave-protohandler/npeph

@@ -0,0 +1,18 @@
+#!/usr/bin/env ruby
+
+# frozen_string_literal: true
+
+require_relative 'config_manager'
+
+# Initialize the ConfigManager
+config = ProtoConfig::ConfigManager.instance
+
+# Your logic to start the Protocol Handler goes here
+# For example, you could initialize your server with the IP and port from the config
+# Or pass the recently_connected_time to your connection handling logic
+
+puts "NETRAVE Packet Exchange Protocol Handler (NPEPH) is starting..."
+puts "Listening on #{config.listen_ip}:#{config.listen_port}"
+puts "Recently connected time threshold: #{config.recently_connected_time} seconds"
+
+# Your server initialization and start logic here

lib/pkg/sumdb/sum.golang.org/latest

@@ -1,5 +1,5 @@
 go.sum database tree
-18982882
+19688221
-5hq5RwlCLIiagi6hZScdHlumu5XIpITikSz2+tAr9LA=
+k58/ugqvqq6DqYjvo1m1COmAGb067rn6gQNOPDdtELg=
 
-— sum.golang.org Az3grhw2ZJlhe/saIZhYrbHraYSRxzcFocBkNuDAACA6cT3QTOA1P1fa+rhYfXNq2KVoROv5wGw5Y/2naw4QGoEoYAE=
+— sum.golang.org Az3grhYRJ7p1GIdTipHl5Sqe55vwSI8JugOEHx10umcUxmC5CFJxCpjrU3SFPAnm1YuxAZ3TqifbpfWXZncfjYtidgY=