There's a lot here...
So, let's get into it. First, we are straying from the in-memory ideation of storing all our pcap chunks in the memory I fear that we might bottle up the ram if we depend solely on it. So, i've made things default to sqlite flatfile, with the option for both remote and in memory This way, dependant on the users needs, they can use a different implementation. That's why the creation of a great deal of files and an entire new directory space I've almost finished implementing the database system into itself, but I want the DB system to be largely independent of the rest of the system. Because, you know me, I am a fan of encapsulation. and I like making software coded to be independent. Yes, if the rest of the system isn't working, the database has nothing to docker but at least it makes the system more robust, if even a major system can collapse, and the system as a whole remains functional. Philisophically, I find this is largely lacking in modern software, and I believe that these practices will see the protohandler performing much better than we would expect Since this system as a whole will be independent of the main NETRAVE system, it opens the door for further expanding, or perhaps even completely repurposing this protocol going forward.
This commit is contained in:
parent
5dde3f6ca4
commit
5df588674e
53
docker/netrave-protohandler/config_manager.rb
Normal file
53
docker/netrave-protohandler/config_manager.rb
Normal file
|
@ -0,0 +1,53 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require 'json'
|
||||||
|
require 'singleton'
|
||||||
|
|
||||||
|
module ProtoConfig
|
||||||
|
class ConfigManager
|
||||||
|
include Singleton
|
||||||
|
|
||||||
|
attr_reader :recently_connected_time, :listen_ip, :listen_port, :db_type, :db_params, :db_username, :db_password
|
||||||
|
|
||||||
|
def initialize
|
||||||
|
# Initialize with default settings
|
||||||
|
@recently_connected_time = 120 # default value
|
||||||
|
@listen_ip = '0.0.0.0' # default value
|
||||||
|
@listen_port = 3080 # default value
|
||||||
|
@db_type = 'sqlite'
|
||||||
|
@db_params = { username: nil, password: nil } # default value
|
||||||
|
|
||||||
|
# Check if the config file exists in the current working directory
|
||||||
|
config_path = 'config.json'
|
||||||
|
create_default_config(config_path) unless File.exist?(config_path)
|
||||||
|
|
||||||
|
load_config(config_path)
|
||||||
|
end
|
||||||
|
|
||||||
|
def create_default_config(config_path)
|
||||||
|
default_config = {
|
||||||
|
'recently_connected_time' => @recently_connected_time,
|
||||||
|
'listen_ip' => @listen_ip,
|
||||||
|
'listen_port' => @listen_port,
|
||||||
|
'db_type' => @db_type,
|
||||||
|
'db_params' => @db_params,
|
||||||
|
'db_username' => @db_username,
|
||||||
|
'db_password' => @db_password
|
||||||
|
}
|
||||||
|
File.write(config_path, JSON.pretty_generate(default_config))
|
||||||
|
end
|
||||||
|
|
||||||
|
def load_config(config_path)
|
||||||
|
# Load configuration from a JSON file
|
||||||
|
config_data = JSON.parse(File.read(config_path))
|
||||||
|
|
||||||
|
@recently_connected_time = config_data['recently_connected_time'] if config_data['recently_connected_time']
|
||||||
|
@listen_ip = config_data['listen_ip'] if config_data['listen_ip']
|
||||||
|
@listen_port = config_data['listen_port'] if config_data['listen_port']
|
||||||
|
@db_type = config_data['db_type'] if config_data['db_type']
|
||||||
|
@db_params = config_data['db_params'] if config_data['db_params']
|
||||||
|
rescue StandardError => e
|
||||||
|
puts "Failed to load configuration: #{e.message}"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
27
docker/netrave-protohandler/db/database_manager_factory.rb
Normal file
27
docker/netrave-protohandler/db/database_manager_factory.rb
Normal file
|
@ -0,0 +1,27 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
# The 'factory' that constructs the constructors for our various database managers
|
||||||
|
class DatabaseManagerFactory
|
||||||
|
def self.create(db_type, username: nil, password: nil)
|
||||||
|
db_manager = Object.new
|
||||||
|
case db_type
|
||||||
|
when 'sqlite_memory'
|
||||||
|
db_manager.extend(SQLiteMemoryManager)
|
||||||
|
when 'sqlite_flatfile'
|
||||||
|
db_manager.extend(SQLiteFlatFileManager)
|
||||||
|
when 'mysql'
|
||||||
|
db_manager.extend(MySQLManager)
|
||||||
|
when 'postgres'
|
||||||
|
db_manager.extend(PostgresManager)
|
||||||
|
when 'mongodb'
|
||||||
|
db_manager.extend(MongoDBManager)
|
||||||
|
else
|
||||||
|
raise "Unknown database type: #{db_type}"
|
||||||
|
end
|
||||||
|
|
||||||
|
# Initialize the database manager with username and password if they exist
|
||||||
|
db_manager.initialize(username, password) if db_manager.respond_to?(:initialize)
|
||||||
|
|
||||||
|
db_manager
|
||||||
|
end
|
||||||
|
end
|
22
docker/netrave-protohandler/db/dbmanager_module.rb
Normal file
22
docker/netrave-protohandler/db/dbmanager_module.rb
Normal file
|
@ -0,0 +1,22 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
# CRUD operations for Database modularity
|
||||||
|
module DatabaseManager
|
||||||
|
def initialize_db(username = nil, password = nil); end
|
||||||
|
|
||||||
|
def insert(table, data)
|
||||||
|
raise NotImplementedError, 'This method is defined in a mixin and must be overridden'
|
||||||
|
end
|
||||||
|
|
||||||
|
def query(table, condition)
|
||||||
|
raise NotImplementedError, 'This method is defined in a mixin and must be overridden'
|
||||||
|
end
|
||||||
|
|
||||||
|
def update(table, query, update)
|
||||||
|
raise NotImplementedError, 'This method is defined in a mixin and must be overridden'
|
||||||
|
end
|
||||||
|
|
||||||
|
def delete(table, query)
|
||||||
|
raise NotImplementedError, 'This method is defined in a mixin and must be overridden'
|
||||||
|
end
|
||||||
|
end
|
29
docker/netrave-protohandler/db/mongodb_module.rb
Normal file
29
docker/netrave-protohandler/db/mongodb_module.rb
Normal file
|
@ -0,0 +1,29 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require 'mongo'
|
||||||
|
require_relative 'dbmanager_module'
|
||||||
|
|
||||||
|
module MongoDBManager
|
||||||
|
include DatabaseManager
|
||||||
|
|
||||||
|
def initialize_db(username = nil, password = nil)
|
||||||
|
credentials = username && password ? { user: username, password: password } : {}
|
||||||
|
@client = Mongo::Client.new(['127.0.0.1:27017'], { database: 'my_db' }.merge(credentials))
|
||||||
|
end
|
||||||
|
|
||||||
|
def insert(table, data)
|
||||||
|
@client[table].insert_one(data)
|
||||||
|
end
|
||||||
|
|
||||||
|
def query(table, condition)
|
||||||
|
@client[table].find(condition).to_a
|
||||||
|
end
|
||||||
|
|
||||||
|
def update(table, query, update)
|
||||||
|
@client[table].find_one_and_update(query, { '$set' => update })
|
||||||
|
end
|
||||||
|
|
||||||
|
def delete(table, query)
|
||||||
|
@client[table].find_one_and_delete(query)
|
||||||
|
end
|
||||||
|
end
|
14
docker/netrave-protohandler/db/mysqldb_module.rb
Normal file
14
docker/netrave-protohandler/db/mysqldb_module.rb
Normal file
|
@ -0,0 +1,14 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require 'sequel'
|
||||||
|
|
||||||
|
module MySQLManager
|
||||||
|
include DatabaseManager
|
||||||
|
|
||||||
|
def initialize_db
|
||||||
|
@db = Sequel.connect(adapter: 'mysql2', host: 'localhost', database: 'my_db', user: 'user', password: 'password')
|
||||||
|
# ... setup tables
|
||||||
|
end
|
||||||
|
|
||||||
|
# ... implement CRUD operations
|
||||||
|
end
|
14
docker/netrave-protohandler/db/postgre_module.rb
Normal file
14
docker/netrave-protohandler/db/postgre_module.rb
Normal file
|
@ -0,0 +1,14 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require 'sequel'
|
||||||
|
|
||||||
|
module PostgresManager
|
||||||
|
include DatabaseManager
|
||||||
|
|
||||||
|
def initialize_db
|
||||||
|
@db = Sequel.connect(adapter: 'postgres', host: 'localhost', database: 'my_db', user: 'user', password: 'password')
|
||||||
|
# ... setup tables
|
||||||
|
end
|
||||||
|
|
||||||
|
# ... implement CRUD operations
|
||||||
|
end
|
47
docker/netrave-protohandler/db/protohandler_dbmanager.rb
Normal file
47
docker/netrave-protohandler/db/protohandler_dbmanager.rb
Normal file
|
@ -0,0 +1,47 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require 'sequel'
|
||||||
|
|
||||||
|
class ProtohandlerDBManager
|
||||||
|
attr_reader :db
|
||||||
|
|
||||||
|
def initialize
|
||||||
|
setup_database
|
||||||
|
end
|
||||||
|
|
||||||
|
def setup_database
|
||||||
|
@db = Sequel.sqlite # In-memory database
|
||||||
|
|
||||||
|
create_processors_table unless table_exists?(:processors)
|
||||||
|
create_orchestrators_table unless table_exists?(:orchestrators)
|
||||||
|
create_blacklist_table unless table_exists?(:blacklist)
|
||||||
|
end
|
||||||
|
|
||||||
|
def table_exists?(table_name)
|
||||||
|
@db.table_exists?(table_name)
|
||||||
|
end
|
||||||
|
|
||||||
|
def create_processors_table
|
||||||
|
@db.create_table :processors do
|
||||||
|
primary_key :proto_handler_id
|
||||||
|
String :uuid
|
||||||
|
String :domain
|
||||||
|
Integer :port
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def create_orchestrators_table
|
||||||
|
@db.create_table :orchestrators do
|
||||||
|
primary_key :id
|
||||||
|
String :domain
|
||||||
|
Integer :port
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def create_blacklist_table
|
||||||
|
@db.create_table :blacklist do
|
||||||
|
primary_key :id
|
||||||
|
String :uuid
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
14
docker/netrave-protohandler/db/sqliteflatfile_module.rb
Normal file
14
docker/netrave-protohandler/db/sqliteflatfile_module.rb
Normal file
|
@ -0,0 +1,14 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require 'sequel'
|
||||||
|
|
||||||
|
module SQLiteFlatFileManager
|
||||||
|
include DatabaseManager
|
||||||
|
|
||||||
|
def initialize_db
|
||||||
|
@db = Sequel.sqlite('my_database.db')
|
||||||
|
# ... setup tables
|
||||||
|
end
|
||||||
|
|
||||||
|
# ... implement CRUD operations
|
||||||
|
end
|
14
docker/netrave-protohandler/db/sqlitememory_module.rb
Normal file
14
docker/netrave-protohandler/db/sqlitememory_module.rb
Normal file
|
@ -0,0 +1,14 @@
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require 'sequel'
|
||||||
|
|
||||||
|
module SQLiteMemoryManager
|
||||||
|
include DatabaseManager
|
||||||
|
|
||||||
|
def initialize_db
|
||||||
|
@db = Sequel.sqlite
|
||||||
|
# ... setup tables
|
||||||
|
end
|
||||||
|
|
||||||
|
# ... implement CRUD operations
|
||||||
|
end
|
|
@ -2,42 +2,30 @@
|
||||||
|
|
||||||
require 'socket'
|
require 'socket'
|
||||||
require 'async'
|
require 'async'
|
||||||
require 'sequel'
|
|
||||||
require 'openssl'
|
require 'openssl'
|
||||||
require 'securerandom'
|
require 'securerandom'
|
||||||
|
require_relative 'db/protohandler_dbmanager'
|
||||||
|
|
||||||
# Set up the database
|
# main class for the protocol handler server
|
||||||
DB = Sequel.sqlite # In-memory database
|
class ProtoServer
|
||||||
|
attr_reader :listen_ip, :listen_port, :db_manager
|
||||||
|
|
||||||
# Create processors table
|
def initialize(listen_ip, listen_port)
|
||||||
DB.create_table :processors do
|
@listen_ip = listen_ip
|
||||||
primary_key :proto_handler_id
|
@listen_port = listen_port
|
||||||
String :uuid
|
@db_manager = ProtohandlerDBManager.new
|
||||||
String :domain
|
end
|
||||||
Integer :port
|
|
||||||
end
|
end
|
||||||
|
|
||||||
# Create orchestrators table
|
# Initialize server with config
|
||||||
DB.create_table :orchestrators do
|
config = ServerConfig.new('0.0.0.0', 3080)
|
||||||
primary_key :id
|
server = TCPServer.new(config.listen_ip, config.listen_port)
|
||||||
String :domain
|
|
||||||
Integer :port
|
|
||||||
end
|
|
||||||
|
|
||||||
# Create blacklist table
|
# Initialize database tables
|
||||||
DB.create_table :blacklist do
|
db_manager = config.db_manager
|
||||||
primary_key :id
|
processors = db_manager.db[:processors]
|
||||||
String :uuid
|
orchestrators = db_manager.db[:orchestrators]
|
||||||
end
|
blacklist = db_manager.db[:blacklist]
|
||||||
|
|
||||||
processors = DB[:processors]
|
|
||||||
orchestrators = DB[:orchestrators]
|
|
||||||
blacklist = DB[:blacklist]
|
|
||||||
|
|
||||||
listen_ip = ENV['LISTEN_IP'] || '0.0.0.0'
|
|
||||||
listen_port = ENV['LISTEN_PORT'] || 3080
|
|
||||||
|
|
||||||
server = TCPServer.new(listen_ip, listen_port)
|
|
||||||
|
|
||||||
# This hash will store the recently connected UUIDs with their last validation timestamp
|
# This hash will store the recently connected UUIDs with their last validation timestamp
|
||||||
recently_connected = {}
|
recently_connected = {}
|
||||||
|
@ -58,21 +46,37 @@ def create_socket(ip, port) # rubocop:disable Metrics/MethodLength
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
def handle_input(line, processors, orchestrators, blacklist, recently_connected) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
|
# Check if the UUID is blacklisted
|
||||||
|
def blacklisted?(uuid, blacklist)
|
||||||
|
blacklist.where(uuid:).count.positive?
|
||||||
|
end
|
||||||
|
|
||||||
|
# Validate if the UUID has recently connected
|
||||||
|
def recently_connected?(uuid, recently_connected)
|
||||||
|
recently_connected[uuid] && Time.now - recently_connected[uuid] < 120
|
||||||
|
end
|
||||||
|
|
||||||
|
# Handle unregistered UUIDs
|
||||||
|
def handle_unregistered_uuid(uuid, processors, blacklist)
|
||||||
|
if uuid.nil? || processors.where(uuid:).count.zero?
|
||||||
|
blacklist.insert(uuid:)
|
||||||
|
return 'ERROR Unrecognized UUID'
|
||||||
|
end
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
def handle_input(line, processors, orchestrators, blacklist, recently_connected)
|
||||||
command, *args = line.split
|
command, *args = line.split
|
||||||
uuid = args.shift if command != 'REGISTER'
|
uuid = args.shift if command != 'REGISTER'
|
||||||
|
|
||||||
# Check if the UUID is blacklisted
|
return 'TERMINATE' if blacklisted?(uuid, blacklist)
|
||||||
return 'TERMINATE' if blacklist.where(uuid:).count.positive?
|
|
||||||
|
|
||||||
# Check if the UUID is in the recently connected cache
|
if recently_connected?(uuid, recently_connected)
|
||||||
if recently_connected[uuid] && Time.now - recently_connected[uuid] < 120
|
|
||||||
# UUID is recently connected and within 2 minutes, no need to re-validate
|
# UUID is recently connected and within 2 minutes, no need to re-validate
|
||||||
elsif command != 'REGISTER' && (uuid.nil? || processors.where(uuid:).count.zero?)
|
|
||||||
# UUID is not recently connected or is invalid, add to blacklist
|
|
||||||
blacklist.insert(uuid:)
|
|
||||||
return 'ERROR Unrecognized UUID'
|
|
||||||
else
|
else
|
||||||
|
unregistered_response = handle_unregistered_uuid(uuid, processors, blacklist)
|
||||||
|
return unregistered_response if unregistered_response
|
||||||
|
|
||||||
# UUID is valid, update the recently connected cache
|
# UUID is valid, update the recently connected cache
|
||||||
recently_connected[uuid] = Time.now
|
recently_connected[uuid] = Time.now
|
||||||
end
|
end
|
||||||
|
@ -108,6 +112,7 @@ def register_orchestrator(line, orchestrators)
|
||||||
puts "Orchestrator registered with domain: #{domain}, port: #{port}"
|
puts "Orchestrator registered with domain: #{domain}, port: #{port}"
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# Main Async loop
|
||||||
Async do
|
Async do
|
||||||
loop do
|
loop do
|
||||||
Async::Task.new do
|
Async::Task.new do
|
||||||
|
@ -120,7 +125,7 @@ Async do
|
||||||
register_orchestrator(line, orchestrators)
|
register_orchestrator(line, orchestrators)
|
||||||
else
|
else
|
||||||
# Here we handle each line of input from the client
|
# Here we handle each line of input from the client
|
||||||
handle_input(line, processors, blacklist, recently_connected)
|
handle_input(line, processors, orchestrators, blacklist, recently_connected)
|
||||||
end
|
end
|
||||||
ensure
|
ensure
|
||||||
# This code will be executed when the fiber is finished, regardless of whether an exception was raised
|
# This code will be executed when the fiber is finished, regardless of whether an exception was raised
|
||||||
|
|
18
docker/netrave-protohandler/npeph
Normal file
18
docker/netrave-protohandler/npeph
Normal file
|
@ -0,0 +1,18 @@
|
||||||
|
#!/usr/bin/env ruby
|
||||||
|
|
||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
require_relative 'config_manager'
|
||||||
|
|
||||||
|
# Initialize the ConfigManager
|
||||||
|
config = ProtoConfig::ConfigManager.instance
|
||||||
|
|
||||||
|
# Your logic to start the Protocol Handler goes here
|
||||||
|
# For example, you could initialize your server with the IP and port from the config
|
||||||
|
# Or pass the recently_connected_time to your connection handling logic
|
||||||
|
|
||||||
|
puts "NETRAVE Packet Exchange Protocol Handler (NPEPH) is starting..."
|
||||||
|
puts "Listening on #{config.listen_ip}:#{config.listen_port}"
|
||||||
|
puts "Recently connected time threshold: #{config.recently_connected_time} seconds"
|
||||||
|
|
||||||
|
# Your server initialization and start logic here
|
|
@ -1,5 +1,5 @@
|
||||||
go.sum database tree
|
go.sum database tree
|
||||||
18982882
|
19688221
|
||||||
5hq5RwlCLIiagi6hZScdHlumu5XIpITikSz2+tAr9LA=
|
k58/ugqvqq6DqYjvo1m1COmAGb067rn6gQNOPDdtELg=
|
||||||
|
|
||||||
— sum.golang.org Az3grhw2ZJlhe/saIZhYrbHraYSRxzcFocBkNuDAACA6cT3QTOA1P1fa+rhYfXNq2KVoROv5wGw5Y/2naw4QGoEoYAE=
|
— sum.golang.org Az3grhYRJ7p1GIdTipHl5Sqe55vwSI8JugOEHx10umcUxmC5CFJxCpjrU3SFPAnm1YuxAZ3TqifbpfWXZncfjYtidgY=
|
||||||
|
|
Loading…
Reference in New Issue
Block a user