Merge pull request 'Beginning on the meat of the program' (#4) from development into main

Reviewed-on: #4

CHANGELOG.md

@@ -0,0 +1,44 @@
+# Changelog
+
+**Note:** This changelog might be incomplete. For a complete list of changes, please check the commit history in the repository.
+
+## Commits on July 30, 2023
+
+- **Implement SSL, Docker, Async Fibers, and Refactor Code for Improved Functionality** (Commit: [link-to-commit](https://git.pixelridgesoftworks.com/PixelRidge-Softworks/NETRAVE/{restoflink}))
+  - Implemented SSL to encrypt network traffic for secure communication over the open internet. This includes the creation of SSL certificates and the configuration of the server to use these certificates.
+  - Created a Dockerfile for containerizing the application. This includes setting up the base image, installing necessary dependencies, and defining the command to run the application.
+  - Added Go code for handling pcap files. This includes defining the structure of pcap files and implementing functions for reading and writing these files.
+  - Implemented Async fibers for handling multiple connections concurrently. This includes creating a new fiber for each connection and managing these fibers to ensure efficient use of resources.
+  - Added functionality to detect if the bind address is a loopback address and replace it with the correct loopback address. This allows the server to run on a single machine for testing and development purposes.
+  - Refactored the code to improve readability and maintainability. This includes breaking down complex functions into smaller, more manageable functions and improving the naming of variables and functions for clarity.
+  - Updated the code to properly send requests to the destination service. This includes creating a new socket for each request and ensuring that the request is sent over the correct connection.
+  - Added error handling to ensure that the server can recover gracefully from errors and continue to function correctly.
+
+## Commits on Jun 10, 2023
+
+- **Refactoring Database Interaction Methods** (Commit: [335cd89](https://github.com/PixelRidge-Softworks/NETRAVE/commit/335cd89ebee60252cb9948c84d981aff715b0be1))
+  - Significant refactoring of the database interaction methods in the NETRAVE project. Simplified the interaction with the database and resolved a `Sequel::DatabaseError` that was encountered during the execution of the `store_services` method.
+
+## Commits on Jun 7, 2023
+
+- **Setup a project dependant rubocop for ease of development** (Commit: [f96cf9e](https://github.com/PixelRidge-Softworks/NETRAVE/commit/f96cf9e20a2f1325be4d33a8d2ceb698011b343e))
+  - Included the project specific settings files so that fellow VS Code users can just clone down the repo, and install the needed extensions.
+
+- **Update TODO.md** (Commit: [3d2a632](https://github.com/PixelRidge-Softworks/NETRAVE/commit/3d2a632b0e812327ae7592534449c695c90c89e5))
+
+- **Merge pull request #2 from Pixelated-Studios/vetheon-patch-2** (Commit: [982c404](https://github.com/PixelRidge-Softworks/NETRAVE/commit/982c404472655cf34bf316dcf2ab7d6ae1bae266))
+  - Enhancing User Experience by Streamlining Database Connection Setup
+
+- **Enhancing User Experience by Streamlining Database Connection Setup** (Commit: [6768cf1](https://github.com/PixelRidge-Softworks/NETRAVE/commit/6768cf14b3d95e68b6150bc50419ed88e41c661b))
+  - Made significant improvements to the user experience of the NETRAVE project by streamlining the process of setting up the database connection.
+
+## Commits on Jun 5, 2023
+
+- **Merge pull request #1 from Pixelated-Studios/VetheonGames-patch-1** (Commit: [72a7775](https://github.com/PixelRidge-Softworks/NETRAVE/commit/72a77754cdba263a4fde12907f688ef0d9d7a70b))
+  - Update README.md
+
+- **Update README.md** (Commit: [28960fa](https://github.com/PixelRidge-Softworks/NETRAVE/commit/28960fa3fceea8f7d70467d9ba88dd641b9e12f8))
+
+- **Update issue templates** (Commit: [0a27925](https://github.com/PixelRidge-Softworks/NETRAVE/commit/0a279254052e09d87c897f327ed0e680e3bb4c4d), [92499ec](https://github.com/PixelRidge-Softworks/NETRAVE/commit/92499eccd59efcd4a30e5baef8d99d7e046e6045))
+
+- **Create SECURITY.md** (Commit: [6615a99](https://github.com/PixelRidge-Softworks/NETRAVE/commit/6615a99317a954888bfaa775bc09c5be6d1d6e8f))

NPEP.spec.md

@@ -0,0 +1,31 @@
+# Netrave Packet Exchange Protocol (NPEP)
+
+## Overview
+
+NPEP is a custom protocol designed for the NETRAVE project to facilitate communication between the Orchestrator and the Packet Processor Consumers. It is designed to be lightweight, efficient, and secure.
+
+## Message Types
+
+NPEP supports the following message types:
+
+1. `REQUEST`: Used by a consumer to request a new pcap chunk from the Orchestrator or to request permission to output a processed pcap chunk. The format is `REQUEST {CONSUMER_ID} {REQUEST_TYPE} {PCAP_CHUNK_ID}` where `{REQUEST_TYPE}` can be either `NEW` for a new pcap chunk or `OUTPUT` for output permission.
+
+2. `UPSTATE`: Used by a consumer to update its state in the Orchestrator. The format is `UPSTATE {CONSUMER_ID} {STATE} {PCAP_CHUNK_ID}` where `{STATE}` is the current state of the consumer. The states can be `RECEIVED`, `PROCESSING`, `STASIS`, or `AT_REST`.
+
+3. `FINISHED`: Used by a consumer to inform the Orchestrator that it has finished processing a pcap chunk. The format is `FINISHED {CONSUMER_ID} {PCAP_CHUNK_ID}`.
+
+4. `CHANGE`: Used by the Orchestrator to request a consumer to change its state. The format is `CHANGE {CONSUMER_ID} {STATE} {URGENCY}` where `{STATE}` is the state the Orchestrator wants the consumer to change to and `{URGENCY}` is the urgency level of the state change request. The urgency levels can be `PLS` (Please), `SOON`, or `NOW`.
+
+5. `SHUTDOWN`: Used by the Orchestrator to request a consumer to shut down. The format is `SHUTDOWN {CONSUMER_ID} {URGENCY}` where `{URGENCY}` is the urgency level of the shutdown request. The urgency levels can be `PLS` (Please), `SOON`, or `NOW`.
+
+## Security
+
+To ensure only legitimate consumers can connect to the Orchestrator, a unique `{HASH}` is generated by the Orchestrator and used to authenticate each consumer. This `{HASH}` is included in the `REQUEST` message.
+
+## Error Handling
+
+If a consumer crashes or disconnects unexpectedly, the Orchestrator will detect this through a lack of heartbeat messages and reassign the pcap chunk to another consumer.
+
+## Data Transmission
+
+NPEP is designed to transmit pcap data in its raw binary form to minimize overhead and maximize efficiency. The Orchestrator sends pcap data to the consumers using the `SEND PCAP {PCAP_CHUNK_ID} {PCAP_DATA}` message, where `{PCAP_DATA}` is the raw pcap data.

README.md

@@ -12,6 +12,7 @@ Come chat with us on [![Discord](https://img.shields.io/discord/8656525939326321
 7. [Contact](#contact)
 8. [Conclusion](#conclusion)
 9. [Special Thanks](#special-thanks)
+10. [NPEP Specification](https://git.pixelridgesoftworks.com/PixelRidge-Softworks/NETRAVE/NPEP.spec.md)
 
 ## Introduction
 Welcome to NETRAVE, a cybersecurity solution designed to provide a high level of security, vigilance, and response to threats, ensuring that network environments remain secure and resilient. This README provides a comprehensive overview of the software, its purpose, functions, objectives, and unique methods of handling issues/challenges.
@@ -25,9 +26,11 @@ Connor Crawford, also known as Vetheon, is currently the only developer behind N
 ## Technical Overview
 NETRAVE is designed with a modular architecture, which allows for easy customization, upgrades, and maintenance. Each module in NETRAVE is designed to perform a specific function, and they can work independently or in harmony, depending on the needs of the user.
 
-The software is built using Ruby, a high-level, interpreted programming language that emphasizes simplicity and productivity. Ruby's dynamic typing and object-oriented design make it an ideal choice for a modular system like NETRAVE.
+The software is built using a combination of Ruby and Go. Ruby, a high-level, interpreted programming language that emphasizes simplicity and productivity, is used for the main orchestrator and the overall structure of the software. Ruby's dynamic typing and object-oriented design make it an ideal choice for a modular system like NETRAVE.
 
-Recent updates include the implementation of a secure sudo system, and the creation of a NetworkingGenie class to handle network interface setup and traffic mirroring. The AlertQueueManager class has also been updated to use a standard Array queue instead of a Ring Buffer for managing the queue of alerts.
+Go, a statically typed, compiled language known for its efficiency and concurrency, is used for the packet capture and processing modules. These modules are compiled into shared C libraries using cgo, a feature of Go that enables the creation of Go packages that call C code. These shared libraries are then accessed from Ruby using the Foreign Function Interface (FFI), which allows Ruby code to call functions written in other languages directly.
+
+This combination of Ruby and Go, along with the use of shared C libraries, allows NETRAVE to leverage the strengths of both languages. The simplicity and productivity of Ruby for the overall structure and orchestration, and the efficiency and concurrency of Go for the performance-critical packet capture and processing tasks.
 
 ## Modular Design
 The modular design of NETRAVE is one of its key features. Each module is designed to perform a specific function, and they can work independently or in harmony, depending on the needs of the user. This design allows for easy customization, upgrades, and maintenance.
@@ -37,10 +40,10 @@ The modular design also allows for easy expansion. New modules can be added to t
 ## Contributing
 NETRAVE is an open-source project, and contributions are welcome. If you're interested in contributing, please fork the repository and make your changes. Once you're done, submit a pull request. All contributions, no matter how small, are greatly appreciated.
 
-If you're looking for something to do, you can have a look in our [TODO List](https://github.com/Pixelated-Studios/NETRAVE/blob/main/TODO.md)
+If you're looking for something to do, you can have a look in our [TODO List](https://git.pixelridgesoftworks.com/PixelRidge-Softworks/NETRAVE/blob/main/TODO.md)
 
 ## Contact
-Connor is always open to receiving questions. If you have any questions, suggestions, or just want to say hi, you can reach him at vetheon@pixelatedstudios.net or on discord via VetheonGames#0001.
+Connor is always open to receiving questions. If you have any questions, suggestions, or just want to say hi, you can reach him at connorc@pixelridgesoftworks.com or on discord via VetheonGames#0001.
 
 ## Conclusion
 NETRAVE is a robust, modular cybersecurity solution that provides a high level of security, vigilance, and response to threats. Its open-source nature and modular design make it a flexible and adaptable solution that can meet the changing needs of any network environment. Whether you're a network administrator looking for a comprehensive security solution, or a developer interested in contributing to an open-source project, NETRAVE has something to offer you.

docker/netrave-protohandler/Gemfile

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+gem 'async'
+gem 'sequel'

docker/netrave-protohandler/netrave-protohandler.dockerfile

@@ -0,0 +1,20 @@
+# Dockerfile
+FROM ruby:3.2.2
+
+# Set environment variables for IP and Port
+ENV LISTEN_IP=0.0.0.0
+ENV LISTEN_PORT=3080
+ENV ORCHESTRATOR_DOMAIN=orchestrator_domain
+ENV ORCHESTRATOR_PORT=orchestrator_port
+
+# Set the working directory in the container
+WORKDIR /netrave-protohandler
+
+# Copy the current directory contents into the container at /netrave-protohandler
+COPY . /netrave-protohandler
+
+# Install any needed packages specified in Gemfile
+RUN bundle install
+
+# Run server.rb when the container launches
+CMD ["ruby", "netrave-protohandler.rb"]

docker/netrave-protohandler/netrave_protohandler.rb

@@ -0,0 +1,128 @@
+# frozen_string_literal: true
+
+require 'socket'
+require 'async'
+require 'sequel'
+require 'openssl'
+
+# Set up the database
+DB = Sequel.sqlite # In-memory database
+processors = DB[:processors]
+
+listen_ip = ENV['LISTEN_IP'] || '0.0.0.0'
+listen_port = ENV['LISTEN_PORT'] || 3080
+
+server = TCPServer.new(listen_ip, listen_port)
+
+# This hash will store the connections to the consumers
+connections = {}
+
+def create_socket(ip, port) # rubocop:disable Metrics/MethodLength
+  # If the IP address is set to "loopback", replace it with the actual loopback IP address
+  ip = '127.0.0.1' if ip.downcase == 'loopback'
+
+  if ip =~ Resolv::IPv4::Regex
+    # If the IP address is an IPv4 address, create an unencrypted socket
+    TCPSocket.new(ip, port)
+  else
+    # If the IP address is a domain name, create an SSL socket
+    ssl_context = OpenSSL::SSL::SSLContext.new
+    ssl_context.verify_mode = OpenSSL::SSL::VERIFY_PEER
+
+    tcp_socket = TCPSocket.new(ip, port)
+    ssl_socket = OpenSSL::SSL::SSLSocket.new(tcp_socket, ssl_context)
+    ssl_socket.sync_close = true
+    ssl_socket.connect
+    ssl_socket
+  end
+end
+
+Async do
+  loop do
+    Async::Task.new do
+      client = server.accept
+
+      begin
+        while (line = client.gets)
+          # Here we handle each line of input from the client
+          handle_input(line, connections)
+        end
+      ensure
+        # This code will be executed when the fiber is finished, regardless of whether an exception was raised
+        client.close
+        Async::Task.current.stop # Stop the current task
+      end
+    end
+  end
+end
+
+def handle_input(line, connections) # rubocop:disable Metrics/AbcSize, Metrics/MethodLength
+  # Split the line into command and args
+  command, *args = line.split
+
+  case command
+  when 'NEW_PROCESSOR'
+    # Handle new processor registration
+    id, domain, port = args
+
+    # Create a new connection to the processor
+    Async do
+      processor_connection = create_socket(domain, port)
+
+      # Store the connection in the hash
+      connections[id] = processor_connection
+
+      # Add the processor to the database
+      processors.insert(consumer_id: id, ip: domain, port: port)
+    end
+
+  when 'REQUEST'
+    # Handle request from orchestrator
+    consumer_id, request_type, pcap_chunk_id = args
+
+    # Get the connection for this consumer
+    connection = connections[consumer_id]
+
+    # Send the request to the consumer
+    connection.puts "REQUEST #{request_type} #{pcap_chunk_id}"
+
+  when 'UPSTATE'
+    # Handle upstate from consumer
+    consumer_id, state, pcap_chunk_id = args
+
+    # Do something with the state update...
+    # For now, we'll just print it out
+    puts "Consumer #{consumer_id} is now #{state} for chunk #{pcap_chunk_id}"
+
+  when 'FINISHED'
+    # Handle finished from consumer
+    consumer_id, pcap_chunk_id = args
+
+    # Do something with the finished message...
+    # For now, we'll just print it out
+    puts "Consumer #{consumer_id} has finished chunk #{pcap_chunk_id}"
+
+  when 'CHANGE'
+    # Handle change from orchestrator
+    consumer_id, state, urgency = args
+
+    # Get the connection for this consumer
+    connection = connections[consumer_id]
+
+    # Send the change to the consumer
+    connection.puts "CHANGE #{state} #{urgency}"
+
+  when 'SHUTDOWN'
+    # Handle shutdown from orchestrator
+    consumer_id, urgency = args
+
+    # Get the connection for this consumer
+    connection = connections[consumer_id]
+
+    # Send the shutdown to the consumer
+    connection.puts "SHUTDOWN #{urgency}"
+
+  else
+    puts "Unknown command: #{command}"
+  end
+end

go/go.mod

@@ -0,0 +1,8 @@
+module pixelridgesoftworks.com/NETRAVE
+
+go 1.20
+
+require (
+	github.com/google/gopacket v1.1.19 // indirect
+	golang.org/x/sys v0.0.0-20190412213103-97732733099d // indirect
+)

go/go.sum

@@ -0,0 +1,15 @@
+github.com/google/gopacket v1.1.19 h1:ves8RnFZPGiFnTS0uPQStjwru6uO6h+nlr9j6fL7kF8=
+github.com/google/gopacket v1.1.19/go.mod h1:iJ8V8n6KS+z2U1A8pUwu8bW5SyEMkXJB8Yo/Vo+TKTo=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
+golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

go/packet-cap.go

@@ -0,0 +1,34 @@
+package main
+
+import (
+	"fmt"
+	"log"
+	"time"
+
+	"github.com/google/gopacket"
+	"github.com/google/gopacket/pcap"
+)
+
+var (
+	device       string = "netrave0"
+	snapshot_len int32  = 2048
+	promiscuous  bool   = false
+	err          error
+	timeout      time.Duration = 30 * time.Second
+	handle       *pcap.Handle
+)
+
+func main() {
+	// Open device
+	handle, err = pcap.OpenLive(device, snapshot_len, promiscuous, timeout)
+	if err != nil {
+		log.Fatal(err)
+	}
+	defer handle.Close()
+
+	// Use the handle as a packet source to process all packets
+	packetSource := gopacket.NewPacketSource(handle, handle.LinkType())
+	for packet := range packetSource.Packets() {
+		fmt.Println(packet) // Do something with a packet here.
+	}
+}

lib/pkg/sumdb/sum.golang.org/latest

@@ -0,0 +1,5 @@
+go.sum database tree
+18772708
+zUQL+Z/7glb/JRJWTIOwKpTurAD4UkqBoSM/R6Wxx1U=
+
+— sum.golang.org Az3grlYd+7jVJcF9/zAb4Pytelt3fRZySKTY7/bjVB7mPom5EO6a2ir09DWokk6RPoUjmc7yd4zez+i6mUr2sYCddg8=

lib/utils/packet_capture.rb

@@ -1,47 +0,0 @@
-# frozen_string_literal: true
-
-require 'pcaprub'
-require 'socket'
-require_relative 'databasemanager'
-require_relative 'logg_man'
-require_relative 'redis_queue'
-
-# Class used to capture packets and not much else
-class PacketCapture
-  INTERFACE_NAME = 'netrave0'
-
-  def initialize(queue, logger)
-    @loggman = logger
-    @loggman.log_info("Initializing packet capture for #{INTERFACE_NAME}...")
-    @capture = Pcap.open_live(INTERFACE_NAME, 65_535, true, 1)
-    @capture.setfilter('')
-    @loggman.log_info('Packet capture initialized successfully!')
-    @queue = queue
-  end
-
-  def start_capture_loop # rubocop:disable Metrics/MethodLength
-    @loggman.log_info("Starting packet capture loop for #{@interface}...")
-    packet_count = 0
-    begin
-      @loggman.log_info("Packet capture loop started for #{@interface}...")
-      @capture.each_packet do |packet|
-        # Add packet to queue
-        @queue.push(packet)
-        @loggman.log_info("Packet #{packet_count += 1} added to queue.")
-      end
-    rescue StopIteration
-      @loggman.log_warn("Packet capture loop stopped for #{@interface}.")
-    rescue StandardError => e
-      @loggman.log_fatal("Packet capture loop stopped for #{@interface}: #{e.message}\n#{e.backtrace}", false)
-      sleep 1
-      retry
-    ensure
-      @capture.close
-    end
-  end
-
-  def stop_capture
-    @loggman.log_warn("Stopping packet capture loop for #{@interface}...")
-    @stop_flag = true
-  end
-end