So, let's get into it.
First, we are straying from the in-memory ideation of storing all our pcap chunks in the memory
I fear that we might bottle up the ram if we depend solely on it.
So, i've made things default to sqlite flatfile, with the option for both remote and in memory
This way, dependant on the users needs, they can use a different implementation.
That's why the creation of a great deal of files and an entire new directory space
I've almost finished implementing the database system into itself, but I want the DB system
to be largely independent of the rest of the system. Because, you know me,
I am a fan of encapsulation. and I like making software coded to be independent.
Yes, if the rest of the system isn't working, the database has nothing to docker
but at least it makes the system more robust, if even a major system can collapse,
and the system as a whole remains functional.
Philisophically, I find this is largely lacking in modern software, and I believe
that these practices will see the protohandler performing much better than we would expect
Since this system as a whole will be independent of the main NETRAVE system,
it opens the door for further expanding, or perhaps even completely repurposing
this protocol going forward.
Fix the proto_handler to separate request handling from input handling
Make the proto_handler just pass messages between. We don't need to actually process anything
Add some logic to determine if the request is coming from the orchestrator or consumer
Some other various proto_handler fixes
Setup registration for orchestrators
make the proto_handler properly use it's in-memory DB for registrations
Ensure the Proto_handler goes both ways
Add a cache system so that consumers don't need to constantly re-register
Add a blacklist system to prevent unauthorized Consumers from getting access to the message queue
Add a ID assignment for each registered consumer to the proto_handler so we can keep track of them in the proto_handler
Add a async task to actually handle the input from the consumers
This commit includes several significant changes:
1. Implemented SSL to encrypt network traffic for secure communication over the open internet. This includes the creation of SSL certificates and the configuration of the server to use these certificates.
2. Created a Dockerfile for containerizing the application. This includes setting up the base image, installing necessary dependencies, and defining the command to run the application.
3. Added Go code for handling pcap files. This includes defining the structure of pcap files and implementing functions for reading and writing these files.
4. Implemented Async fibers for handling multiple connections concurrently. This includes creating a new fiber for each connection and managing these fibers to ensure efficient use of resources.
5. Added functionality to detect if the bind address is a loopback address and replace it with the correct loopback address. This allows the server to run on a single machine for testing and development purposes.
6. Refactored the code to improve readability and maintainability. This includes breaking down complex functions into smaller, more manageable functions and improving the naming of variables and functions for clarity.
7. Updated the code to properly send requests to the destination service. This includes creating a new socket for each request and ensuring that the request is sent over the correct connection.
8. Added error handling to ensure that the server can recover gracefully from errors and continue to function correctly.
9. Created code for use in the ProtocolHandler
10. Create various empty dockerfiles for the other containerized services
11. Create various empty code files for the code to run in the other containerized services
12. Added a Gemfile for each Containerized Service for independant dependency management
13. Defined how the server should respond to NPEP queries
14. Roughed in the systems for using a FQDN & SSL secured internet for data transfer, or loopback for local connections
15. Setup system for requiring at least one Orchestrator to be connected manually (The "Primary")
16. Added documentation for the NPEP protocol
17. Rebase the file structure to keep things more organized
