This commit includes several significant changes:
1. Implemented SSL to encrypt network traffic for secure communication over the open internet. This includes the creation of SSL certificates and the configuration of the server to use these certificates.
2. Created a Dockerfile for containerizing the application. This includes setting up the base image, installing necessary dependencies, and defining the command to run the application.
3. Added Go code for handling pcap files. This includes defining the structure of pcap files and implementing functions for reading and writing these files.
4. Implemented Async fibers for handling multiple connections concurrently. This includes creating a new fiber for each connection and managing these fibers to ensure efficient use of resources.
5. Added functionality to detect if the bind address is a loopback address and replace it with the correct loopback address. This allows the server to run on a single machine for testing and development purposes.
6. Refactored the code to improve readability and maintainability. This includes breaking down complex functions into smaller, more manageable functions and improving the naming of variables and functions for clarity.
7. Updated the code to properly send requests to the destination service. This includes creating a new socket for each request and ensuring that the request is sent over the correct connection.
8. Added error handling to ensure that the server can recover gracefully from errors and continue to function correctly.
9. Created code for use in the ProtocolHandler
10. Create various empty dockerfiles for the other containerized services
11. Create various empty code files for the code to run in the other containerized services
12. Added a Gemfile for each Containerized Service for independant dependency management
13. Defined how the server should respond to NPEP queries
14. Roughed in the systems for using a FQDN & SSL secured internet for data transfer, or loopback for local connections
15. Setup system for requiring at least one Orchestrator to be connected manually (The "Primary")
16. Added documentation for the NPEP protocol
17. Rebase the file structure to keep things more organized
This commit introduces several significant enhancements to the way commands are executed and logged in the application. The changes are primarily focused on improving the robustness, reliability, and transparency of the command execution process, as well as enhancing the quality and usefulness of the log output.
1. Command Execution Enhancements: The use_sudo method has been refactored to handle commands that do not return any output. Previously, the method was designed to capture and return the output of the command being executed. However, some commands (such as modprobe) do not return any output, which caused issues with the previous implementation. The method now checks the exit status of the command to determine whether it was successful or not, and returns a success or failure message accordingly. This change improves the robustness of the command execution process and ensures that it can handle a wider range of commands.
2. Error Handling Improvements: The use_sudo method now includes more comprehensive error handling. If a command fails to execute within a specified timeout period, an error message is logged and the method returns a failure message. Additionally, if a command fails to execute for any other reason, the method logs the error and returns a failure message with the command's exit status. These changes make it easier to identify and troubleshoot issues with command execution.
3. Logging Enhancements: The logging mechanism has been enhanced to provide more detailed and useful information. The use_sudo method now logs the command being executed and its outcome (success or failure). If a command fails, the method logs the command's exit status. These changes improve the transparency of the command execution process and make it easier to identify and troubleshoot issues.
4. Code Refactoring: Several methods have been refactored for improved readability and maintainability. The use_sudo method has been refactored to reduce its complexity and improve its readability. The first_run_setup method has been refactored to ensure that the main interface name and the dummy interface name are properly passed to the setup_traffic_mirroring method.
5. Bug Fixes: A bug in the create_dummy_interface method that caused it to return an array of Alert objects instead of the dummy interface name has been fixed. The method now correctly returns the dummy interface name.
These changes represent a significant improvement to the command execution and logging mechanisms in the application, and lay the groundwork for further enhancements in the future.
In the AlertQueueManager class of the NETRAVE project, we made a
significant change to improve code simplicity, efficiency, and maintainability.
We replaced the previously used Ring Buffer with a standard Array queue for
managing the queue of alerts. This change simplifies the code structure and
improves the efficiency of queue management. As a result of this change, the
Ring Buffer file, which is no longer needed, was also removed from the project.
I would also like to thank Havenwood, Lapizistik, Sampersand, and Crimson
from the Ruby discord in helping me determine the best data structure to use
and helping me generally expand my deeper knowledge of Ruby
This commit introduces substantial enhancements to both the NetworkingGenie class and the Utilities module, focusing on the secure usage of sudo permissions, improved error handling, and the creation of new sudo methods.
1. Secure Sudo Usage: The `use_sudo` method from the Utilities module is now integrated within the `find_main_interface`, `create_dummy_interface`, and `setup_traffic_mirroring` methods of the NetworkingGenie class. This method ensures that commands requiring sudo permissions are executed securely. It achieves this by encrypting the sudo password, using it for the required command, and then immediately clearing it from memory. This approach significantly enhances the security of operations that require elevated permissions.
2. New Sudo Methods in Utilities Module: This commit introduces several new methods in the Utilities module to handle sudo operations securely. The `ask_for_sudo`, `test_sudo`, `deescalate_sudo`, and `use_sudo` methods have been created. These methods handle the secure acquisition, testing, de-escalation, and usage of sudo permissions, respectively. They ensure that sudo operations are performed securely and efficiently, with the sudo password being encrypted and cleared from memory immediately after use.
3. Improved Error Handling: The error handling within the `setup_traffic_mirroring` method has been refined. Now, when an exception is raised during the execution of a command, the error message is not only logged using the LoggMan logger but also an alert is enqueued into the AlertQueueManager. This dual approach ensures that errors are properly logged for debugging purposes and also communicated to the user in real-time.
4. AlertQueueManager Integration: The `initialize` method of NetworkingGenie has been updated to accept an `alert_queue_manager` parameter. This allows the NetworkingGenie class to enqueue alerts directly into the AlertQueueManager, thereby improving the communication of errors and important information to the user.
5. Dummy Interface Creation: The `create_dummy_interface` method now checks if the dummy interface already exists before attempting to create it. This prevents unnecessary system calls and potential errors.
These modifications significantly contribute to the overall security and reliability of the NETRAVE system. They ensure that network setup and traffic mirroring operations are performed securely and efficiently, with clear and immediate communication of any issues that may occur.
This commit marks a significant milestone in the NETRAVE project. The first major system has been successfully completed, laying a solid foundation for the subsequent development phases.
Key changes include:
1. Completion of the Initial Setup Process: The initial setup process has been successfully implemented. This process gathers necessary information from the user and sets up the basic environment for the application to run.
2. Removal of Redundant Thread in Alert System: An unnecessary thread in the alert system was identified and removed. This change simplifies the alert system's architecture and reduces the potential for concurrency-related issues.
3. Program Efficiency Improvements: Several optimizations have been made to enhance the efficiency of the program. These include changes to the way alerts are handled and displayed, as well as improvements to the handling of the alert queue.
These changes have been verified and tested to ensure they work as expected and contribute to the overall functionality and performance of the NETRAVE project.
This commit introduces a significant enhancement to our Ruby program by implementing a thread-safe alert system using a ring buffer data structure.
New Classes:
1. Alert: This class is responsible for creating and displaying alerts in the Curses Text User Interface (TUI). It takes a message and severity level as arguments and uses these to display color-coded alerts to the user.
2. AlertQueueManager: This class manages the queues for alerts using a ring buffer data structure. It continuously checks the queue and displays alerts as they arrive. It uses a mutex lock to ensure thread safety when accessing the ring buffer.
3. RingBuffer: This class is a custom implementation of a ring buffer, also known as a circular buffer. It's a fixed-size buffer that effectively overwrites old data when it is full. The buffer size has been optimized to 2MB to balance memory usage and performance.
Modifications to Existing Methods:
The DatabaseManager class has been updated to integrate the new alert system. The methods in this class now create Alert instances and enqueue them in the AlertQueueManager instead of directly displaying alerts to the user. This change ensures that alerts are displayed in a thread-safe manner and allows for better control over the timing and order of alert displays.
Thread Safety Measures:
Mutex locks and condition variables have been used to synchronize access to the ring buffer and prevent race conditions. This ensures that only one thread can access the buffer at a time, preventing data corruption and ensuring the correct operation of the alert system.
Testing:
Rigorous testing has been conducted to validate the correct functioning of the new system and to handle edge cases. This includes tests for the correct display of alerts, the correct operation of the ring buffer, and the correct synchronization of threads.
Documentation:
Detailed comments have been added to the code to explain the purpose and operation of the new classes and methods. This documentation will serve as a valuable reference for future development and maintenance of the codebase.
This commit represents a significant improvement in the functionality and robustness of our Ruby program's alert system.
1. Initial System Setup:
- Implemented a first run initialization process that guides the user through setting up the necessary environment variables.
- Created a method to securely ask for the user's sudo password, test it, and store it in an encrypted form in an environment variable for use during the first run setup process.
- Added a method to clear the sudo password from memory and the environment variables at the end of the first run setup process.
2. Packet Capture:
- Created a PacketCapture class that uses the PCAPRUB library to capture packets from a specified network interface.
- Refactored the packet capture process to add each captured packet to a Redis queue for further processing, instead of processing the packets directly.
- Removed the manual packet dissection from the packet capture process, as this will be handled by the workers.
3. Networking Setup:
- Created a NetworkingGenie class to handle the setup of the necessary networking components.
- Added methods to identify the main network interface, create a dummy network interface, and set up traffic mirroring from the main interface to the dummy interface.
4. Logging:
- Implemented logging for all major actions and errors throughout the system.
5. General Refactoring and Code Cleanup:
- Refactored and cleaned up various parts of the code to improve readability and maintainability.
- Fixed various minor bugs and issues.
This commit lays the groundwork for the packet processing workers and the orchestrator that will manage them. The next steps will be to implement these components and integrate them with the existing system.
- Refactored the entry-point to bypass first run setup if valid configuration information already exists.
- Enhanced data validation before inserting into the database.
- Ensured correct data placement in the appropriate tables and columns in the database.
- Added logging for database connection attempts and results.
- Fixed a bug in the speed conversion from Gbps to Mbps.
This commit includes several significant changes aimed at improving the functionality, user experience, and code quality of our Ruby project.
1. Decryption Fix: We identified and resolved an issue where the database password was being decrypted multiple times, leading to connection problems. The code was refactored to ensure that decryption only occurs once, thereby enhancing the efficiency and reliability of our database connections.
2. Infinite Loop Resolution: We addressed a critical issue where the program would enter an infinite loop if the .env file was missing or contained incorrect information. The code was updated to handle these situations appropriately, providing meaningful feedback to the user and preventing unnecessary resource consumption.
3. .env File Handling Improvement: We improved the handling of the .env file, ensuring that the program can function correctly even in the absence of this file or if it contains incorrect data. This change enhances the robustness of our application.
4. Curses Alerts Integration: We integrated a new feature to display alerts in the terminal using the Curses library. These alerts can have different severity levels (info, warning, error), which are displayed in different colors (blue, yellow, red). This feature improves the user experience by providing clear and immediate feedback on the program's status.
5. Automatic Alert Dismissal: We implemented a feature where alerts automatically disappear after 5 seconds. This was achieved using Ruby threads, ensuring that the rest of the program is not blocked while the alert is displayed. This change enhances the user experience by preventing the screen from being cluttered with old alerts.
6. Debugging Libraries Exploration: We explored the possibility of using the Tracer and Debug libraries to trace the execution of the program and assist with debugging. While these libraries were not integrated in this commit, they remain a potential resource for future debugging efforts.
This commit represents a significant step forward in the development of our Ruby project, improving both the user experience and the quality of our codebase.
This commit introduces several significant changes to improve the security and configuration management of the application.
Encryption of Sensitive Data: We've introduced encryption for sensitive data such as database passwords. This is done using the Blowfish algorithm. The encryption and decryption methods are added to the Utilities module. The encryption is applied when the user enters their database password during the first run setup. The encrypted password is then stored in the .env file. When the application needs to use the password, it is decrypted using the key that was generated during encryption.
Switch from YAML to .env for Configuration Storage: The application now uses a .env file for storing configuration data instead of a YAML file. This change was made to take advantage of the .env file's ability to store environment variables, which can be easily accessed by the application. The .env file stores the database username, encrypted password, encryption key, and database name.
First Run Setup Changes: The FirstRunInit class has been updated to ask the user for their database details, encrypt the password, and store these details in the .env file. It also checks if any of the necessary details are missing and runs the first run setup if they are.
Database Connection Testing: The DatabaseManager class now decrypts the password before using it to test the database connection. It also checks if the password or key are nil before attempting to decrypt the password.
Logging: A new LoggMan class has been added to handle logging. This class provides a simple method for logging messages, which can be used throughout the application to log errors or other important information.
Bug Fixes and Error Handling: Several bugs were fixed and error handling was improved. For example, a bug that caused the application to crash if the .env file was missing has been fixed. Also, the application now checks if the .env file exists and creates it if it doesn't.
These changes significantly improve the security, reliability, and usability of the application.
1. **Refactoring of Database Interaction Methods**
- Refactored the `store_services` method in the `DatabaseManager` class to handle an array of services instead of a hash. This change was made to simplify the interaction with the database and to avoid unnecessary complexity in the data structure.
- The `store_services` method now iterates over an array of services and inserts each service into the database with a default status of true. This design choice was made to ensure that all services are active by default.
2. **Modification of Database Schema**
- Modified the `create_services_table` method in the `DatabaseManager` class to create a table with only two columns: `id` and `services`. This change was made to align the database schema with the new data structure used in the `store_services` method.
- The `status` column was removed from the `services` table because the status of all services is now assumed to be true by default.
3. **Error Handling and Debugging**
- Encountered a `Sequel::DatabaseError` with the message "Operand should contain 1 column(s)" during the execution of the `store_services` method. This error was caused by an attempt to insert a hash into a single database column.
- The error was resolved by refactoring the `store_services` method to handle an array of services instead of a hash.
4. **Unorthodox Design Choices**
- The decision to use an array of services instead of a hash and to assume that the status of all services is true by default may seem unorthodox. However, these design choices were made to simplify the interaction with the database and to avoid unnecessary complexity in the data structure.
- These design choices also helped to resolve the `Sequel::DatabaseError` that was encountered during the execution of the `store_services` method.
This commit represents a significant refactoring of the database interaction methods in the NETRAVE project. The changes made in this commit have simplified the interaction with the database and have resolved a `Sequel::DatabaseError` that was encountered during the execution of the `store_services` method.
In this commit, we have made significant improvements to the user experience of the NETRAVE project by streamlining the process of setting up the database connection. This was achieved by modifying the first_run_setup method in the FirstRunInit class to intelligently handle the presence or absence of a configuration file (config.yml).
The key changes include:
Checking for the existence of config.yml: Before prompting the user for database details, the program now checks if a config.yml file already exists. This file holds the necessary details for connecting to the database.
Reading and using existing configuration: If a config.yml file is found, the program attempts to connect to the database using the details from the file. This eliminates the need for the user to re-enter database details every time the program runs, thereby improving the user experience.
Handling unsuccessful connections: If the connection attempt using the details from config.yml is unsuccessful, the program prompts the user for new database details. These new details are then used to overwrite the existing config.yml file. This ensures that the program can recover from changes in the database setup, such as a change in password.
Creating config.yml if it does not exist: If no config.yml file is found, the program creates one and then prompts the user for the database details. This makes the initial setup process smoother for new users.
These changes have several benefits:
Improved User Experience: By avoiding unnecessary prompts for database details, the program becomes more user-friendly.
Efficient Onboarding: The streamlined process makes onboarding new users more efficient.
Cost Savings: By reducing the number of requests for help with database details, we can potentially save on support staff time.
Employee Satisfaction: Improving user experience can lead to greater employee satisfaction and advocacy for the product.
Ethical Considerations: These changes respect the rights of the users by giving them control over their personal data and ensuring transparency in how their data is handled.
This commit represents a significant step forward in making the NETRAVE project more user-friendly and efficient. We believe these changes will improve the usage and adoption of the product while also respecting the rights of the users.
In this commit, we've made substantial changes to the DatabaseManager and SystemInformationGather classes to improve the functionality, efficiency, and maintainability of the system.
Refactoring DatabaseManager: The DatabaseManager class was refactored to improve the separation of concerns. Initially, the DatabaseManager was responsible for creating an instance of SystemInformationGather, which was not ideal as it violated the Single Responsibility Principle. The responsibility of creating an instance of SystemInformationGather was moved to the FirstRunInit class, which is more appropriate as it is responsible for the initial setup of the system. This change improves the maintainability of the code and makes it easier to understand and modify in the future.
Adding Database Connection Test: A method test_db_connection was added to the DatabaseManager to test the database connection before attempting to interact with it. This method improves the robustness of the system by ensuring that a valid connection exists before proceeding. It also provides a better user experience by providing a clear error message if the connection fails.
Refactoring SystemInformationGather: The SystemInformationGather class was refactored to improve its functionality and efficiency. The methods ask_for_uplink_speed and ask_for_downlink_speed were modified to convert the user's input to Mbps immediately, reducing the need for conversion later. This change improves the efficiency of the system by reducing unnecessary conversions.
Adding Services Table: A new table for services was added to the database. This table stores the services that the system should be aware of, with each service represented as a boolean value. This change improves the flexibility of the system by allowing it to handle a variable number of services. It also improves the efficiency of the system by reducing the need to parse the services from a string each time they are needed.
Storing Total Bandwidth: The total bandwidth (the sum of the uplink and downlink speeds) is now calculated and stored in the system_info table. This change improves the efficiency of the system by reducing the need to calculate the total bandwidth each time it is needed.
Error Handling and Debugging: Throughout the process, various bugs and errors were encountered and fixed. These included issues with method arguments, missing method calls, and incorrect method usage. Fixing these issues improved the stability and reliability of the system.
In conclusion, this commit significantly improves the functionality, efficiency, and maintainability of the system. The changes made adhere to good software engineering principles, such as the Single Responsibility Principle, and make the system more robust and user-friendly.
In this commit, we have undertaken a significant refactoring of the codebase to improve modularity, promote code reuse, and consider security aspects.
Modularity and Code Reuse: We introduced classes and modules to better organize the code and facilitate code reuse. Specifically, we created a new Utilities module to hold common methods that are used across different parts of the application. This module includes methods such as convert_speed_to_mbps, services_to_hash, and calculate_total_bandwidth. By placing these methods in a module, we can include this module in any class that needs these methods, thereby promoting code reuse and reducing duplication.
We also created two new classes, SystemInformationGather and DatabaseManager. The SystemInformationGather class is responsible for gathering system information, such as uplink and downlink speed, and services. The DatabaseManager class handles database operations, including testing the database connection, creating the system information table, and storing system information in the database. By encapsulating these responsibilities within their respective classes, we have made the code more organized and easier to maintain.
Database Connection Details: We made a decision to store the database connection details in a config file (config.yml) instead of the database itself. This decision was made for security reasons, as storing these details in the database could expose them to potential security risks. Storing these details in a config file allows us to better control access to these details.
Database Encryption: We discussed the possibility of encrypting the database for additional security. While we have not implemented this feature in this commit, we have laid the groundwork for it by discussing potential libraries/gems that could be used for this purpose. This is an area that we will revisit in the future.
System Information Gathering: We wrote methods to gather system information, such as uplink and downlink speed, and services, and store this information in the database. This information will be useful for monitoring system performance and for making decisions about resource allocation.
This refactoring effort has made the codebase more organized, easier to maintain, and more secure. It also aligns with our project goals and roadmap. However, it's important to note that with any refactoring effort, there may be potential issues or trade-offs. We will need to thoroughly test the refactored code to ensure that it works as expected and that there are no unintended side effects.
1. Added a method to ask for the default mode (TUI, GUI, or WebApp) during the first run setup.
2. Implemented a method to validate the input mode.
3. Improved the database connection test method to handle exceptions and return a boolean value.
4. Added a method to ask for database details (username, password, and database name).
5. Enhanced the user interface by adding Curses.clear before each question to make the interface cleaner.
6. Improved the password input process by disabling echo to hide the input from the screen.
7. Added validation for uplink and downlink speeds.
8. Added a method to ask for services the system should be aware of.
