Core fixes. Provide global variables. Update runner. Upgrade module-package system. (#14)

* Clean init all the modules with the __init__.py file

* Remove unused gitkeeps

* Add base inits for script packages

* Update main file

* Set __file__ variable at the module level

* Add inits for the modules

* Modify all the modules

* Delete one more .gitkeep

* Add WIP test for user_greeting

* Update test module for user_greeting

* Fix module runner a bit

* Add relative imports to the modules

* Add run tests script

* Update README.md

* Fix module-package runner

* Update requirements

* Format everything with Black

* Handle module and execution errors with runner

Author: manmolecular

README.md

@@ -6,11 +6,79 @@
 virtualenv -p python3 venv
     (or python3 -m venv venv)
 pip3 install -r requirements.txt
+```
+
+## Running as a framework
+_First of all: provide some arguments in the `main.py` file to collect information based on your data (WIP now, will be improved later)_  
+
+To run the framework:
+```bash
 python3 main.py
 ```
+To run the tests:
+```bash
+chmod +x run_tests.sh
+./run_tests.sh
+```
+_or you can run them like this, of course:_
+```bash
+python3 -m unittest discover -v
+```
+## Running as a separated module
+Basic:
+```python3
+python3 -m src.scripts.<category>.<name> any_arguments_here
+```
+Example command:
+```bash
+python3 -m src.scripts.other.user_greeting JohnDoe
+```
+Example output:
+```
+{'message': "Successfully finished! (args: (), kwargs: {'username': "
+            "'johndoe'})",
+ 'result': 'Hello, JohnDoe!',
+ 'status': 'success'}
+
+```
 
 ## Create your own script
 Use the following structure:  
+1. Create your own module directory in the following way:
+```
+/src/scripts/<choose_your_category_here>/<your_script_name>/<script_files>
+```
+2. Provide the following structure of your script directory:
+```
+your_script_name
+├── __init__.py     - use this module to set the default parent directory (you can copy this file from any other script)
+├── __main__.py     - use this module to provide some basic interface to use your script as a module (the same as if __name__ == "__main__")
+├── module.py       - use this module to describe the basic logic of your module (you can import it in the __main__.py to provide interface)
+└── test_module.py  - use this module for unittest tests
+```
+3. Create the `__init__.py` file. An example of the `__init__.py` boilerplate structure can be seen below:
+```python3
+import sys
+from pathlib import Path
+
+__root_dir = Path(__file__).parents[4]
+sys.path.append(str(__root_dir))
+
+```
+4. Create the `__main__.py` file. An example of the `__main__.py` boilerplate structure can be seen below:
+```python3
+#!/usr/bin/env python3
+
+from pprint import pprint
+from sys import argv
+
+from src.core.utils.module import run_module
+from .module import Runner
+
+result = run_module(Runner, args=argv, arg_name="username", arg_default="johndoe")
+pprint(result)
+```
+5. Create the module itself. An example of the basic `module.py` file can be seen below:
 ```python3
 #!/usr/bin/env python3
 
@@ -60,3 +128,4 @@ class Runner(OsintRunner):
         argument = kwargs.get("my_argument", "Arguments were not provided!")
         return ScriptResponse.success(message=f"Script finished with argument {argument}")
 ```
+6. For `test_module.py` you can use any required tests (as you wish). A test case for your module is required to keep the project clean.

main.py

@@ -13,26 +13,33 @@
 from src.core.case.osint import OsintCase
 from src.core.case.recon import ReconCase
 from src.core.case.base import BaseCase
+from src.core.utils.log import Logger
+
+
+logger = Logger.get_logger(name="osint-framework")
+
+
+def run_case(case_class: type, *args, **kwargs):
+    """
+    Define and smoke run the BaseCase
+    :param case_class: original class of the case
+    :param args: some args
+    :param kwargs: some kwargs
+    :return: result of the execution
+    """
+    logger.info(f"start {case_class.__name__} case processing")
+    case = case_class()
+    case.process(*args, **kwargs)
+    return case.get_results()
 
 
 if __name__ == "__main__":
-    # Will return 2 results from "other" category scripts
-    other_case = BaseCase()
-    other_case.process(
-        username="johndoe", email="johndoe@gmail.com", fullname="John Doe",
-    )
-    other_case_results = other_case.get_results()
-
-    # Will return 1 result from "recon" category scripts
-    recon_case = ReconCase()
-    recon_case.process(url="https://facebook.com")
-    recon_case_results = recon_case.get_results()
-
-    # Will return nothing (no scripts for now, sorry!)
-    osint_case = OsintCase()
-    osint_case.process(username="any_value_here")
-    osint_case_results = osint_case.get_results()
-
-    # Print out all the results
-    for result in other_case_results, recon_case_results, osint_case_results:
-        pprint(result)
+    # fmt: off
+    base_case = run_case(case_class=BaseCase, username="johndoe", email="johndoe@gmail.com", fullname="John Doe")
+    osint_case = run_case(case_class=OsintCase, username="johndoe", email="johndoe@gmail.com", fullname="John Doe")
+    recon_case = run_case(case_class=ReconCase, url="https://facebook.com")
+
+    pprint(base_case)
+    pprint(osint_case)
+    pprint(recon_case)
+    # fmt: on

requirements.txt

@@ -1,5 +1,18 @@
+aiodns==2.0.0
+aiohttp==3.6.2
+aiosmtpd==1.2
+async-timeout==3.0.1
+atpublic==1.0
+attrs==19.3.0
 certifi==2020.6.20
+cffi==1.14.0
 chardet==3.0.4
 idna==2.10
+mmh3==2.5.1
+multidict==4.7.6
+pycares==3.1.1
+pycparser==2.20
 requests==2.24.0
 urllib3==1.25.9
+verify-email==2.4.1
+yarl==1.4.2

run_tests.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+
+python3 -m unittest discover -v