Error when Adding Finding

[tzyang]

Hi,

I have been trying to get a Kubernetes deployment of Defect Dojo working on my local machine (using Kubernetes on Docker Desktop). I am aware that a helm chart exists in the repo already, but for certain reasons I want to deploy in a manner as similar to the docker-compose as possible. So, I created a helm chart based off the docker-compose and other files in the repo. In particular the postgres-redis profile of the docker-compose. I was able to get things running and everything seems to be working correctly except for the add findings feature. I am able to create products, product types, etc. but trying to create a new finding will cause a 500 server error.

Looking in the uwsgi logs (full logs below), I am seeing a python error which states:

2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 476, in _reraise_as_library_errors 2023-10-25 17:17:44 raise ConnectionError(str(exc)) from exc 2023-10-25 17:17:44 kombu.exceptions.OperationalError: (sqlite3.OperationalError) unable to open database file 2023-10-25 17:17:44 (Background on this error at: https://sqlalche.me/e/20/e3q8)

With pgadmin I can confirm the postgres seems to be functioning normally. I can see the other created entities such as the product types in the database.

I suspect that the root cause is some incorrect setting or environment variables I missed. If anybody has any ideas on what I am doing wrong, please let me know.
 

Full Logs of uwsgi Error

2023-10-25 17:12:08 *** uWSGI is running in multiple interpreter mode ***
2023-10-25 17:12:08 spawned uWSGI worker 1 (pid: 1, cores: 2)
2023-10-25 17:12:08 spawned uWSGI worker 2 (pid: 77, cores: 2)
2023-10-25 17:16:30 [pid: 77|app: -|req: -/-] 192.168.65.3 (-) {60 vars in 1127 bytes} [Wed Oct 25 21:16:30 2023] GET / => generated 0 bytes in 32 msecs (HTTP/1.1 302) 9 headers in 346 bytes (1 switches on core 0)
2023-10-25 17:16:32 [pid: 77|app: -|req: -/-] 192.168.65.3 (-) {60 vars in 1106 bytes} [Wed Oct 25 21:16:30 2023] GET /login?next=/ => generated 15636 bytes in 1917 msecs (HTTP/1.1 200) 10 headers in 481 bytes (1 switches on core 1)
2023-10-25 17:16:38 [25/Oct/2023 21:16:38] INFO [dojo.utils:2331] login user: admin via ip: 192.168.65.3
2023-10-25 17:16:38 [pid: 1|app: -|req: -/-] 192.168.65.3 (-) {68 vars in 1318 bytes} [Wed Oct 25 21:16:36 2023] POST /login?next=/ => generated 0 bytes in 1673 msecs (HTTP/1.1 302) 13 headers in 885 bytes (1 switches on core 0)
2023-10-25 17:16:38 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {62 vars in 1387 bytes} [Wed Oct 25 21:16:38 2023] GET / => generated 0 bytes in 236 msecs (HTTP/1.1 302) 8 headers in 244 bytes (1 switches on core 1)
2023-10-25 17:16:38 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {62 vars in 1405 bytes} [Wed Oct 25 21:16:38 2023] GET /dashboard => generated 52498 bytes in 160 msecs (HTTP/1.1 200) 9 headers in 466 bytes (1 switches on core 0)
2023-10-25 17:16:42 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {60 vars in 1175 bytes} [Wed Oct 25 21:16:42 2023] GET /product/type => generated 60404 bytes in 113 msecs (HTTP/1.1 200) 8 headers in 368 bytes (1 switches on core 1)
2023-10-25 17:16:45 [pid: 77|app: -|req: -/-] 192.168.65.3 (admin) {60 vars in 1182 bytes} [Wed Oct 25 21:16:45 2023] GET /product/type/1 => generated 56032 bytes in 83 msecs (HTTP/1.1 200) 8 headers in 368 bytes (1 switches on core 0)
2023-10-25 17:16:55 [pid: 77|app: -|req: -/-] 192.168.65.3 (admin) {58 vars in 1031 bytes} [Wed Oct 25 21:16:55 2023] GET /alerts/count => generated 12 bytes in 22 msecs (HTTP/1.1 200) 7 headers in 212 bytes (1 switches on core 1)
2023-10-25 17:17:05 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {58 vars in 1031 bytes} [Wed Oct 25 21:17:05 2023] GET /alerts/count => generated 12 bytes in 24 msecs (HTTP/1.1 200) 7 headers in 212 bytes (1 switches on core 0)
2023-10-25 17:17:06 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {58 vars in 1029 bytes} [Wed Oct 25 21:17:06 2023] GET /alerts/count => generated 12 bytes in 51 msecs (HTTP/1.1 200) 7 headers in 212 bytes (1 switches on core 1)
2023-10-25 17:17:11 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {60 vars in 1206 bytes} [Wed Oct 25 21:17:11 2023] GET /product/type/1/add_product => generated 62505 bytes in 81 msecs (HTTP/1.1 200) 9 headers in 514 bytes (1 switches on core 0)
2023-10-25 17:17:19 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {68 vars in 1375 bytes} [Wed Oct 25 21:17:19 2023] POST /product/add => generated 0 bytes in 111 msecs (HTTP/1.1 302) 9 headers in 446 bytes (1 switches on core 1)
2023-10-25 17:17:19 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {62 vars in 1374 bytes} [Wed Oct 25 21:17:19 2023] GET /product/1 => generated 89898 bytes in 210 msecs (HTTP/1.1 200) 9 headers in 467 bytes (1 switches on core 0)
2023-10-25 17:17:24 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {60 vars in 1199 bytes} [Wed Oct 25 21:17:24 2023] GET /product/1/ad_hoc_finding => generated 102989 bytes in 176 msecs (HTTP/1.1 200) 8 headers in 370 bytes (1 switches on core 1)
2023-10-25 17:17:34 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {58 vars in 1041 bytes} [Wed Oct 25 21:17:34 2023] GET /alerts/count => generated 12 bytes in 20 msecs (HTTP/1.1 200) 7 headers in 212 bytes (1 switches on core 0)
2023-10-25 17:17:44 [25/Oct/2023 21:17:44] INFO [django.request:241] OK: /product/1/ad_hoc_finding
2023-10-25 17:17:44 Traceback (most recent call last):
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
2023-10-25 17:17:44 self._dbapi_connection = engine.raw_connection()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
2023-10-25 17:17:44 return self.pool.connect()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 452, in connect
2023-10-25 17:17:44 return _ConnectionFairy._checkout(self)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 1267, in _checkout
2023-10-25 17:17:44 fairy = _ConnectionRecord.checkout(pool)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
2023-10-25 17:17:44 rec = pool._do_get()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 170, in _do_get
2023-10-25 17:17:44 self._dec_overflow()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-10-25 17:17:44 raise exc_value.with_traceback(exc_tb)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
2023-10-25 17:17:44 return self._create_connection()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
2023-10-25 17:17:44 return _ConnectionRecord(self)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
2023-10-25 17:17:44 self.__connect()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 903, in __connect
2023-10-25 17:17:44 pool.logger.debug("Error on connect(): %s", e)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-10-25 17:17:44 raise exc_value.with_traceback(exc_tb)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
2023-10-25 17:17:44 self.dbapi_connection = connection = pool._invoke_creator(self)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 637, in connect
2023-10-25 17:17:44 return dialect.connect(*cargs, **cparams)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 615, in connect
2023-10-25 17:17:44 return self.loaded_dbapi.connect(*cargs, **cparams)
2023-10-25 17:17:44 sqlite3.OperationalError: unable to open database file
2023-10-25 17:17:44
2023-10-25 17:17:44 The above exception was the direct cause of the following exception:
2023-10-25 17:17:44
2023-10-25 17:17:44 Traceback (most recent call last):
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 472, in _reraise_as_library_errors
2023-10-25 17:17:44 yield
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 563, in _ensured
2023-10-25 17:17:44 return fun(*args, **kwargs)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 202, in _publish
2023-10-25 17:17:44 [maybe_declare(entity) for entity in declare]
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 202, in
2023-10-25 17:17:44 [maybe_declare(entity) for entity in declare]
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 107, in maybe_declare
2023-10-25 17:17:44 return maybe_declare(entity, self.channel, retry, **retry_policy)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/common.py", line 113, in maybe_declare
2023-10-25 17:17:44 return _maybe_declare(entity, channel)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/common.py", line 153, in _maybe_declare
2023-10-25 17:17:44 entity.declare(channel=channel)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/entity.py", line 617, in declare
2023-10-25 17:17:44 self._create_queue(nowait=nowait, channel=channel)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/entity.py", line 626, in _create_queue
2023-10-25 17:17:44 self.queue_declare(nowait=nowait, passive=False, channel=channel)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/entity.py", line 655, in queue_declare
2023-10-25 17:17:44 ret = channel.queue_declare(
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 537, in queue_declare
2023-10-25 17:17:44 self._new_queue(queue, **kwargs)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/transport/sqlalchemy/__init__.py", line 159, in _new_queue
2023-10-25 17:17:44 self._get_or_create(queue)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/transport/sqlalchemy/__init__.py", line 138, in _get_or_create
2023-10-25 17:17:44 obj = self.session.query(self.queue_cls) \
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/transport/sqlalchemy/__init__.py", line 133, in session
2023-10-25 17:17:44 _, Session = self._open()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/transport/sqlalchemy/__init__.py", line 125, in _open
2023-10-25 17:17:44 metadata.create_all(engine)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5792, in create_all
2023-10-25 17:17:44 bind._run_ddl_visitor(
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3238, in _run_ddl_visitor
2023-10-25 17:17:44 with self.begin() as conn:
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
2023-10-25 17:17:44 return next(self.gen)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3228, in begin
2023-10-25 17:17:44 with self.connect() as conn:
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3264, in connect
2023-10-25 17:17:44 return self._connection_cls(self)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
2023-10-25 17:17:44 Connection._handle_dbapi_exception_noconnection(
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2426, in _handle_dbapi_exception_noconnection
2023-10-25 17:17:44 raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
2023-10-25 17:17:44 self._dbapi_connection = engine.raw_connection()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
2023-10-25 17:17:44 return self.pool.connect()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 452, in connect
2023-10-25 17:17:44 return _ConnectionFairy._checkout(self)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 1267, in _checkout
2023-10-25 17:17:44 fairy = _ConnectionRecord.checkout(pool)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
2023-10-25 17:17:44 rec = pool._do_get()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 170, in _do_get
2023-10-25 17:17:44 self._dec_overflow()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-10-25 17:17:44 raise exc_value.with_traceback(exc_tb)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
2023-10-25 17:17:44 return self._create_connection()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
2023-10-25 17:17:44 return _ConnectionRecord(self)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
2023-10-25 17:17:44 self.__connect()
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 903, in __connect
2023-10-25 17:17:44 pool.logger.debug("Error on connect(): %s", e)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
2023-10-25 17:17:44 raise exc_value.with_traceback(exc_tb)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
2023-10-25 17:17:44 self.dbapi_connection = connection = pool._invoke_creator(self)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 637, in connect
2023-10-25 17:17:44 return dialect.connect(*cargs, **cparams)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 615, in connect
2023-10-25 17:17:44 return self.loaded_dbapi.connect(*cargs, **cparams)
2023-10-25 17:17:44 sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
2023-10-25 17:17:44 (Background on this error at: https://sqlalche.me/e/20/e3q8)
2023-10-25 17:17:44
2023-10-25 17:17:44 The above exception was the direct cause of the following exception:
2023-10-25 17:17:44
2023-10-25 17:17:44 Traceback (most recent call last):
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 56, in inner
2023-10-25 17:17:44 response = get_response(request)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
2023-10-25 17:17:44 response = wrapped_callback(request, *callback_args, **callback_kwargs)
2023-10-25 17:17:44 File "/app/./dojo/authorization/authorization_decorators.py", line 35, in _wrapped
2023-10-25 17:17:44 return func(request, *args, **kwargs)
2023-10-25 17:17:44 File "/app/./dojo/product/views.py", line 1262, in ad_hoc_finding
2023-10-25 17:17:44 new_finding.save()
2023-10-25 17:17:44 File "/app/./dojo/models.py", line 2882, in save
2023-10-25 17:17:44 finding_helper.post_process_finding_save(self, dedupe_option=dedupe_option, rules_option=rules_option, product_grading_option=product_grading_option,
2023-10-25 17:17:44 File "/app/./dojo/decorators.py", line 75, in __wrapper__
2023-10-25 17:17:44 return func(*args, **kwargs)
2023-10-25 17:17:44 File "/app/./dojo/decorators.py", line 46, in __wrapper__
2023-10-25 17:17:44 return func.apply_async(args=args, kwargs=kwargs, countdown=countdown)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/celery/app/task.py", line 594, in apply_async
2023-10-25 17:17:44 return app.send_task(
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/celery/app/base.py", line 798, in send_task
2023-10-25 17:17:44 amqp.send_task_message(P, name, message, **options)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/celery/app/amqp.py", line 517, in send_task_message
2023-10-25 17:17:44 ret = producer.publish(
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 186, in publish
2023-10-25 17:17:44 return _publish(
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 601, in _ensured
2023-10-25 17:17:44 errback and errback(exc, 0)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
2023-10-25 17:17:44 self.gen.throw(typ, value, traceback)
2023-10-25 17:17:44 File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 476, in _reraise_as_library_errors
2023-10-25 17:17:44 raise ConnectionError(str(exc)) from exc
2023-10-25 17:17:44 kombu.exceptions.OperationalError: (sqlite3.OperationalError) unable to open database file
2023-10-25 17:17:44 (Background on this error at: https://sqlalche.me/e/20/e3q8)
2023-10-25 17:17:44 [pid: 1|app: -|req: -/-] 192.168.65.3 (admin) {68 vars in 1399 bytes} [Wed Oct 25 21:17:43 2023] POST /product/1/ad_hoc_finding => generated 44469 bytes in 515 msecs (HTTP/1.1 200) 7 headers in 223 bytes (1 switches on core 1)
2023-10-25 17:17:54 [pid: 77|app: -|req: -/-] 192.168.65.3 (admin) {58 vars in 1041 bytes} [Wed Oct 25 21:17:54 2023] GET /alerts/count => generated 12 bytes in 35 msecs (HTTP/1.1 200) 7 headers in 212 bytes (1 switches on core 0)

[SeanPesce]

Did you ever fix this? I ran into the same problem.

EDIT: I believe the root cause is the same as #1934

I was able to work around this on DefectDojo v2.28.3 by setting the DD_CELERY_BROKER_PATH environment variable to /dojo/settings/dojo.celerydb.sqlite; I don't think it's a complete/proper fix but I can at least view/edit the settings page and submit findings now.