diff --git a/README.md b/README.md index 52abd62..61000fa 100644 --- a/README.md +++ b/README.md @@ -4,105 +4,112 @@ ## Introduction -ServiceX DID finders take a dataset name and turn them into files to be transformed. They interact with ServiceX via the Rabbit MQ message broker. As such, the integration into ServiceX and the API must be the same for all DID finders. This library abstracts away some of that interaction so that the code that interacts with ServiceX can be separated from the code that translates a dataset identifier into a list of files. +ServiceX DID finders take a dataset name and turn them into files to be transformed. They are +implemented as a Celery application with a task called `do_lookup`. Developers of a specific +DID Finder implementation need to write a generator function which yields a dictionary for each +file in the dataset. -The DID Finder author need write only a line of initialization code and then a small routine that translates a DID into a list of files. +The Task interacts with ServiceX through the App's REST endpoint to add files to the dataset and +a separate REST endpoint to signal that the dataset is complete. -## Usage +The app caches DID lookups. The `dataset_id` is the primary key for the cache table. -A very simple [demo](https://github.com/ssl-hep/ServiceX_DID_Finder_Demo) has been created to show how to make a basic DID finder for ServiceX. You can use that as a starting point if authoring one. +Invocations of the `do_lookup` task accepts the following arguments: +* `did`: The dataset identifier to look up +* `dataset_id`: The ID of the dataset in the database +* `endpoint`: The ServiceX endpoint to send the results to +* `user_did_finder`: The user callback that is a generator function that yields file information dictionaries. -Create an async callback method that `yield`s file info dictionaries. For example: +## Creating a DID Finder +You start with a new Python project. You will need to add this library as a dependency to the project +by adding the following to your `pyproject.tom` file: +``` +servicex-did-finder-lib = "^3.0" +``` + +Create a celery app that will run your DID finder. This app will be responsible for starting the +Celery worker and registering your DID finder function as a task. Here is an example of how to do +this. Celery prefers that the app is in a file called `celery.py` in a module in your project. Here +is an example of how to do this: + +## celery.py: ```python - async def my_callback(did_name: str, info: Dict[str, Any]): - for i in range(0, 10): - yield { - 'paths': [f"root://atlas-experiment.cern.ch/dataset1/file{i}.root"], - 'adler32': b183712731, - 'file_size': 0, - 'file_events': 0, - } + +from servicex_did_finder_lib import DIDFinderApp +rucio_adaptor = RucioAdaptor() +app = DIDFinderApp('rucio', did_finder_args={"rucio_adapter": rucio_adaptor}) ``` +Attach the DID finder to the app by using the `did_lookup_task` decorator. This decorator will +register the function as a Celery task. Here is an example of how to do this: + +```python +@app.did_lookup_task(name="did_finder_rucio.lookup_dataset") +def lookup_dataset(self, did: str, dataset_id: int, endpoint: str) -> None: + self.do_lookup(did=did, dataset_id=dataset_id, + endpoint=endpoint, user_did_finder=find_files) +``` + +You will need to implement the `find_files` function. This function is a generator that yields +file information dictionaries. + The arguments to the method are straight forward: * `did_name`: the name of the DID that you should look up. It has the schema stripped off (e.g. if the user sent ServiceX `rucio://dataset_name_in_rucio`, then `did_name` will be `dataset_name_in_rucio`) -* `info` contains a dict of various info about the request that asked for this DID. +* `info` contains a dict of various info about the database ID for this dataset. +* `did_finder_args` contains the arguments that were passed to the DID finder at startup. This is a way to pass command line arguments to your file finder -Yield the results as you find them - ServiceX will actually start processing the files before your DID lookup is finished if you do this. The fields you need to pass back to the library are as follows: +Yield the results as you find them. The fields you need to pass back to the library are as follows: * `paths`: An ordered list of URIs that a transformer in ServiceX can access to get at the file. Often these are either `root://` or `http://` schema URI's. When accessing the file, URIs will be tried in ordered listed. * `adler32`: A CRC number for the file. This CRC is calculated in a special way by rucio and is not used. Leave as 0 if you do not know it. * `file_size`: Number of bytes of the file. Used to calculate statistics. Leave as zero if you do not know it (or it is expensive to look up). * `file_events`: Number of events in the file. Used to calculate statistics. Leave as zero if you do not know it (or it is expensive to look up). -Once the callback is working privately, it is time to build a container. ServiceX will start the container and pass, one way or the other, a few arguments to it. Several arguments are required for the DID finder to work (like `rabbitUrl`). The library will automatically parse these during initialization. - -To initialize the library and start listening for and processing messages from RabbitMQ, you must initialize the library. In all cases, the call to `start_did_finder` will not return. - -If you do not have to process any command line arguments to configure the service, then the following is good enough: +Here's a simple example of a did handler generator: ```python -start_did_finder('my_finder', my_callback) +def find_files(did_name: str, + info: Dict[str, Any], + did_finder_args: Dict[str, Any] + ) -> Generator[Dict[str, Any], None]: + __log.info('DID Lookup request received.', extra={ + 'requestId': info['request-id'], 'dataset': did_name}) + + urls = xrd.glob(cache_prefix + did_name) + if len(urls) == 0: + raise RuntimeError(f"No files found matching {did_name} for request " + f"{info['request-id']} - are you sure it is correct?") + + for url in urls: + yield { + 'paths': [url], + 'adler32': 0, # No clue + 'file_size': 0, # We could look up the size but that would be slow + 'file_events': 0, # And this we do not know + } ``` -The first argument is the name of the schema. The user will use `my_finder://dataset_name` to access this DID lookup (and `my_callback` is called with `dataset_name` as `did_name`). Please make sure everything is lower case: schema in URI's are not case sensitive. -The second argument is the call back. +## Extra Command Line Arguments +Sometimes you need to pass additional information to your DID Finder from the command line. You do +this by creating your own `ArgParser` +```python +import argparse +# Parse command-line arguments +parser = argparse.ArgumentParser(description='DIDFinderApp') +parser.add_argument('--custom-arg', help='Custom argument for DIDFinderApp') +args, unknown = parser.parse_known_args() -If you do need to configure your DID finder from command line arguments, then use the python `argparse` library, being sure to hook in the did finder library as follows (this code is used to start the rucio finder): +# Create the app instance +app = DIDFinderApp('myApp', did_finder_args={"custom-arg": args.custom_arg}) -```python - # Parse the command line arguments - parser = argparse.ArgumentParser() - parser.add_argument('--site', dest='site', action='store', - default=None, - help='XCache Site)') - parser.add_argument('--prefix', dest='prefix', action='store', - default='', - help='Prefix to add to Xrootd URLs') - parser.add_argument('--threads', dest='threads', action='store', - default=10, type=int, help="Number of threads to spawn") - add_did_finder_cnd_arguments(parser) - - args = parser.parse_args() - - site = args.site - prefix = args.prefix - threads = args.threads - logger.info("ServiceX DID Finder starting up: " - f"Threads: {threads} Site: {site} Prefix: {prefix}") - - # Initialize the finder - did_client = DIDClient() - replica_client = ReplicaClient() - rucio_adapter = RucioAdapter(did_client, replica_client) - - # Run the DID Finder - try: - logger.info('Starting rucio DID finder') - - async def callback(did_name, info): - async for f in find_files(rucio_adapter, site, prefix, threads, - did_name, info): - yield f - - start_did_finder('rucio', - callback, - parsed_args=args) - - finally: - logger.info('Done running rucio DID finder') ``` -In particular note: +These parsed args will be passed to your `find_files` function as a dictionary in +the `did_finder_args` parameter. -1. The call to `add_did_finder_cnd_arguments` to setup the arguments required by the finder library. -2. Parsing of the arguments using the usual `parse_args` method -3. Passing the parsed arguments to `start_did_finder`. - -Another pattern in the above code that one might find useful - a thread-safe way of passing global arguments into the callback. Given Python's Global Interpreter Lock, this is probably not necessary. ### Proper Logging @@ -124,7 +131,7 @@ In the end, all DID finders for ServiceX will run under Kubernetes. ServiceX com } ``` -The `start_did_finder` will configure the python root logger properly. +The `DIDFinderApp` will configure the python root logger properly. ## URI Format @@ -135,3 +142,11 @@ All the incoming DID's are expected to be URI's without the schema. As such, the * `get` - If the value is `all` (the default) then all files in the dataset must be returned. If the value is `available`, then only files that are accessible need be returned. As am example, if the following URI is given to ServiceX, "rucio://dataset_name?files=20&get=available", then the first 20 available files of the dataset will be processed by the rest of servicex. + +## Stressful DID Finder +As an example, there is in this repo a simple DID finder that can be used to test the system. It is called `stressful_did_finder.py`. It will return a large number of files, and will take a long time to run. It is useful for testing the system under load. +I'm not quite sure how to use it yet, but I'm sure it will be useful. + +It accepts the following arguments: +* `--num-files` - The number of files to return as part of each request. Default is 10. +* `--file-path` - The DID Finder returns the same file over and over. This is the file to return in the response diff --git a/poetry.lock b/poetry.lock index 68208c5..0b32e12 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1,6 +1,93 @@ +[[package]] +name = "amqp" +version = "5.2.0" +description = "Low-level AMQP client for Python (fork of amqplib)." +category = "main" +optional = false +python-versions = ">=3.6" + +[package.dependencies] +vine = ">=5.0.0,<6.0.0" + +[[package]] +name = "backports-zoneinfo" +version = "0.2.1" +description = "Backport of the standard library zoneinfo module" +category = "main" +optional = false +python-versions = ">=3.6" + +[package.dependencies] +tzdata = {version = "*", optional = true, markers = "extra == \"tzdata\""} + +[package.extras] +tzdata = ["tzdata"] + +[[package]] +name = "billiard" +version = "4.2.0" +description = "Python multiprocessing fork with improvements and bugfixes" +category = "main" +optional = false +python-versions = ">=3.7" + +[[package]] +name = "celery" +version = "5.4.0" +description = "Distributed Task Queue." +category = "main" +optional = false +python-versions = ">=3.8" + +[package.dependencies] +"backports.zoneinfo" = {version = ">=0.2.1", markers = "python_version < \"3.9\""} +billiard = ">=4.2.0,<5.0" +click = ">=8.1.2,<9.0" +click-didyoumean = ">=0.3.0" +click-plugins = ">=1.1.1" +click-repl = ">=0.2.0" +kombu = ">=5.3.4,<6.0" +python-dateutil = ">=2.8.2" +tzdata = ">=2022.7" +vine = ">=5.1.0,<6.0" + +[package.extras] +arangodb = ["pyArango (>=2.0.2)"] +auth = ["cryptography (==42.0.5)"] +azureblockblob = ["azure-storage-blob (>=12.15.0)"] +brotli = ["brotli (>=1.0.0)", "brotlipy (>=0.7.0)"] +cassandra = ["cassandra-driver (>=3.25.0,<4)"] +consul = ["python-consul2 (==0.1.5)"] +cosmosdbsql = ["pydocumentdb (==2.3.5)"] +couchbase = ["couchbase (>=3.0.0)"] +couchdb = ["pycouchdb (==1.14.2)"] +django = ["Django (>=2.2.28)"] +dynamodb = ["boto3 (>=1.26.143)"] +elasticsearch = ["elastic-transport (<=8.13.0)", "elasticsearch (<=8.13.0)"] +eventlet = ["eventlet (>=0.32.0)"] +gcs = ["google-cloud-storage (>=2.10.0)"] +gevent = ["gevent (>=1.5.0)"] +librabbitmq = ["librabbitmq (>=2.0.0)"] +memcache = ["pylibmc (==1.6.3)"] +mongodb = ["pymongo[srv] (>=4.0.2)"] +msgpack = ["msgpack (==1.0.8)"] +pymemcache = ["python-memcached (>=1.61)"] +pyro = ["pyro4 (==4.82)"] +pytest = ["pytest-celery[all] (>=1.0.0)"] +redis = ["redis (>=4.5.2,!=4.5.5,<6.0.0)"] +s3 = ["boto3 (>=1.26.143)"] +slmq = ["softlayer-messaging (>=1.0.3)"] +solar = ["ephem (==4.1.5)"] +sqlalchemy = ["sqlalchemy (>=1.4.48,<2.1)"] +sqs = ["boto3 (>=1.26.143)", "kombu[sqs] (>=5.3.4)", "pycurl (>=7.43.0.5)", "urllib3 (>=1.26.16)"] +tblib = ["tblib (>=1.3.0)", "tblib (>=1.5.0)"] +yaml = ["PyYAML (>=3.10)"] +zookeeper = ["kazoo (>=1.3.1)"] +zstd = ["zstandard (==0.22.0)"] + [[package]] name = "certifi" -version = "2023.11.17" +version = "2024.6.2" description = "Python package for providing Mozilla's CA Bundle." category = "main" optional = false @@ -14,17 +101,68 @@ category = "main" optional = false python-versions = ">=3.7.0" +[[package]] +name = "click" +version = "8.1.7" +description = "Composable command line interface toolkit" +category = "main" +optional = false +python-versions = ">=3.7" + +[package.dependencies] +colorama = {version = "*", markers = "platform_system == \"Windows\""} + +[[package]] +name = "click-didyoumean" +version = "0.3.1" +description = "Enables git-like *did-you-mean* feature in click" +category = "main" +optional = false +python-versions = ">=3.6.2" + +[package.dependencies] +click = ">=7" + +[[package]] +name = "click-plugins" +version = "1.1.1" +description = "An extension module for click to enable registering CLI commands via setuptools entry-points." +category = "main" +optional = false +python-versions = "*" + +[package.dependencies] +click = ">=4.0" + +[package.extras] +dev = ["coveralls", "pytest (>=3.6)", "pytest-cov", "wheel"] + +[[package]] +name = "click-repl" +version = "0.3.0" +description = "REPL plugin for Click" +category = "main" +optional = false +python-versions = ">=3.6" + +[package.dependencies] +click = ">=7.0" +prompt-toolkit = ">=3.0.36" + +[package.extras] +testing = ["pytest (>=7.2.1)", "pytest-cov (>=4.0.0)", "tox (>=4.4.3)"] + [[package]] name = "colorama" version = "0.4.6" description = "Cross-platform colored terminal text." -category = "dev" +category = "main" optional = false python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" [[package]] name = "coverage" -version = "7.4.0" +version = "7.5.4" description = "Code coverage measurement for Python" category = "dev" optional = false @@ -35,7 +173,7 @@ toml = ["tomli"] [[package]] name = "exceptiongroup" -version = "1.2.0" +version = "1.2.1" description = "Backport of PEP 654 (exception groups)" category = "dev" optional = false @@ -46,20 +184,20 @@ test = ["pytest (>=6)"] [[package]] name = "flake8" -version = "3.9.2" +version = "7.1.0" description = "the modular source code checker: pep8 pyflakes and co" category = "dev" optional = false -python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7" +python-versions = ">=3.8.1" [package.dependencies] -mccabe = ">=0.6.0,<0.7.0" -pycodestyle = ">=2.7.0,<2.8.0" -pyflakes = ">=2.3.0,<2.4.0" +mccabe = ">=0.7.0,<0.8.0" +pycodestyle = ">=2.12.0,<2.13.0" +pyflakes = ">=3.2.0,<3.3.0" [[package]] name = "idna" -version = "3.6" +version = "3.7" description = "Internationalized Domain Names in Applications (IDNA)" category = "main" optional = false @@ -73,6 +211,37 @@ category = "dev" optional = false python-versions = ">=3.7" +[[package]] +name = "kombu" +version = "5.3.7" +description = "Messaging library for Python." +category = "main" +optional = false +python-versions = ">=3.8" + +[package.dependencies] +amqp = ">=5.1.1,<6.0.0" +"backports.zoneinfo" = {version = ">=0.2.1", extras = ["tzdata"], markers = "python_version < \"3.9\""} +typing-extensions = {version = "*", markers = "python_version < \"3.10\""} +vine = "*" + +[package.extras] +azureservicebus = ["azure-servicebus (>=7.10.0)"] +azurestoragequeues = ["azure-identity (>=1.12.0)", "azure-storage-queue (>=12.6.0)"] +confluentkafka = ["confluent-kafka (>=2.2.0)"] +consul = ["python-consul2"] +librabbitmq = ["librabbitmq (>=2.0.0)"] +mongodb = ["pymongo (>=4.1.1)"] +msgpack = ["msgpack"] +pyro = ["pyro4"] +qpid = ["qpid-python (>=0.26)", "qpid-tools (>=0.26)"] +redis = ["redis (>=4.5.2,!=4.5.5,!=5.0.2)"] +slmq = ["softlayer-messaging (>=1.0.3)"] +sqlalchemy = ["sqlalchemy (>=1.4.48,<2.1)"] +sqs = ["boto3 (>=1.26.143)", "pycurl (>=7.43.0.5)", "urllib3 (>=1.26.16)"] +yaml = ["PyYAML (>=3.10)"] +zookeeper = ["kazoo (>=2.8.0)"] + [[package]] name = "make-it-sync" version = "1.0.0" @@ -86,35 +255,23 @@ test = ["autopep8", "codecov", "coverage", "flake8", "pytest (>=3.9)", "pytest-a [[package]] name = "mccabe" -version = "0.6.1" +version = "0.7.0" description = "McCabe checker, plugin for flake8" category = "dev" optional = false -python-versions = "*" +python-versions = ">=3.6" [[package]] name = "packaging" -version = "23.2" +version = "24.1" description = "Core utilities for Python packages" category = "dev" optional = false -python-versions = ">=3.7" - -[[package]] -name = "pika" -version = "1.1.0" -description = "Pika Python AMQP Client Library" -category = "main" -optional = false -python-versions = "*" - -[package.extras] -tornado = ["tornado"] -twisted = ["twisted"] +python-versions = ">=3.8" [[package]] name = "pluggy" -version = "1.3.0" +version = "1.5.0" description = "plugin and hook calling mechanisms for python" category = "dev" optional = false @@ -124,40 +281,51 @@ python-versions = ">=3.8" dev = ["pre-commit", "tox"] testing = ["pytest", "pytest-benchmark"] +[[package]] +name = "prompt-toolkit" +version = "3.0.47" +description = "Library for building powerful interactive command lines in Python" +category = "main" +optional = false +python-versions = ">=3.7.0" + +[package.dependencies] +wcwidth = "*" + [[package]] name = "pycodestyle" -version = "2.7.0" +version = "2.12.0" description = "Python style guide checker" category = "dev" optional = false -python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" +python-versions = ">=3.8" [[package]] name = "pyflakes" -version = "2.3.1" +version = "3.2.0" description = "passive checker of Python programs" category = "dev" optional = false -python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" +python-versions = ">=3.8" [[package]] name = "pytest" -version = "7.4.4" +version = "8.2.2" description = "pytest: simple powerful testing with Python" category = "dev" optional = false -python-versions = ">=3.7" +python-versions = ">=3.8" [package.dependencies] colorama = {version = "*", markers = "sys_platform == \"win32\""} exceptiongroup = {version = ">=1.0.0rc8", markers = "python_version < \"3.11\""} iniconfig = "*" packaging = "*" -pluggy = ">=0.12,<2.0" -tomli = {version = ">=1.0.0", markers = "python_version < \"3.11\""} +pluggy = ">=1.5,<2.0" +tomli = {version = ">=1", markers = "python_version < \"3.11\""} [package.extras] -testing = ["argcomplete", "attrs (>=19.2.0)", "hypothesis (>=3.56)", "mock", "nose", "pygments (>=2.7.2)", "requests", "setuptools", "xmlschema"] +dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "pygments (>=2.7.2)", "requests", "setuptools", "xmlschema"] [[package]] name = "pytest-asyncio" @@ -175,25 +343,36 @@ testing = ["coverage", "hypothesis (>=5.7.1)"] [[package]] name = "pytest-mock" -version = "3.12.0" +version = "3.14.0" description = "Thin-wrapper around the mock package for easier use with pytest" category = "dev" optional = false python-versions = ">=3.8" [package.dependencies] -pytest = ">=5.0" +pytest = ">=6.2.5" [package.extras] dev = ["pre-commit", "pytest-asyncio", "tox"] +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +description = "Extensions to the standard Python datetime module" +category = "main" +optional = false +python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7" + +[package.dependencies] +six = ">=1.5" + [[package]] name = "requests" -version = "2.31.0" +version = "2.32.3" description = "Python HTTP for Humans." category = "main" optional = false -python-versions = ">=3.7" +python-versions = ">=3.8" [package.dependencies] certifi = ">=2017.4.17" @@ -225,7 +404,7 @@ tests = ["coverage (>=3.7.1,<6.0.0)", "flake8", "mypy", "pytest (>=4.6)", "pytes name = "six" version = "1.16.0" description = "Python 2 and 3 compatibility utilities" -category = "dev" +category = "main" optional = false python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*" @@ -237,9 +416,25 @@ category = "dev" optional = false python-versions = ">=3.7" +[[package]] +name = "typing-extensions" +version = "4.12.2" +description = "Backported and Experimental Type Hints for Python 3.8+" +category = "main" +optional = false +python-versions = ">=3.8" + +[[package]] +name = "tzdata" +version = "2024.1" +description = "Provider of IANA time zone data" +category = "main" +optional = false +python-versions = ">=2" + [[package]] name = "urllib3" -version = "2.1.0" +version = "2.2.2" description = "HTTP library with thread-safe connection pooling, file post, and more." category = "main" optional = false @@ -247,18 +442,65 @@ python-versions = ">=3.8" [package.extras] brotli = ["brotli (>=1.0.9)", "brotlicffi (>=0.8.0)"] +h2 = ["h2 (>=4,<5)"] socks = ["pysocks (>=1.5.6,!=1.5.7,<2.0)"] zstd = ["zstandard (>=0.18.0)"] +[[package]] +name = "vine" +version = "5.1.0" +description = "Python promises." +category = "main" +optional = false +python-versions = ">=3.6" + +[[package]] +name = "wcwidth" +version = "0.2.13" +description = "Measures the displayed width of unicode strings in a terminal" +category = "main" +optional = false +python-versions = "*" + [metadata] lock-version = "1.1" -python-versions = "^3.8" -content-hash = "bfc5667c9a4cb00abfcaef8a4e28bd0acad26f4eb0f4f7a778c9b732632537f6" +python-versions = "^3.8.1" +content-hash = "a7f04cf1093feb7170b6c3cf9c8f479e5909f6615aace45ce388001fc93f1cd2" [metadata.files] +amqp = [ + {file = "amqp-5.2.0-py3-none-any.whl", hash = "sha256:827cb12fb0baa892aad844fd95258143bce4027fdac4fccddbc43330fd281637"}, + {file = "amqp-5.2.0.tar.gz", hash = "sha256:a1ecff425ad063ad42a486c902807d1482311481c8ad95a72694b2975e75f7fd"}, +] +backports-zoneinfo = [ + {file = "backports.zoneinfo-0.2.1-cp36-cp36m-macosx_10_14_x86_64.whl", hash = "sha256:da6013fd84a690242c310d77ddb8441a559e9cb3d3d59ebac9aca1a57b2e18bc"}, + {file = "backports.zoneinfo-0.2.1-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:89a48c0d158a3cc3f654da4c2de1ceba85263fafb861b98b59040a5086259722"}, + {file = "backports.zoneinfo-0.2.1-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:1c5742112073a563c81f786e77514969acb58649bcdf6cdf0b4ed31a348d4546"}, + {file = "backports.zoneinfo-0.2.1-cp36-cp36m-win32.whl", hash = "sha256:e8236383a20872c0cdf5a62b554b27538db7fa1bbec52429d8d106effbaeca08"}, + {file = "backports.zoneinfo-0.2.1-cp36-cp36m-win_amd64.whl", hash = "sha256:8439c030a11780786a2002261569bdf362264f605dfa4d65090b64b05c9f79a7"}, + {file = "backports.zoneinfo-0.2.1-cp37-cp37m-macosx_10_14_x86_64.whl", hash = "sha256:f04e857b59d9d1ccc39ce2da1021d196e47234873820cbeaad210724b1ee28ac"}, + {file = "backports.zoneinfo-0.2.1-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:17746bd546106fa389c51dbea67c8b7c8f0d14b5526a579ca6ccf5ed72c526cf"}, + {file = "backports.zoneinfo-0.2.1-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:5c144945a7752ca544b4b78c8c41544cdfaf9786f25fe5ffb10e838e19a27570"}, + {file = "backports.zoneinfo-0.2.1-cp37-cp37m-win32.whl", hash = "sha256:e55b384612d93be96506932a786bbcde5a2db7a9e6a4bb4bffe8b733f5b9036b"}, + {file = "backports.zoneinfo-0.2.1-cp37-cp37m-win_amd64.whl", hash = "sha256:a76b38c52400b762e48131494ba26be363491ac4f9a04c1b7e92483d169f6582"}, + {file = "backports.zoneinfo-0.2.1-cp38-cp38-macosx_10_14_x86_64.whl", hash = "sha256:8961c0f32cd0336fb8e8ead11a1f8cd99ec07145ec2931122faaac1c8f7fd987"}, + {file = "backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_i686.whl", hash = "sha256:e81b76cace8eda1fca50e345242ba977f9be6ae3945af8d46326d776b4cf78d1"}, + {file = "backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:7b0a64cda4145548fed9efc10322770f929b944ce5cee6c0dfe0c87bf4c0c8c9"}, + {file = "backports.zoneinfo-0.2.1-cp38-cp38-win32.whl", hash = "sha256:1b13e654a55cd45672cb54ed12148cd33628f672548f373963b0bff67b217328"}, + {file = "backports.zoneinfo-0.2.1-cp38-cp38-win_amd64.whl", hash = "sha256:4a0f800587060bf8880f954dbef70de6c11bbe59c673c3d818921f042f9954a6"}, + {file = "backports.zoneinfo-0.2.1.tar.gz", hash = "sha256:fadbfe37f74051d024037f223b8e001611eac868b5c5b06144ef4d8b799862f2"}, +] +billiard = [ + {file = "billiard-4.2.0-py3-none-any.whl", hash = "sha256:07aa978b308f334ff8282bd4a746e681b3513db5c9a514cbdd810cbbdc19714d"}, + {file = "billiard-4.2.0.tar.gz", hash = "sha256:9a3c3184cb275aa17a732f93f65b20c525d3d9f253722d26a82194803ade5a2c"}, +] +celery = [ + {file = "celery-5.4.0-py3-none-any.whl", hash = "sha256:369631eb580cf8c51a82721ec538684994f8277637edde2dfc0dacd73ed97f64"}, + {file = "celery-5.4.0.tar.gz", hash = "sha256:504a19140e8d3029d5acad88330c541d4c3f64c789d85f94756762d8bca7e706"}, +] certifi = [ - {file = "certifi-2023.11.17-py3-none-any.whl", hash = "sha256:e036ab49d5b79556f99cfc2d9320b34cfbe5be05c5871b51de9329f0603b0474"}, - {file = "certifi-2023.11.17.tar.gz", hash = "sha256:9b469f3a900bf28dc19b8cfbf8019bf47f7fdd1a65a1d4ffb98fc14166beb4d1"}, + {file = "certifi-2024.6.2-py3-none-any.whl", hash = "sha256:ddc6c8ce995e6987e7faf5e3f1b02b302836a0e5d98ece18392cb1a36c72ad56"}, + {file = "certifi-2024.6.2.tar.gz", hash = "sha256:3cd43f1c6fa7dedc5899d69d3ad0398fd018ad1a17fba83ddaf78aa46c747516"}, ] charset-normalizer = [ {file = "charset-normalizer-3.3.2.tar.gz", hash = "sha256:f30c3cb33b24454a82faecaf01b19c18562b1e89558fb6c56de4d9118a032fd5"}, @@ -352,123 +594,147 @@ charset-normalizer = [ {file = "charset_normalizer-3.3.2-cp39-cp39-win_amd64.whl", hash = "sha256:b01b88d45a6fcb69667cd6d2f7a9aeb4bf53760d7fc536bf679ec94fe9f3ff3d"}, {file = "charset_normalizer-3.3.2-py3-none-any.whl", hash = "sha256:3e4d1f6587322d2788836a99c69062fbb091331ec940e02d12d179c1d53e25fc"}, ] +click = [ + {file = "click-8.1.7-py3-none-any.whl", hash = "sha256:ae74fb96c20a0277a1d615f1e4d73c8414f5a98db8b799a7931d1582f3390c28"}, + {file = "click-8.1.7.tar.gz", hash = "sha256:ca9853ad459e787e2192211578cc907e7594e294c7ccc834310722b41b9ca6de"}, +] +click-didyoumean = [ + {file = "click_didyoumean-0.3.1-py3-none-any.whl", hash = "sha256:5c4bb6007cfea5f2fd6583a2fb6701a22a41eb98957e63d0fac41c10e7c3117c"}, + {file = "click_didyoumean-0.3.1.tar.gz", hash = "sha256:4f82fdff0dbe64ef8ab2279bd6aa3f6a99c3b28c05aa09cbfc07c9d7fbb5a463"}, +] +click-plugins = [ + {file = "click-plugins-1.1.1.tar.gz", hash = "sha256:46ab999744a9d831159c3411bb0c79346d94a444df9a3a3742e9ed63645f264b"}, + {file = "click_plugins-1.1.1-py2.py3-none-any.whl", hash = "sha256:5d262006d3222f5057fd81e1623d4443e41dcda5dc815c06b442aa3c02889fc8"}, +] +click-repl = [ + {file = "click-repl-0.3.0.tar.gz", hash = "sha256:17849c23dba3d667247dc4defe1757fff98694e90fe37474f3feebb69ced26a9"}, + {file = "click_repl-0.3.0-py3-none-any.whl", hash = "sha256:fb7e06deb8da8de86180a33a9da97ac316751c094c6899382da7feeeeb51b812"}, +] colorama = [ {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, ] coverage = [ - {file = "coverage-7.4.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:36b0ea8ab20d6a7564e89cb6135920bc9188fb5f1f7152e94e8300b7b189441a"}, - {file = "coverage-7.4.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:0676cd0ba581e514b7f726495ea75aba3eb20899d824636c6f59b0ed2f88c471"}, - {file = "coverage-7.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d0ca5c71a5a1765a0f8f88022c52b6b8be740e512980362f7fdbb03725a0d6b9"}, - {file = "coverage-7.4.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a7c97726520f784239f6c62506bc70e48d01ae71e9da128259d61ca5e9788516"}, - {file = "coverage-7.4.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:815ac2d0f3398a14286dc2cea223a6f338109f9ecf39a71160cd1628786bc6f5"}, - {file = "coverage-7.4.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:80b5ee39b7f0131ebec7968baa9b2309eddb35b8403d1869e08f024efd883566"}, - {file = "coverage-7.4.0-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:5b2ccb7548a0b65974860a78c9ffe1173cfb5877460e5a229238d985565574ae"}, - {file = "coverage-7.4.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:995ea5c48c4ebfd898eacb098164b3cc826ba273b3049e4a889658548e321b43"}, - {file = "coverage-7.4.0-cp310-cp310-win32.whl", hash = "sha256:79287fd95585ed36e83182794a57a46aeae0b64ca53929d1176db56aacc83451"}, - {file = "coverage-7.4.0-cp310-cp310-win_amd64.whl", hash = "sha256:5b14b4f8760006bfdb6e08667af7bc2d8d9bfdb648351915315ea17645347137"}, - {file = "coverage-7.4.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:04387a4a6ecb330c1878907ce0dc04078ea72a869263e53c72a1ba5bbdf380ca"}, - {file = "coverage-7.4.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:ea81d8f9691bb53f4fb4db603203029643caffc82bf998ab5b59ca05560f4c06"}, - {file = "coverage-7.4.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:74775198b702868ec2d058cb92720a3c5a9177296f75bd97317c787daf711505"}, - {file = "coverage-7.4.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:76f03940f9973bfaee8cfba70ac991825611b9aac047e5c80d499a44079ec0bc"}, - {file = "coverage-7.4.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:485e9f897cf4856a65a57c7f6ea3dc0d4e6c076c87311d4bc003f82cfe199d25"}, - {file = "coverage-7.4.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:6ae8c9d301207e6856865867d762a4b6fd379c714fcc0607a84b92ee63feff70"}, - {file = "coverage-7.4.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:bf477c355274a72435ceb140dc42de0dc1e1e0bf6e97195be30487d8eaaf1a09"}, - {file = "coverage-7.4.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:83c2dda2666fe32332f8e87481eed056c8b4d163fe18ecc690b02802d36a4d26"}, - {file = "coverage-7.4.0-cp311-cp311-win32.whl", hash = "sha256:697d1317e5290a313ef0d369650cfee1a114abb6021fa239ca12b4849ebbd614"}, - {file = "coverage-7.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:26776ff6c711d9d835557ee453082025d871e30b3fd6c27fcef14733f67f0590"}, - {file = "coverage-7.4.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:13eaf476ec3e883fe3e5fe3707caeb88268a06284484a3daf8250259ef1ba143"}, - {file = "coverage-7.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:846f52f46e212affb5bcf131c952fb4075b55aae6b61adc9856222df89cbe3e2"}, - {file = "coverage-7.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:26f66da8695719ccf90e794ed567a1549bb2644a706b41e9f6eae6816b398c4a"}, - {file = "coverage-7.4.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:164fdcc3246c69a6526a59b744b62e303039a81e42cfbbdc171c91a8cc2f9446"}, - {file = "coverage-7.4.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:316543f71025a6565677d84bc4df2114e9b6a615aa39fb165d697dba06a54af9"}, - {file = "coverage-7.4.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:bb1de682da0b824411e00a0d4da5a784ec6496b6850fdf8c865c1d68c0e318dd"}, - {file = "coverage-7.4.0-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:0e8d06778e8fbffccfe96331a3946237f87b1e1d359d7fbe8b06b96c95a5407a"}, - {file = "coverage-7.4.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:a56de34db7b7ff77056a37aedded01b2b98b508227d2d0979d373a9b5d353daa"}, - {file = "coverage-7.4.0-cp312-cp312-win32.whl", hash = "sha256:51456e6fa099a8d9d91497202d9563a320513fcf59f33991b0661a4a6f2ad450"}, - {file = "coverage-7.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:cd3c1e4cb2ff0083758f09be0f77402e1bdf704adb7f89108007300a6da587d0"}, - {file = "coverage-7.4.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:e9d1bf53c4c8de58d22e0e956a79a5b37f754ed1ffdbf1a260d9dcfa2d8a325e"}, - {file = "coverage-7.4.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:109f5985182b6b81fe33323ab4707011875198c41964f014579cf82cebf2bb85"}, - {file = "coverage-7.4.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3cc9d4bc55de8003663ec94c2f215d12d42ceea128da8f0f4036235a119c88ac"}, - {file = "coverage-7.4.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:cc6d65b21c219ec2072c1293c505cf36e4e913a3f936d80028993dd73c7906b1"}, - {file = "coverage-7.4.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5a10a4920def78bbfff4eff8a05c51be03e42f1c3735be42d851f199144897ba"}, - {file = "coverage-7.4.0-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:b8e99f06160602bc64da35158bb76c73522a4010f0649be44a4e167ff8555952"}, - {file = "coverage-7.4.0-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:7d360587e64d006402b7116623cebf9d48893329ef035278969fa3bbf75b697e"}, - {file = "coverage-7.4.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:29f3abe810930311c0b5d1a7140f6395369c3db1be68345638c33eec07535105"}, - {file = "coverage-7.4.0-cp38-cp38-win32.whl", hash = "sha256:5040148f4ec43644702e7b16ca864c5314ccb8ee0751ef617d49aa0e2d6bf4f2"}, - {file = "coverage-7.4.0-cp38-cp38-win_amd64.whl", hash = "sha256:9864463c1c2f9cb3b5db2cf1ff475eed2f0b4285c2aaf4d357b69959941aa555"}, - {file = "coverage-7.4.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:936d38794044b26c99d3dd004d8af0035ac535b92090f7f2bb5aa9c8e2f5cd42"}, - {file = "coverage-7.4.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:799c8f873794a08cdf216aa5d0531c6a3747793b70c53f70e98259720a6fe2d7"}, - {file = "coverage-7.4.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e7defbb9737274023e2d7af02cac77043c86ce88a907c58f42b580a97d5bcca9"}, - {file = "coverage-7.4.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a1526d265743fb49363974b7aa8d5899ff64ee07df47dd8d3e37dcc0818f09ed"}, - {file = "coverage-7.4.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bf635a52fc1ea401baf88843ae8708591aa4adff875e5c23220de43b1ccf575c"}, - {file = "coverage-7.4.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:756ded44f47f330666843b5781be126ab57bb57c22adbb07d83f6b519783b870"}, - {file = "coverage-7.4.0-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:0eb3c2f32dabe3a4aaf6441dde94f35687224dfd7eb2a7f47f3fd9428e421058"}, - {file = "coverage-7.4.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:bfd5db349d15c08311702611f3dccbef4b4e2ec148fcc636cf8739519b4a5c0f"}, - {file = "coverage-7.4.0-cp39-cp39-win32.whl", hash = "sha256:53d7d9158ee03956e0eadac38dfa1ec8068431ef8058fe6447043db1fb40d932"}, - {file = "coverage-7.4.0-cp39-cp39-win_amd64.whl", hash = "sha256:cfd2a8b6b0d8e66e944d47cdec2f47c48fef2ba2f2dff5a9a75757f64172857e"}, - {file = "coverage-7.4.0-pp38.pp39.pp310-none-any.whl", hash = "sha256:c530833afc4707fe48524a44844493f36d8727f04dcce91fb978c414a8556cc6"}, - {file = "coverage-7.4.0.tar.gz", hash = "sha256:707c0f58cb1712b8809ece32b68996ee1e609f71bd14615bd8f87a1293cb610e"}, + {file = "coverage-7.5.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:6cfb5a4f556bb51aba274588200a46e4dd6b505fb1a5f8c5ae408222eb416f99"}, + {file = "coverage-7.5.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2174e7c23e0a454ffe12267a10732c273243b4f2d50d07544a91198f05c48f47"}, + {file = "coverage-7.5.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2214ee920787d85db1b6a0bd9da5f8503ccc8fcd5814d90796c2f2493a2f4d2e"}, + {file = "coverage-7.5.4-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1137f46adb28e3813dec8c01fefadcb8c614f33576f672962e323b5128d9a68d"}, + {file = "coverage-7.5.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b385d49609f8e9efc885790a5a0e89f2e3ae042cdf12958b6034cc442de428d3"}, + {file = "coverage-7.5.4-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:b4a474f799456e0eb46d78ab07303286a84a3140e9700b9e154cfebc8f527016"}, + {file = "coverage-7.5.4-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:5cd64adedf3be66f8ccee418473c2916492d53cbafbfcff851cbec5a8454b136"}, + {file = "coverage-7.5.4-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:e564c2cf45d2f44a9da56f4e3a26b2236504a496eb4cb0ca7221cd4cc7a9aca9"}, + {file = "coverage-7.5.4-cp310-cp310-win32.whl", hash = "sha256:7076b4b3a5f6d2b5d7f1185fde25b1e54eb66e647a1dfef0e2c2bfaf9b4c88c8"}, + {file = "coverage-7.5.4-cp310-cp310-win_amd64.whl", hash = "sha256:018a12985185038a5b2bcafab04ab833a9a0f2c59995b3cec07e10074c78635f"}, + {file = "coverage-7.5.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:db14f552ac38f10758ad14dd7b983dbab424e731588d300c7db25b6f89e335b5"}, + {file = "coverage-7.5.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:3257fdd8e574805f27bb5342b77bc65578e98cbc004a92232106344053f319ba"}, + {file = "coverage-7.5.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3a6612c99081d8d6134005b1354191e103ec9705d7ba2754e848211ac8cacc6b"}, + {file = "coverage-7.5.4-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d45d3cbd94159c468b9b8c5a556e3f6b81a8d1af2a92b77320e887c3e7a5d080"}, + {file = "coverage-7.5.4-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ed550e7442f278af76d9d65af48069f1fb84c9f745ae249c1a183c1e9d1b025c"}, + {file = "coverage-7.5.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:7a892be37ca35eb5019ec85402c3371b0f7cda5ab5056023a7f13da0961e60da"}, + {file = "coverage-7.5.4-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:8192794d120167e2a64721d88dbd688584675e86e15d0569599257566dec9bf0"}, + {file = "coverage-7.5.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:820bc841faa502e727a48311948e0461132a9c8baa42f6b2b84a29ced24cc078"}, + {file = "coverage-7.5.4-cp311-cp311-win32.whl", hash = "sha256:6aae5cce399a0f065da65c7bb1e8abd5c7a3043da9dceb429ebe1b289bc07806"}, + {file = "coverage-7.5.4-cp311-cp311-win_amd64.whl", hash = "sha256:d2e344d6adc8ef81c5a233d3a57b3c7d5181f40e79e05e1c143da143ccb6377d"}, + {file = "coverage-7.5.4-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:54317c2b806354cbb2dc7ac27e2b93f97096912cc16b18289c5d4e44fc663233"}, + {file = "coverage-7.5.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:042183de01f8b6d531e10c197f7f0315a61e8d805ab29c5f7b51a01d62782747"}, + {file = "coverage-7.5.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a6bb74ed465d5fb204b2ec41d79bcd28afccf817de721e8a807d5141c3426638"}, + {file = "coverage-7.5.4-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b3d45ff86efb129c599a3b287ae2e44c1e281ae0f9a9bad0edc202179bcc3a2e"}, + {file = "coverage-7.5.4-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5013ed890dc917cef2c9f765c4c6a8ae9df983cd60dbb635df8ed9f4ebc9f555"}, + {file = "coverage-7.5.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1014fbf665fef86cdfd6cb5b7371496ce35e4d2a00cda501cf9f5b9e6fced69f"}, + {file = "coverage-7.5.4-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:3684bc2ff328f935981847082ba4fdc950d58906a40eafa93510d1b54c08a66c"}, + {file = "coverage-7.5.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:581ea96f92bf71a5ec0974001f900db495488434a6928a2ca7f01eee20c23805"}, + {file = "coverage-7.5.4-cp312-cp312-win32.whl", hash = "sha256:73ca8fbc5bc622e54627314c1a6f1dfdd8db69788f3443e752c215f29fa87a0b"}, + {file = "coverage-7.5.4-cp312-cp312-win_amd64.whl", hash = "sha256:cef4649ec906ea7ea5e9e796e68b987f83fa9a718514fe147f538cfeda76d7a7"}, + {file = "coverage-7.5.4-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:cdd31315fc20868c194130de9ee6bfd99755cc9565edff98ecc12585b90be882"}, + {file = "coverage-7.5.4-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:02ff6e898197cc1e9fa375581382b72498eb2e6d5fc0b53f03e496cfee3fac6d"}, + {file = "coverage-7.5.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d05c16cf4b4c2fc880cb12ba4c9b526e9e5d5bb1d81313d4d732a5b9fe2b9d53"}, + {file = "coverage-7.5.4-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c5986ee7ea0795a4095ac4d113cbb3448601efca7f158ec7f7087a6c705304e4"}, + {file = "coverage-7.5.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5df54843b88901fdc2f598ac06737f03d71168fd1175728054c8f5a2739ac3e4"}, + {file = "coverage-7.5.4-cp38-cp38-musllinux_1_2_aarch64.whl", hash = "sha256:ab73b35e8d109bffbda9a3e91c64e29fe26e03e49addf5b43d85fc426dde11f9"}, + {file = "coverage-7.5.4-cp38-cp38-musllinux_1_2_i686.whl", hash = "sha256:aea072a941b033813f5e4814541fc265a5c12ed9720daef11ca516aeacd3bd7f"}, + {file = "coverage-7.5.4-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:16852febd96acd953b0d55fc842ce2dac1710f26729b31c80b940b9afcd9896f"}, + {file = "coverage-7.5.4-cp38-cp38-win32.whl", hash = "sha256:8f894208794b164e6bd4bba61fc98bf6b06be4d390cf2daacfa6eca0a6d2bb4f"}, + {file = "coverage-7.5.4-cp38-cp38-win_amd64.whl", hash = "sha256:e2afe743289273209c992075a5a4913e8d007d569a406ffed0bd080ea02b0633"}, + {file = "coverage-7.5.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:b95c3a8cb0463ba9f77383d0fa8c9194cf91f64445a63fc26fb2327e1e1eb088"}, + {file = "coverage-7.5.4-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:3d7564cc09dd91b5a6001754a5b3c6ecc4aba6323baf33a12bd751036c998be4"}, + {file = "coverage-7.5.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:44da56a2589b684813f86d07597fdf8a9c6ce77f58976727329272f5a01f99f7"}, + {file = "coverage-7.5.4-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e16f3d6b491c48c5ae726308e6ab1e18ee830b4cdd6913f2d7f77354b33f91c8"}, + {file = "coverage-7.5.4-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dbc5958cb471e5a5af41b0ddaea96a37e74ed289535e8deca404811f6cb0bc3d"}, + {file = "coverage-7.5.4-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:a04e990a2a41740b02d6182b498ee9796cf60eefe40cf859b016650147908029"}, + {file = "coverage-7.5.4-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:ddbd2f9713a79e8e7242d7c51f1929611e991d855f414ca9996c20e44a895f7c"}, + {file = "coverage-7.5.4-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:b1ccf5e728ccf83acd313c89f07c22d70d6c375a9c6f339233dcf792094bcbf7"}, + {file = "coverage-7.5.4-cp39-cp39-win32.whl", hash = "sha256:56b4eafa21c6c175b3ede004ca12c653a88b6f922494b023aeb1e836df953ace"}, + {file = "coverage-7.5.4-cp39-cp39-win_amd64.whl", hash = "sha256:65e528e2e921ba8fd67d9055e6b9f9e34b21ebd6768ae1c1723f4ea6ace1234d"}, + {file = "coverage-7.5.4-pp38.pp39.pp310-none-any.whl", hash = "sha256:79b356f3dd5b26f3ad23b35c75dbdaf1f9e2450b6bcefc6d0825ea0aa3f86ca5"}, + {file = "coverage-7.5.4.tar.gz", hash = "sha256:a44963520b069e12789d0faea4e9fdb1e410cdc4aab89d94f7f55cbb7fef0353"}, ] exceptiongroup = [ - {file = "exceptiongroup-1.2.0-py3-none-any.whl", hash = "sha256:4bfd3996ac73b41e9b9628b04e079f193850720ea5945fc96a08633c66912f14"}, - {file = "exceptiongroup-1.2.0.tar.gz", hash = "sha256:91f5c769735f051a4290d52edd0858999b57e5876e9f85937691bd4c9fa3ed68"}, + {file = "exceptiongroup-1.2.1-py3-none-any.whl", hash = "sha256:5258b9ed329c5bbdd31a309f53cbfb0b155341807f6ff7606a1e801a891b29ad"}, + {file = "exceptiongroup-1.2.1.tar.gz", hash = "sha256:a4785e48b045528f5bfe627b6ad554ff32def154f42372786903b7abcfe1aa16"}, ] flake8 = [ - {file = "flake8-3.9.2-py2.py3-none-any.whl", hash = "sha256:bf8fd333346d844f616e8d47905ef3a3384edae6b4e9beb0c5101e25e3110907"}, - {file = "flake8-3.9.2.tar.gz", hash = "sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b"}, + {file = "flake8-7.1.0-py2.py3-none-any.whl", hash = "sha256:2e416edcc62471a64cea09353f4e7bdba32aeb079b6e360554c659a122b1bc6a"}, + {file = "flake8-7.1.0.tar.gz", hash = "sha256:48a07b626b55236e0fb4784ee69a465fbf59d79eec1f5b4785c3d3bc57d17aa5"}, ] idna = [ - {file = "idna-3.6-py3-none-any.whl", hash = "sha256:c05567e9c24a6b9faaa835c4821bad0590fbb9d5779e7caa6e1cc4978e7eb24f"}, - {file = "idna-3.6.tar.gz", hash = "sha256:9ecdbbd083b06798ae1e86adcbfe8ab1479cf864e4ee30fe4e46a003d12491ca"}, + {file = "idna-3.7-py3-none-any.whl", hash = "sha256:82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0"}, + {file = "idna-3.7.tar.gz", hash = "sha256:028ff3aadf0609c1fd278d8ea3089299412a7a8b9bd005dd08b9f8285bcb5cfc"}, ] iniconfig = [ {file = "iniconfig-2.0.0-py3-none-any.whl", hash = "sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374"}, {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, ] +kombu = [ + {file = "kombu-5.3.7-py3-none-any.whl", hash = "sha256:5634c511926309c7f9789f1433e9ed402616b56836ef9878f01bd59267b4c7a9"}, + {file = "kombu-5.3.7.tar.gz", hash = "sha256:011c4cd9a355c14a1de8d35d257314a1d2456d52b7140388561acac3cf1a97bf"}, +] make-it-sync = [ {file = "make-it-sync-1.0.0.tar.gz", hash = "sha256:26f5dd7d066295d14eee934326228f9d05cc42267c795e0f35a8966ba54b8b4c"}, {file = "make_it_sync-1.0.0-py3-none-any.whl", hash = "sha256:e06808ce449f765b0f86a2badc2bbe72a3f6a3d4793472b2b8c221bbdba66511"}, ] mccabe = [ - {file = "mccabe-0.6.1-py2.py3-none-any.whl", hash = "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42"}, - {file = "mccabe-0.6.1.tar.gz", hash = "sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"}, + {file = "mccabe-0.7.0-py2.py3-none-any.whl", hash = "sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e"}, + {file = "mccabe-0.7.0.tar.gz", hash = "sha256:348e0240c33b60bbdf4e523192ef919f28cb2c3d7d5c7794f74009290f236325"}, ] packaging = [ - {file = "packaging-23.2-py3-none-any.whl", hash = "sha256:8c491190033a9af7e1d931d0b5dacc2ef47509b34dd0de67ed209b5203fc88c7"}, - {file = "packaging-23.2.tar.gz", hash = "sha256:048fb0e9405036518eaaf48a55953c750c11e1a1b68e0dd1a9d62ed0c092cfc5"}, -] -pika = [ - {file = "pika-1.1.0-py2.py3-none-any.whl", hash = "sha256:4e1a1a6585a41b2341992ec32aadb7a919d649eb82904fd8e4a4e0871c8cf3af"}, - {file = "pika-1.1.0.tar.gz", hash = "sha256:9fa76ba4b65034b878b2b8de90ff8660a59d925b087c5bb88f8fdbb4b64a1dbf"}, + {file = "packaging-24.1-py3-none-any.whl", hash = "sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"}, + {file = "packaging-24.1.tar.gz", hash = "sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"}, ] pluggy = [ - {file = "pluggy-1.3.0-py3-none-any.whl", hash = "sha256:d89c696a773f8bd377d18e5ecda92b7a3793cbe66c87060a6fb58c7b6e1061f7"}, - {file = "pluggy-1.3.0.tar.gz", hash = "sha256:cf61ae8f126ac6f7c451172cf30e3e43d3ca77615509771b3a984a0730651e12"}, + {file = "pluggy-1.5.0-py3-none-any.whl", hash = "sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669"}, + {file = "pluggy-1.5.0.tar.gz", hash = "sha256:2cffa88e94fdc978c4c574f15f9e59b7f4201d439195c3715ca9e2486f1d0cf1"}, +] +prompt-toolkit = [ + {file = "prompt_toolkit-3.0.47-py3-none-any.whl", hash = "sha256:0d7bfa67001d5e39d02c224b663abc33687405033a8c422d0d675a5a13361d10"}, + {file = "prompt_toolkit-3.0.47.tar.gz", hash = "sha256:1e1b29cb58080b1e69f207c893a1a7bf16d127a5c30c9d17a25a5d77792e5360"}, ] pycodestyle = [ - {file = "pycodestyle-2.7.0-py2.py3-none-any.whl", hash = "sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068"}, - {file = "pycodestyle-2.7.0.tar.gz", hash = "sha256:c389c1d06bf7904078ca03399a4816f974a1d590090fecea0c63ec26ebaf1cef"}, + {file = "pycodestyle-2.12.0-py2.py3-none-any.whl", hash = "sha256:949a39f6b86c3e1515ba1787c2022131d165a8ad271b11370a8819aa070269e4"}, + {file = "pycodestyle-2.12.0.tar.gz", hash = "sha256:442f950141b4f43df752dd303511ffded3a04c2b6fb7f65980574f0c31e6e79c"}, ] pyflakes = [ - {file = "pyflakes-2.3.1-py2.py3-none-any.whl", hash = "sha256:7893783d01b8a89811dd72d7dfd4d84ff098e5eed95cfa8905b22bbffe52efc3"}, - {file = "pyflakes-2.3.1.tar.gz", hash = "sha256:f5bc8ecabc05bb9d291eb5203d6810b49040f6ff446a756326104746cc00c1db"}, + {file = "pyflakes-3.2.0-py2.py3-none-any.whl", hash = "sha256:84b5be138a2dfbb40689ca07e2152deb896a65c3a3e24c251c5c62489568074a"}, + {file = "pyflakes-3.2.0.tar.gz", hash = "sha256:1c61603ff154621fb2a9172037d84dca3500def8c8b630657d1701f026f8af3f"}, ] pytest = [ - {file = "pytest-7.4.4-py3-none-any.whl", hash = "sha256:b090cdf5ed60bf4c45261be03239c2c1c22df034fbffe691abe93cd80cea01d8"}, - {file = "pytest-7.4.4.tar.gz", hash = "sha256:2cf0005922c6ace4a3e2ec8b4080eb0d9753fdc93107415332f50ce9e7994280"}, + {file = "pytest-8.2.2-py3-none-any.whl", hash = "sha256:c434598117762e2bd304e526244f67bf66bbd7b5d6cf22138be51ff661980343"}, + {file = "pytest-8.2.2.tar.gz", hash = "sha256:de4bb8104e201939ccdc688b27a89a7be2079b22e2bd2b07f806b6ba71117977"}, ] pytest-asyncio = [ {file = "pytest-asyncio-0.16.0.tar.gz", hash = "sha256:7496c5977ce88c34379df64a66459fe395cd05543f0a2f837016e7144391fcfb"}, {file = "pytest_asyncio-0.16.0-py3-none-any.whl", hash = "sha256:5f2a21273c47b331ae6aa5b36087047b4899e40f03f18397c0e65fa5cca54e9b"}, ] pytest-mock = [ - {file = "pytest-mock-3.12.0.tar.gz", hash = "sha256:31a40f038c22cad32287bb43932054451ff5583ff094bca6f675df2f8bc1a6e9"}, - {file = "pytest_mock-3.12.0-py3-none-any.whl", hash = "sha256:0972719a7263072da3a21c7f4773069bcc7486027d7e8e1f81d98a47e701bc4f"}, + {file = "pytest-mock-3.14.0.tar.gz", hash = "sha256:2719255a1efeceadbc056d6bf3df3d1c5015530fb40cf347c0f9afac88410bd0"}, + {file = "pytest_mock-3.14.0-py3-none-any.whl", hash = "sha256:0b72c38033392a5f4621342fe11e9219ac11ec9d375f8e2a0c164539e0d70f6f"}, +] +python-dateutil = [ + {file = "python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3"}, + {file = "python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427"}, ] requests = [ - {file = "requests-2.31.0-py3-none-any.whl", hash = "sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f"}, - {file = "requests-2.31.0.tar.gz", hash = "sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1"}, + {file = "requests-2.32.3-py3-none-any.whl", hash = "sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6"}, + {file = "requests-2.32.3.tar.gz", hash = "sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760"}, ] responses = [ {file = "responses-0.14.0-py2.py3-none-any.whl", hash = "sha256:57bab4e9d4d65f31ea5caf9de62095032c4d81f591a8fac2f5858f7777b8567b"}, @@ -482,7 +748,23 @@ tomli = [ {file = "tomli-2.0.1-py3-none-any.whl", hash = "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc"}, {file = "tomli-2.0.1.tar.gz", hash = "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f"}, ] +typing-extensions = [ + {file = "typing_extensions-4.12.2-py3-none-any.whl", hash = "sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"}, + {file = "typing_extensions-4.12.2.tar.gz", hash = "sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"}, +] +tzdata = [ + {file = "tzdata-2024.1-py2.py3-none-any.whl", hash = "sha256:9068bc196136463f5245e51efda838afa15aaeca9903f49050dfa2679db4d252"}, + {file = "tzdata-2024.1.tar.gz", hash = "sha256:2674120f8d891909751c38abcdfd386ac0a5a1127954fbc332af6b5ceae07efd"}, +] urllib3 = [ - {file = "urllib3-2.1.0-py3-none-any.whl", hash = "sha256:55901e917a5896a349ff771be919f8bd99aff50b79fe58fec595eb37bbc56bb3"}, - {file = "urllib3-2.1.0.tar.gz", hash = "sha256:df7aa8afb0148fa78488e7899b2c59b5f4ffcfa82e6c54ccb9dd37c1d7b52d54"}, + {file = "urllib3-2.2.2-py3-none-any.whl", hash = "sha256:a448b2f64d686155468037e1ace9f2d2199776e17f0a46610480d311f73e3472"}, + {file = "urllib3-2.2.2.tar.gz", hash = "sha256:dd505485549a7a552833da5e6063639d0d177c04f23bc3864e41e5dc5f612168"}, +] +vine = [ + {file = "vine-5.1.0-py3-none-any.whl", hash = "sha256:40fdf3c48b2cfe1c38a49e9ae2da6fda88e4794c810050a728bd7413811fb1dc"}, + {file = "vine-5.1.0.tar.gz", hash = "sha256:8b62e981d35c41049211cf62a0a1242d8c1ee9bd15bb196ce38aefd6799e61e0"}, +] +wcwidth = [ + {file = "wcwidth-0.2.13-py2.py3-none-any.whl", hash = "sha256:3da69048e4540d84af32131829ff948f1e022c1c6bdb8d6102117aac784f6859"}, + {file = "wcwidth-0.2.13.tar.gz", hash = "sha256:72ea0c06399eb286d978fdedb6923a9eb47e1c486ce63e9b4e64fc18303972b5"}, ] diff --git a/pyproject.toml b/pyproject.toml index b7ef894..796fc24 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,17 +5,18 @@ description = "ServiceX DID Library Routines" authors = ["Gordon Watts "] [tool.poetry.dependencies] -python = "^3.8" -pika = "1.1.0" +python = "^3.8.1" make-it-sync = "^1.0.0" requests = "^2.25.0" +Celery= "^5.4" + [tool.poetry.group.dev] optional = true [tool.poetry.group.dev.dependencies] -pytest = "^7.4" -flake8 = "^3.9.1" +pytest = "^8.2" +flake8 = "^7.1" pytest-mock = "^3.12.0" coverage = "^7.4.0" responses = "^0.14.0" diff --git a/src/servicex_did_finder_lib/__init__.py b/src/servicex_did_finder_lib/__init__.py index be03fba..f92f66c 100644 --- a/src/servicex_did_finder_lib/__init__.py +++ b/src/servicex_did_finder_lib/__init__.py @@ -1,4 +1 @@ -__version__ = '1.0.0a1' - -from .communication import start_did_finder, \ - add_did_finder_cnd_arguments # NOQA +from .did_finder_app import DIDFinderApp # NOQA F401 diff --git a/src/servicex_did_finder_lib/accumulator.py b/src/servicex_did_finder_lib/accumulator.py new file mode 100644 index 0000000..53e328b --- /dev/null +++ b/src/servicex_did_finder_lib/accumulator.py @@ -0,0 +1,80 @@ +# Copyright (c) 2024, IRIS-HEP +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# * Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +from typing import List, Dict, Any, Union + +from servicex_did_finder_lib.did_summary import DIDSummary +from servicex_did_finder_lib.servicex_adaptor import ServiceXAdapter + + +class Accumulator: + """Track or cache files depending on the mode we are operating in""" + + def __init__(self, sx: ServiceXAdapter, sum: DIDSummary): + self.servicex = sx + self.summary = sum + self.file_cache: List[Dict[str, Any]] = [] + + def add(self, file_info: Union[Dict[str, Any], List[Dict[str, Any]]]): + """ + Track and inject the file back into the system + :param file_info: The file information to track can be a single record or a list + """ + if isinstance(file_info, dict): + self.file_cache.append(file_info) + elif isinstance(file_info, list): + self.file_cache.extend(file_info) + else: + raise ValueError("Invalid input: expected a dictionary or a list of dictionaries") + + @property + def cache_len(self) -> int: + return len(self.file_cache) + + def send_on(self, count): + """ + Send the accumulated files + :param count: The number of files to send. Set to -1 to send all + """ + + # Sort the list to insure reproducibility + files = sorted(self.file_cache, key=lambda x: x["paths"]) + if count == -1: + self.send_bulk(files) + else: + self.send_bulk(files[:count]) + + self.file_cache.clear() + + def send_bulk(self, file_list: List[Dict[str, Any]]): + """ + does a bulk put of files + :param file_list: The list of files to send + """ + for ifl in file_list: + self.summary.add_file(ifl) + self.servicex.put_file_add_bulk(file_list) diff --git a/src/servicex_did_finder_lib/did_finder_app.py b/src/servicex_did_finder_lib/did_finder_app.py new file mode 100644 index 0000000..87407e5 --- /dev/null +++ b/src/servicex_did_finder_lib/did_finder_app.py @@ -0,0 +1,166 @@ +# Copyright (c) 2022, IRIS-HEP +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# * Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +import logging +from datetime import datetime +from typing import Any, Generator, Callable, Dict, Optional + +from celery import Celery, Task + +from servicex_did_finder_lib.accumulator import Accumulator +from servicex_did_finder_lib.did_logging import initialize_root_logger +from servicex_did_finder_lib.did_summary import DIDSummary +from servicex_did_finder_lib.servicex_adaptor import ServiceXAdapter +from servicex_did_finder_lib.util_uri import parse_did_uri + +# The type for the callback method to handle DID's, supplied by the user. +# Arguments are: +# - The DID to process +# - A dictionary of information about the DID request +# - A dictionary of arguments passed to the DID finder +UserDIDHandler = Callable[ + [str, Dict[str, Any], Dict[str, Any]], + Generator[Dict[str, Any], None, None] +] + + +__logging = logging.getLogger(__name__) +__logging.addHandler(logging.NullHandler()) + + +class DIDFinderTask(Task): + """ + A Celery task that will process a single DID request. This task will + call the user supplied DID finder to get the list of files associated + with the DID, and then send that list to ServiceX for processing. + """ + def __init__(self): + super().__init__() + self.logger = logging.getLogger(__name__) + self.logger.addHandler(logging.NullHandler()) + + def do_lookup(self, did: str, dataset_id: int, endpoint: str, user_did_finder: UserDIDHandler): + """ + Perform the DID lookup for the given DID. This will call the user supplied + DID finder to get the list of files associated with the DID, and then send + that list to ServiceX for processing. + After all of the files have been sent, send a message to ServiceX indicating + that the fileset is complete + Args: + did: The DID to process + dataset_id: The dataset ID for the request + endpoint: The ServiceX endpoint to send the request to + user_did_finder: The user supplied DID finder to call to get the list of files + """ + + self.logger.info( + f"Received DID request {did}", + extra={"dataset_id": dataset_id} + ) + + servicex = ServiceXAdapter(dataset_id=dataset_id, endpoint=endpoint) + + info = { + "dataset-id": dataset_id, + } + + start_time = datetime.now() + + summary = DIDSummary(did) + did_info = parse_did_uri(did) + acc = Accumulator(servicex, summary) + + try: + for file_info in user_did_finder(did_info.did, info, self.app.did_finder_args): + acc.add(file_info) + + acc.send_on(did_info.file_count) + except Exception: + # noinspection PyTypeChecker + self.logger.error( + f"Error processing DID {did}", + extra={"dataset_id": dataset_id}, + exc_info=1 + ) + finally: + elapsed_time = int((datetime.now() - start_time).total_seconds()) + servicex.put_fileset_complete( + { + "files": summary.file_count, + "files-skipped": summary.files_skipped, + "total-events": summary.total_events, + "total-bytes": summary.total_bytes, + "elapsed-time": elapsed_time, + } + ) + + +class DIDFinderApp(Celery): + """ + The main application for a DID finder. This will setup the Celery application + and start the worker to process the DID requests. + """ + def __init__(self, did_finder_name: str, + did_finder_args: Optional[Dict[str, Any]] = None, + *args, **kwargs): + """ + Initialize the DID finder application + Args: + did_finder_name: The name of the DID finder. + did_finder_args: The parsed command line arguments and other objects you want + to make available to the tasks + """ + + self.name = did_finder_name + initialize_root_logger(self.name) + + super().__init__(f"did_finder_{self.name}", *args, + broker_connection_retry_on_startup=True, + **kwargs) + + # Cache the args in the App, so they are accessible to the tasks + self.did_finder_args = did_finder_args + + def did_lookup_task(self, name): + """ + Decorator to create a new task to handle a DID lookup request wihout + needing to know about Celery tasks. + Usage: + @app.did_lookup_task(name="did_finder_cern_opendata.lookup_dataset") + def lookup_dataset(self, did: str, dataset_id: int, endpoint: str) -> None: + self.do_lookup(did=did, dataset_id=dataset_id, + endpoint=endpoint, user_did_finder=find_files) + + Args: + name: The name of the task + """ + def decorator(func): + @self.task(base=DIDFinderTask, bind=True, name=name) + def wrapper(*args, **kwargs): + return func(*args, **kwargs) + return wrapper + return decorator diff --git a/src/servicex_did_finder_lib/servicex_adaptor.py b/src/servicex_did_finder_lib/servicex_adaptor.py index 6e8b4e7..eccc831 100644 --- a/src/servicex_did_finder_lib/servicex_adaptor.py +++ b/src/servicex_did_finder_lib/servicex_adaptor.py @@ -51,23 +51,6 @@ def _create_json(self, file_info): 'file_events': file_info['file_events'] } - def put_file_add(self, file_info): - success = False - attempts = 0 - while not success and attempts < MAX_RETRIES: - try: - mesg = self._create_json(file_info) - requests.put(f"{self.endpoint}{self.dataset_id}/files", json=mesg) - self.logger.info("adding file:", extra=file_info) - success = True - except requests.exceptions.ConnectionError: - self.logger.exception(f'Connection error to ServiceX App. Will retry ' - f'(try {attempts} out of {MAX_RETRIES}') - attempts += 1 - if not success: - self.logger.error(f'After {attempts} tries, failed to send ServiceX App a put_file ' - f'message: {str(file_info)} - Ignoring error.') - def put_file_add_bulk(self, file_list, chunk_length=300): # we first chunk up file_list as it can be very large in # case there are a lot of replicas and a lot of files. diff --git a/tests/servicex_did_finder_lib/test_communication.py b/tests/servicex_did_finder_lib/test_communication.py deleted file mode 100644 index 13f8bc9..0000000 --- a/tests/servicex_did_finder_lib/test_communication.py +++ /dev/null @@ -1,692 +0,0 @@ -import argparse -import json -from typing import Any, AsyncGenerator, Dict, Optional -from unittest.mock import MagicMock, patch - -import pika -import pytest -from servicex_did_finder_lib.communication import ( - add_did_finder_cnd_arguments, - init_rabbit_mq, - start_did_finder, - run_file_fetch_loop, -) - - -class RabbitAdaptor: - def __init__(self, channel): - self._channel = channel - pass - - def send_did_request(self, did_name: str): - # Get out the callback so we can send it. - self._channel.basic_consume.assert_called_once() - callback = self._channel.basic_consume.call_args[1]["on_message_callback"] - channel = MagicMock() - method = MagicMock() - properties = MagicMock() - body = json.dumps( - { - "did": did_name, - "dataset_id": "000-111-222-444", - "endpoint": "http://localhost:2334/", - } - ) - - callback(channel, method, properties, body) - - -@pytest.fixture() -def SXAdaptor(mocker): - "Return a ServiceXAdaptor for testing" - with patch( - "servicex_did_finder_lib.communication.ServiceXAdapter", autospec=True - ) as sx_ctor: - sx_adaptor = mocker.MagicMock() - sx_ctor.return_value = sx_adaptor - - yield sx_adaptor - - -@pytest.fixture() -def rabbitmq(mocker): - with patch( - "servicex_did_finder_lib.communication.pika.BlockingConnection", autospec=True - ) as block_connection_ctor: - - block_connection = mocker.MagicMock() - block_connection_ctor.return_value = block_connection - - channel = mocker.MagicMock() - block_connection.channel.return_value = channel - - control = RabbitAdaptor(channel) - yield control - - -@pytest.fixture() -def rabbitmq_fail_once(mocker): - with patch( - "servicex_did_finder_lib.communication.pika.BlockingConnection", autospec=True - ) as block_connection_ctor: - - block_connection = mocker.MagicMock() - block_connection_ctor.side_effect = [ - pika.exceptions.AMQPConnectionError(), # type: ignore - block_connection, - ] - - channel = mocker.MagicMock() - block_connection.channel.return_value = channel - - control = RabbitAdaptor(channel) - yield control - - -@pytest.fixture() -def init_rabbit_callback(mocker): - with patch( - "servicex_did_finder_lib.communication.init_rabbit_mq", autospec=True - ) as call_back: - yield call_back - - -@pytest.fixture() -def simple_argument_parser(mocker): - with patch( - "servicex_did_finder_lib.communication.argparse.ArgumentParser", autospec=True - ) as ctor_ArgumentParser: - parser = mocker.MagicMock(spec=argparse.ArgumentParser) - ctor_ArgumentParser.return_value = parser - - parsed_args = mocker.MagicMock() - parsed_args.rabbit_uri = "test_queue_address" - - parser.parse_args = mocker.MagicMock(return_value=parsed_args) - - yield parser - - -def test_one_file_call(rabbitmq, SXAdaptor): - "Test a working, simple, one file call" - - seen_name = None - - async def my_callback(did_name: str, info: Dict[str, Any]): - nonlocal seen_name - seen_name = did_name - yield { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - - rabbitmq.send_did_request("hi-there") - - # Make sure callback was called - assert seen_name == "hi-there" - - # Make sure the file was sent along, along with the completion - SXAdaptor.put_file_add.assert_called_once_with( - { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - ) - SXAdaptor.put_fileset_complete.assert_called_once() - - -def test_one_file_call_with_param(rabbitmq, SXAdaptor): - "Test a working, simple, one file call with parameter" - - seen_name = None - - async def my_callback(did_name: str, info: Dict[str, Any]): - nonlocal seen_name - seen_name = did_name - yield { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - - rabbitmq.send_did_request("hithere?files=10") - - # Make sure callback was called - assert seen_name == "hithere" - - # Make sure the file was sent along, along with the completion - SXAdaptor.put_file_add_bulk.assert_called_once() - SXAdaptor.put_fileset_complete.assert_called_once() - - -def test_bulk_file_call_with_param(rabbitmq, SXAdaptor): - "Test a working, simple, two file call with parameter" - - seen_name = None - - async def my_callback(did_name: str, info: Dict[str, Any]): - nonlocal seen_name - seen_name = did_name - yield [ - { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - }, - { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - }, - ] - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - - rabbitmq.send_did_request("hithere?files=10") - - # Make sure callback was called - assert seen_name == "hithere" - - # Make sure the file was sent along, along with the completion - SXAdaptor.put_file_add_bulk.assert_called_once() - SXAdaptor.put_fileset_complete.assert_called_once() - - -def test_bulk_file_call_with_file_limit(rabbitmq, SXAdaptor): - "Test a working, simple, two file call with parameter" - - seen_name = None - - f1 = { - "paths": ["fork/it/over/1"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - f2 = { - "paths": ["fork/it/over/2"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - - async def my_callback(did_name: str, info: Dict[str, Any]): - nonlocal seen_name - seen_name = did_name - yield [f1, f2] - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - - rabbitmq.send_did_request("hithere?files=1") - - # Make sure callback was called - assert seen_name == "hithere" - - # Make sure only one file was sent along. - SXAdaptor.put_file_add_bulk.assert_called_with([f1]) - SXAdaptor.put_fileset_complete.assert_called_once() - - -def test_with_scope(rabbitmq, SXAdaptor): - seen_name: Optional[str] = None - - async def my_callback(did_name: str, info: Dict[str, Any]): - nonlocal seen_name - seen_name = did_name - yield { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - - did = ( - "cms:DYJetsToLL_M-50_TuneCP5_13TeV-amcatnloFXFX-pythia8" - "/RunIIAutumn18NanoAODv7-Nano02Apr2020_102X_upgrade2018_realistic_v21_ext2-v1/NANOAODSIM" - ) - rabbitmq.send_did_request(did) - - assert seen_name == did - - -def test_failed_file(rabbitmq, SXAdaptor): - "Test a callback that fails before any files are sent" - - async def my_callback( - did_name: str, info: Dict[str, Any] - ) -> AsyncGenerator[Dict[str, Any], None]: - if False: - yield {"ops": "no"} - raise Exception("that did not work") - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - rabbitmq.send_did_request("hi-there") - - # Make sure the file was sent along, along with the completion - SXAdaptor.put_file_add.assert_not_called() - SXAdaptor.put_fileset_complete.assert_not_called() - - -def test_failed_file_after_good(rabbitmq, SXAdaptor): - "Test a callback that fails before any files are sent" - - async def my_callback( - did_name: str, info: Dict[str, Any] - ) -> AsyncGenerator[Dict[str, Any], None]: - yield { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - raise Exception("that did not work") - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - rabbitmq.send_did_request("hi-there") - - # Make sure the file was sent along, along with the completion - SXAdaptor.put_file_add.assert_called_once() - SXAdaptor.put_fileset_complete.assert_not_called() - - -def test_failed_file_after_good_with_avail(rabbitmq, SXAdaptor): - "Test a callback that fails before any files are sent" - - async def my_callback( - did_name: str, info: Dict[str, Any] - ) -> AsyncGenerator[Dict[str, Any], None]: - yield { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - raise Exception("that did not work") - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - rabbitmq.send_did_request("hi-there?get=available") - - # Make sure the file was sent along, along with the completion - SXAdaptor.put_file_add.assert_called_once() - SXAdaptor.put_fileset_complete.assert_called_once() - - -def test_failed_file_after_good_with_avail_limited_number(rabbitmq, SXAdaptor): - "Files are sent back, only available allowed, but want a certian number" - - async def my_callback( - did_name: str, info: Dict[str, Any] - ) -> AsyncGenerator[Dict[str, Any], None]: - yield { - "paths": ["fork/it/over1"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - yield { - "paths": ["fork/it/over2"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - raise Exception("that did not work") - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - rabbitmq.send_did_request("hi-there?get=available&files=1") - - # Make sure the file was sent along, along with the completion - SXAdaptor.put_file_add_bulk.assert_called_once() - SXAdaptor.put_fileset_complete.assert_called_once() - - -def test_no_files_returned(rabbitmq, SXAdaptor): - "Test a callback that fails before any files are sent" - - async def my_callback( - did_name: str, info: Dict[str, Any] - ) -> AsyncGenerator[Dict[str, Any], None]: - if False: - yield {"ops": "no"} - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=12, - retry_interval=10, - ) - rabbitmq.send_did_request("hi-there") - - # Make sure the file was sent along, along with the completion - SXAdaptor.put_file_add.assert_not_called() - SXAdaptor.put_fileset_complete.assert_called_once() - assert SXAdaptor.put_fileset_complete.call_args[0][0]["files"] == 0 - - -def test_rabbitmq_connection_failure(rabbitmq_fail_once, SXAdaptor): - "Make sure that when we have a connection failure we try again" - - called = False - - async def my_callback( - did_name: str, info: Dict[str, Any] - ) -> AsyncGenerator[Dict[str, Any], None]: - nonlocal called - called = True - yield { - "paths": ["fork/it/over"], - "adler32": "no clue", - "file_size": 22323, - "file_events": 0, - } - - init_rabbit_mq( - my_callback, - "http://myrabbit.com", - "test_queue_name", - retries=1, - retry_interval=0.1, - ) - rabbitmq_fail_once.send_did_request("hi-there") - - assert called - - -def test_auto_args_callback(init_rabbit_callback, simple_argument_parser): - "If there is a missing argument on the command line it should cause a total failure" - - async def my_callback( - did_name: str, info: Dict[str, Any] - ) -> AsyncGenerator[Dict[str, Any], None]: - if False: - yield {"ops": "no"} - - start_did_finder("special", my_callback) - - assert init_rabbit_callback.call_args[0][1] == "test_queue_address" - assert init_rabbit_callback.call_args[0][2] == "special_did_requests" - - -def test_arg_required(init_rabbit_callback): - "Make sure the argument passed on the command line makes it in" - # This option is only available in 3.9, so for less than 3.9 we can't run this test. - parser = argparse.ArgumentParser() - add_did_finder_cnd_arguments(parser) - parser.add_argument("--dude", dest="sort_it", action="store") - - args = parser.parse_args(["--rabbit-uri", "not-really-there"]) - - async def my_callback( - did_name: str, info: Dict[str, Any] - ) -> AsyncGenerator[Dict[str, Any], None]: - if False: - yield {"ops": "no"} - - start_did_finder("test_finder", my_callback, args) - - assert init_rabbit_callback.call_args[0][1] == "not-really-there" - - -@pytest.mark.asyncio -async def test_run_file_fetch_loop(SXAdaptor, mocker): - async def my_user_callback(did, info): - return_values = [ - { - "paths": ["/tmp/foo"], - "adler32": "13e4f", - "file_size": 1024, - "file_events": 128, - }, - { - "paths": ["/tmp/bar"], - "adler32": "f33d", - "file_size": 2046, - "file_events": 64, - }, - ] - for v in return_values: - yield v - - await run_file_fetch_loop("123-456", SXAdaptor, {}, my_user_callback) - - assert SXAdaptor.put_file_add.call_count == 2 - assert SXAdaptor.put_file_add.call_args_list[0][0][0]["paths"][0] == "/tmp/foo" - assert SXAdaptor.put_file_add.call_args_list[1][0][0]["paths"][0] == "/tmp/bar" - - SXAdaptor.put_fileset_complete.assert_called_once - assert SXAdaptor.put_fileset_complete.call_args[0][0]["files"] == 2 - assert SXAdaptor.put_fileset_complete.call_args[0][0]["files-skipped"] == 0 - assert SXAdaptor.put_fileset_complete.call_args[0][0]["total-events"] == 192 - assert SXAdaptor.put_fileset_complete.call_args[0][0]["total-bytes"] == 3070 - - -@pytest.mark.asyncio -async def test_run_file_bulk_fetch_loop(SXAdaptor, mocker): - async def my_user_callback(did, info): - return_values = [ - { - "paths": ["/tmp/foo"], - "adler32": "13e4f", - "file_size": 1024, - "file_events": 128, - }, - { - "paths": ["/tmp/bar"], - "adler32": "f33d", - "file_size": 2046, - "file_events": 64, - }, - ] - yield return_values - - await run_file_fetch_loop("123-456", SXAdaptor, {}, my_user_callback) - - assert SXAdaptor.put_file_add_bulk.call_count == 1 - assert ( - SXAdaptor.put_file_add_bulk.call_args_list[0][0][0][0]["paths"][0] == "/tmp/foo" - ) - assert ( - SXAdaptor.put_file_add_bulk.call_args_list[0][0][0][1]["paths"][0] == "/tmp/bar" - ) - - SXAdaptor.put_fileset_complete.assert_called_once - assert SXAdaptor.put_fileset_complete.call_args[0][0]["files"] == 2 - assert SXAdaptor.put_fileset_complete.call_args[0][0]["files-skipped"] == 0 - assert SXAdaptor.put_fileset_complete.call_args[0][0]["total-events"] == 192 - assert SXAdaptor.put_fileset_complete.call_args[0][0]["total-bytes"] == 3070 - - -@pytest.mark.asyncio -async def test_run_file_fetch_one(SXAdaptor, mocker): - async def my_user_callback(did, info): - return_values = [ - { - "paths": ["/tmp/foo"], - "adler32": "13e4f", - "file_size": 1024, - "file_events": 128, - }, - { - "paths": ["/tmp/bar"], - "adler32": "f33d", - "file_size": 2046, - "file_events": 64, - }, - ] - for v in return_values: - yield v - - await run_file_fetch_loop("123-456?files=1", SXAdaptor, {}, my_user_callback) - - assert SXAdaptor.put_file_add_bulk.call_count == 1 - SXAdaptor.put_file_add_bulk.assert_called_with( - [ - { - "paths": ["/tmp/bar"], - "adler32": "f33d", - "file_size": 2046, - "file_events": 64, - } - ] - ) - - SXAdaptor.put_fileset_complete.assert_called_once - assert SXAdaptor.put_fileset_complete.call_args[0][0]["files"] == 1 - - -@pytest.mark.asyncio -async def test_run_file_fetch_one_reverse(SXAdaptor, mocker): - "The files should be sorted so they return the same" - - async def my_user_callback(did, info): - return_values = [ - { - "paths": ["/tmp/bar"], - "adler32": "f33d", - "file_size": 2046, - "file_events": 64, - }, - { - "paths": ["/tmp/foo"], - "adler32": "13e4f", - "file_size": 1024, - "file_events": 128, - }, - ] - for v in return_values: - yield v - - await run_file_fetch_loop("123-456?files=1", SXAdaptor, {}, my_user_callback) - - assert SXAdaptor.put_file_add_bulk.call_count == 1 - SXAdaptor.put_file_add_bulk.assert_called_with( - [ - { - "paths": ["/tmp/bar"], - "adler32": "f33d", - "file_size": 2046, - "file_events": 64, - } - ] - ) - - SXAdaptor.put_fileset_complete.assert_called_once - assert SXAdaptor.put_fileset_complete.call_args[0][0]["files"] == 1 - - -@pytest.mark.asyncio -async def test_run_file_fetch_one_multi(SXAdaptor, mocker): - async def my_user_callback(did, info): - return_values = [ - { - "paths": ["/tmp/foo", "others:/tmp/foo"], - "adler32": "13e4f", - "file_size": 1024, - "file_events": 128, - }, - { - "paths": ["/tmp/bar", "others:/tmp/bar"], - "adler32": "f33d", - "file_size": 2046, - "file_events": 64, - }, - ] - for v in return_values: - yield v - - await run_file_fetch_loop("123-456?files=1", SXAdaptor, {}, my_user_callback) - - assert SXAdaptor.put_file_add_bulk.call_count == 1 - SXAdaptor.put_file_add_bulk.assert_called_with( - [ - { - "paths": ["/tmp/bar", "others:/tmp/bar"], - "adler32": "f33d", - "file_size": 2046, - "file_events": 64, - } - ] - ) - - SXAdaptor.put_fileset_complete.assert_called_once - assert SXAdaptor.put_fileset_complete.call_args[0][0]["files"] == 1 - - -@pytest.mark.asyncio -async def test_run_file_fetch_loop_bad_did(SXAdaptor, mocker): - async def my_user_callback(did, info): - return_values = [] - for v in return_values: - yield v - - await run_file_fetch_loop("123-456", SXAdaptor, {}, my_user_callback) - - assert SXAdaptor.put_file_add.assert_not_called - SXAdaptor.put_fileset_complete.assert_called_once() diff --git a/tests/servicex_did_finder_lib/test_servicex_did.py b/tests/servicex_did_finder_lib/test_servicex_did.py deleted file mode 100644 index e28e22e..0000000 --- a/tests/servicex_did_finder_lib/test_servicex_did.py +++ /dev/null @@ -1,70 +0,0 @@ -import json - -import responses -from servicex_did_finder_lib import __version__ -from servicex_did_finder_lib.servicex_adaptor import ServiceXAdapter - - -def test_version(): - assert __version__ == '1.0.0a1' - - -@responses.activate -def test_put_file_add(): - responses.add(responses.PUT, 'http://servicex.org/12345/files', status=206) - sx = ServiceXAdapter("http://servicex.org/", '12345') - sx.put_file_add({ - 'paths': ['root://foo.bar.ROOT'], - 'adler32': '32', - 'file_size': 1024, - 'file_events': 3141 - }) - - assert len(responses.calls) == 1 - submitted = json.loads(responses.calls[0].request.body) - assert submitted['paths'][0] == 'root://foo.bar.ROOT' - assert submitted['adler32'] == '32' - assert submitted['file_events'] == 3141 - assert submitted['file_size'] == 1024 - - -@responses.activate -def test_put_file_add_bulk(): - responses.add(responses.PUT, 'http://servicex.org/12345/files', status=206) - sx = ServiceXAdapter("http://servicex.org/", '12345') - sx.put_file_add_bulk([{ - 'paths': ['root://foo.bar.ROOT'], - 'adler32': '32', - 'file_size': 1024, - 'file_events': 3141 - }, { - 'paths': ['root://foo.bar1.ROOT'], - 'adler32': '33', - 'file_size': 1025, - 'file_events': 3142 - }]) - - assert len(responses.calls) == 1 - submitted = json.loads(responses.calls[0].request.body) - assert submitted[0]['paths'][0] == 'root://foo.bar.ROOT' - assert submitted[0]['adler32'] == '32' - assert submitted[0]['file_events'] == 3141 - assert submitted[0]['file_size'] == 1024 - - assert submitted[1]['paths'][0] == 'root://foo.bar1.ROOT' - assert submitted[1]['adler32'] == '33' - assert submitted[1]['file_events'] == 3142 - assert submitted[1]['file_size'] == 1025 - - -@responses.activate -def test_put_file_add_bulk_large(): - responses.add(responses.PUT, 'http://servicex.org/12345/files', status=206) - sx = ServiceXAdapter("http://servicex.org/", '12345') - sx.put_file_add_bulk([{ - 'paths': ['root://foo.bar.ROOT'], - 'adler32': '32', - 'file_size': 1024, - 'file_events': 3141 - }] * 320) - assert len(responses.calls) == 2 diff --git a/tests/servicex_did_finder_lib/__init__.py b/tests/servicex_did_finder_lib_tests/__init__.py similarity index 100% rename from tests/servicex_did_finder_lib/__init__.py rename to tests/servicex_did_finder_lib_tests/__init__.py diff --git a/tests/servicex_did_finder_lib_tests/conftest.py b/tests/servicex_did_finder_lib_tests/conftest.py new file mode 100644 index 0000000..0ec57f4 --- /dev/null +++ b/tests/servicex_did_finder_lib_tests/conftest.py @@ -0,0 +1,38 @@ +# Copyright (c) 2024, IRIS-HEP +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# * Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +import pytest + + +@pytest.fixture +def single_file_info(): + return { + "paths": ["fork/it/over"], + "adler32": "no clue", + "file_size": 22323, + "file_events": 0, + } diff --git a/tests/servicex_did_finder_lib_tests/test_accumulator.py b/tests/servicex_did_finder_lib_tests/test_accumulator.py new file mode 100644 index 0000000..3262c66 --- /dev/null +++ b/tests/servicex_did_finder_lib_tests/test_accumulator.py @@ -0,0 +1,82 @@ +# Copyright (c) 2024, IRIS-HEP +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# * Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +import pytest + +from servicex_did_finder_lib.accumulator import Accumulator +from servicex_did_finder_lib.did_summary import DIDSummary +from servicex_did_finder_lib.servicex_adaptor import ServiceXAdapter + + +@pytest.fixture +def servicex(mocker) -> ServiceXAdapter: + return mocker.MagicMock(ServiceXAdapter) + + +@pytest.fixture +def did_summary_obj() -> DIDSummary: + return DIDSummary('did') + + +def test_add_single_file(servicex, did_summary_obj, single_file_info): + acc = Accumulator(sx=servicex, sum=did_summary_obj) + acc.add(single_file_info) + assert acc.cache_len == 1 + + +def test_add_list_of_files(servicex, did_summary_obj, single_file_info): + acc = Accumulator(sx=servicex, sum=did_summary_obj) + acc.add([single_file_info, single_file_info, single_file_info]) + assert acc.cache_len == 3 + + +def test_send_on(servicex, did_summary_obj, single_file_info): + acc = Accumulator(sx=servicex, sum=did_summary_obj) + acc.add([single_file_info, single_file_info, single_file_info, single_file_info]) + acc.send_on(3) + servicex.put_file_add_bulk.assert_called_with( + [single_file_info, single_file_info, single_file_info] + ) + assert acc.cache_len == 0 + assert acc.summary.file_count == 3 + + +def test_send_on_all_files(servicex, did_summary_obj, single_file_info): + acc = Accumulator(sx=servicex, sum=did_summary_obj) + acc.add([single_file_info, single_file_info, single_file_info]) + acc.send_on(-1) + servicex.put_file_add_bulk.assert_called_with( + [single_file_info, single_file_info, single_file_info] + ) + assert acc.cache_len == 0 + assert acc.summary.file_count == 3 + + +def test_invalid_constructor_arg(servicex, did_summary_obj): + with pytest.raises(ValueError): + acc = Accumulator(sx=servicex, sum=did_summary_obj) + acc.add("not a dict!") diff --git a/tests/servicex_did_finder_lib_tests/test_did_finder_app.py b/tests/servicex_did_finder_lib_tests/test_did_finder_app.py new file mode 100644 index 0000000..83f0d34 --- /dev/null +++ b/tests/servicex_did_finder_lib_tests/test_did_finder_app.py @@ -0,0 +1,116 @@ +# Copyright (c) 2024, IRIS-HEP +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# * Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +from unittest.mock import patch +import pytest +from celery import Celery + +from servicex_did_finder_lib.accumulator import Accumulator +from servicex_did_finder_lib.did_finder_app import DIDFinderTask, DIDFinderApp + + +@pytest.fixture() +def servicex(mocker): + """Return a ServiceXAdaptor for testing""" + with patch( + "servicex_did_finder_lib.did_finder_app.ServiceXAdapter", autospec=True + ) as sx_ctor: + sx_adaptor = mocker.MagicMock() + sx_ctor.return_value = sx_adaptor + + yield sx_ctor + + +def test_did_finder_task(mocker, servicex, single_file_info): + did_finder_task = DIDFinderTask() + # did_finder_task.app = mocker.Mock() + did_finder_task.app.did_finder_args = {} + mock_generator = mocker.Mock(return_value=iter([single_file_info])) + + mock_accumulator = mocker.MagicMock(Accumulator) + with patch( + "servicex_did_finder_lib.did_finder_app.Accumulator", autospec=True + ) as acc: + acc.return_value = mock_accumulator + did_finder_task.do_lookup('did', 1, 'https://my-servicex', mock_generator) + servicex.assert_called_with(dataset_id=1, endpoint="https://my-servicex") + acc.assert_called_once() + + mock_accumulator.add.assert_called_with(single_file_info) + mock_accumulator.send_on.assert_called_with(-1) + + servicex.return_value.put_fileset_complete.assert_called_with( + { + "files": 0, # Aught to have a side effect in mock accumulator that updates this + "files-skipped": 0, + "total-events": 0, + "total-bytes": 0, + "elapsed-time": 0, + } + ) + + +def test_did_finder_task_exception(mocker, servicex, single_file_info): + did_finder_task = DIDFinderTask() + # did_finder_task.app = mocker.Mock() + did_finder_task.app.did_finder_args = {} + mock_generator = mocker.Mock(side_effect=Exception("Boom")) + + mock_accumulator = mocker.MagicMock(Accumulator) + with patch( + "servicex_did_finder_lib.did_finder_app.Accumulator", autospec=True + ) as acc: + acc.return_value = mock_accumulator + did_finder_task.do_lookup('did', 1, 'https://my-servicex', mock_generator) + servicex.assert_called_with(dataset_id=1, endpoint="https://my-servicex") + acc.assert_called_once() + + mock_accumulator.add.assert_not_called() + mock_accumulator.send_on.assert_not_called() + + servicex.return_value.put_fileset_complete.assert_called_with( + { + "files": 0, # Aught to have a side effect in mock accumulator that updates this + "files-skipped": 0, + "total-events": 0, + "total-bytes": 0, + "elapsed-time": 0, + } + ) + + +def test_celery_app(): + app = DIDFinderApp('foo') + assert isinstance(app, Celery) + assert app.name == 'foo' + + @app.did_lookup_task(name="did_finder_rucio.lookup_dataset") + def lookup_dataset(self, did: str, dataset_id: int, endpoint: str) -> None: + self.do_lookup(did=did, dataset_id=dataset_id, + endpoint=endpoint, user_did_finder=lambda x, y, z: None) + + assert lookup_dataset.__name__ == 'wrapper' diff --git a/tests/servicex_did_finder_lib_tests/test_did_summary.py b/tests/servicex_did_finder_lib_tests/test_did_summary.py new file mode 100644 index 0000000..0ca0f3e --- /dev/null +++ b/tests/servicex_did_finder_lib_tests/test_did_summary.py @@ -0,0 +1,50 @@ +# Copyright (c) 2024, IRIS-HEP +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# * Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +from servicex_did_finder_lib.did_summary import DIDSummary + + +def test_did_summary(): + summary = DIDSummary('did') + print(summary) + + summary.add_file({ + "file_size": 22323, + "file_events": 100 + }) + + assert summary.file_count == 1 + assert summary.total_bytes == 22323 + assert summary.total_events == 100 + + summary.add_file({ + "bytes": 100, + "file_events": 300 + }) + assert summary.file_count == 2 + assert summary.total_bytes == 22423 + assert summary.total_events == 400 diff --git a/tests/servicex_did_finder_lib_tests/test_servicex_adaptor.py b/tests/servicex_did_finder_lib_tests/test_servicex_adaptor.py new file mode 100644 index 0000000..6b1f3fa --- /dev/null +++ b/tests/servicex_did_finder_lib_tests/test_servicex_adaptor.py @@ -0,0 +1,144 @@ +import json + +import requests +import responses +from servicex_did_finder_lib.servicex_adaptor import ServiceXAdapter + + +@responses.activate +def test_put_file_add_bulk(): + call_count = 0 + + def request_callback(request): + nonlocal call_count + call_count += 1 + + if call_count == 1: + raise requests.exceptions.ConnectionError("Connection failed") + else: + return (206, {}, "") + + responses.add_callback(responses.PUT, + 'http://servicex.org/12345/files', + callback=request_callback) + + sx = ServiceXAdapter("http://servicex.org/", '12345') + sx.put_file_add_bulk([{ + 'paths': ['root://foo.bar.ROOT'], + 'adler32': '32', + 'file_size': 1024, + 'file_events': 3141 + }, { + 'paths': ['root://foo.bar1.ROOT'], + 'adler32': '33', + 'file_size': 1025, + 'file_events': 3142 + }]) + + assert len(responses.calls) == 1 + 1 # 1 retry + submitted = json.loads(responses.calls[0].request.body) + assert submitted[0]['paths'][0] == 'root://foo.bar.ROOT' + assert submitted[0]['adler32'] == '32' + assert submitted[0]['file_events'] == 3141 + assert submitted[0]['file_size'] == 1024 + + assert submitted[1]['paths'][0] == 'root://foo.bar1.ROOT' + assert submitted[1]['adler32'] == '33' + assert submitted[1]['file_events'] == 3142 + assert submitted[1]['file_size'] == 1025 + + +@responses.activate +def test_put_file_add_bulk_large(): + + responses.add(responses.PUT, 'http://servicex.org/12345/files', status=206) + + sx = ServiceXAdapter("http://servicex.org/", '12345') + sx.put_file_add_bulk([{ + 'paths': ['root://foo.bar.ROOT'], + 'adler32': '32', + 'file_size': 1024, + 'file_events': 3141 + }] * 320) + assert len(responses.calls) == 2 # No retries + + +@responses.activate +def test_put_file_add_bulk_failure(): + + def request_callback(request): + raise requests.exceptions.ConnectionError("Connection failed") + + responses.add_callback(responses.PUT, + 'http://servicex.org/12345/files', + callback=request_callback) + + sx = ServiceXAdapter("http://servicex.org/", '12345') + sx.put_file_add_bulk([{ + 'paths': ['root://foo.bar.ROOT'], + 'adler32': '32', + 'file_size': 1024, + 'file_events': 3141 + }, { + 'paths': ['root://foo.bar1.ROOT'], + 'adler32': '33', + 'file_size': 1025, + 'file_events': 3142 + }]) + + assert len(responses.calls) == 3 # Max retries + + +@responses.activate +def test_put_file_complete(): + call_count = 0 + + def request_callback(request): + nonlocal call_count + call_count += 1 + + if call_count == 1: + raise requests.exceptions.ConnectionError("Connection failed") + else: + return (206, {}, "") + + responses.add_callback(responses.PUT, + 'http://servicex.org/12345/complete', + callback=request_callback) + + sx = ServiceXAdapter("http://servicex.org/", '12345') + sx.put_fileset_complete({ + "files": 100, + "files-skipped": 0, + "total-events": 1000, + "total-bytes": 10000, + "elapsed-time": 10 + }) + assert len(responses.calls) == 1 + 1 # 1 retry + submitted = json.loads(responses.calls[0].request.body) + assert submitted['files'] == 100 + assert submitted['files-skipped'] == 0 + assert submitted['total-events'] == 1000 + assert submitted['total-bytes'] == 10000 + assert submitted['elapsed-time'] == 10 + + +@responses.activate +def test_put_file_complete_failure(): + + def request_callback(request): + raise requests.exceptions.ConnectionError("Connection failed") + + responses.add_callback(responses.PUT, + 'http://servicex.org/12345/complete', + callback=request_callback) + + sx = ServiceXAdapter("http://servicex.org/", '12345') + sx.put_fileset_complete({ + "files": 100, + "files-skipped": 0, + "total-events": 1000, + "total-bytes": 10000, + "elapsed-time": 10 + }) + assert len(responses.calls) == 3 # Max retries diff --git a/tests/servicex_did_finder_lib/test_util_uri.py b/tests/servicex_did_finder_lib_tests/test_util_uri.py similarity index 100% rename from tests/servicex_did_finder_lib/test_util_uri.py rename to tests/servicex_did_finder_lib_tests/test_util_uri.py diff --git a/tests/stresstest/stressful_did_finder.py b/tests/stresstest/stressful_did_finder.py new file mode 100644 index 0000000..3dc634a --- /dev/null +++ b/tests/stresstest/stressful_did_finder.py @@ -0,0 +1,74 @@ +# Copyright (c) 2024, IRIS-HEP +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# * Redistributions of source code must retain the above copyright notice, this +# list of conditions and the following disclaimer. +# +# * Redistributions in binary form must reproduce the above copyright notice, +# this list of conditions and the following disclaimer in the documentation +# and/or other materials provided with the distribution. +# +# * Neither the name of the copyright holder nor the names of its +# contributors may be used to endorse or promote products derived from +# this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +import argparse +import logging +from typing import Any, Dict, Generator + +from servicex_did_finder_lib import DIDFinderApp + +__log = logging.getLogger(__name__) + + +def find_files(did_name: str, + info: Dict[str, Any], + did_finder_args: dict = None) -> Generator[Dict[str, Any], None, None]: + for i in range(int(did_finder_args['num_files'])): + yield { + 'paths': did_finder_args['file_path'], + 'adler32': 0, # No clue + 'file_size': 0, # Size in bytes if known + 'file_events': i, # Include clue of how far we've come + } + + +def run_open_data(): + # Parse the command line arguments + parser = argparse.ArgumentParser() + parser.add_argument('--num-files', dest='num_files', action='store', + default='10', + help='Number of files to generate for each dataset') + + parser.add_argument('--file-path', dest='file_path', action='store', + default='', + help='Path to a file to be returned in each response') + + DIDFinderApp.add_did_finder_cnd_arguments(parser) + + __log.info('Starting Stressful DID finder') + app = DIDFinderApp('stressful_did_finder', parsed_args=parser.parse_args()) + + @app.did_lookup_task(name="stressful_did_finder.lookup_dataset") + def lookup_dataset(self, did: str, dataset_id: int, endpoint: str) -> None: + self.do_lookup(did=did, dataset_id=dataset_id, + endpoint=endpoint, user_did_finder=find_files) + + app.start() + + +if __name__ == "__main__": + run_open_data()