Coinbase Data Download Failures

[RyanAshbaugh]

Expected Behavior

Running the following lean data download command I expect to get Coinbase data for ETHUSD beginning at the earlier of the starting data or first available date.

lean data download \
   --data-provider-historical "Coinbase Advanced Trade"
   --security-type Crypto
   --data-type Trade
   --resolution Daily
   --ticker ETHUSD
   --market Coinbase
   --start 20150720
   --end 20250901
   --coinbase-api-name ""
   --coinbase-api-private-key ""

I know that the lean data download command is technically from the lean-cli, however I believe the data downloading functionality actually comes from this repo.

Actual Behavior

However, running the above command, no ethusd_trade.zip file is created. The last few lines of the output look like this

20250925 13:18:47.523 TRACE:: DownloaderDataProvider.Main(): Starting download Symbol: ETHUSD 2XR, Resolution: Daily, StartUtc: 7/20/2015 12:00:00 AM, EndUtc: 9/1/2025 
12:00:00 AM, TickType: Trade
20250925 13:18:47.817 TRACE:: DownloaderDataProvider.RunDownload(): 100.00% complete (1 out of 1 symbols)
20250925 13:18:47.817 TRACE:: DownloaderDataProvider.RunDownload(): Download completed for ETHUSD 2XR at Daily resolution, covering the period from 7/20/2015 12:00:00 AM to
9/1/2025 12:00:00 AM.
20250925 13:18:47.817 TRACE:: All downloads completed in 0.30 seconds.

The completed in 0.30 seconds hints at the fact it did not complete successfully, due to it being so fast.

Potential Solution

Interestingly, when we instead bump up the starting date to 2016-01-01, it does successfully retrieve the ticker data and generate the correct ethusd_trade.zip file

lean data download \
   --data-provider-historical "Coinbase Advanced Trade"
   --security-type Crypto
   --data-type Trade
   --resolution Daily
   --ticker ETHUSD
   --market Coinbase
   --start 20160101
   --end 20250901
   --coinbase-api-name ""
   --coinbase-api-private-key ""

The last few lines of the output look like

20250925 13:34:25.460 TRACE:: DownloaderDataProvider.Main(): Starting download Symbol: ETHUSD 2XR, Resolution: Daily, StartUtc: 1/1/2016 12:00:00 AM, EndUtc: 9/1/2025 
12:00:00 AM, TickType: Trade
20250925 13:34:29.163 TRACE:: DownloaderDataProvider.RunDownload(): 100.00% complete (1 out of 1 symbols)
20250925 13:34:29.163 TRACE:: DownloaderDataProvider.RunDownload(): Download completed for ETHUSD 2XR at Daily resolution, covering the period from 1/1/2016 12:00:00 AM to 
9/1/2025 12:00:00 AM.
20250925 13:34:29.163 TRACE:: All downloads completed in 3.71 seconds.

Furthermore, the first date that ETHUSD data is available is 2016-05-18, even though we used the starting date of 2016-01-01. This suggests the downloader is able to work with dates earlier than the first available date to some extent, but is not robust to any earlier date. In fact, for ETHUSD the cutoff appears to be 2015-09-21, 241 days before the first available day. So if we use 2015-09-21 as the start date it will correctly generate the daily data file, but if we use 2015-09-20 as the starting date it fails to generate the data file.

This suggests a potential solution is to efficiently identify the actual first date available internally, before initializing the full data download, and replacing the starting date with the first available date, such that it is more reliably able to generate the data file. This would be especially helpful considering there are hundreds of tickers available on Coinbase, so manually identifying the first available date for each of these would be very time consuming.

Reproducing the Problem

The problem can be reproduced with the lean data download calls detailed above.

System Information

OS - Ubuntu 22.04
Lean version - 1.0.220

Checklist

• I have completely filled out this template
• I have confirmed that this issue exists on the current master branch
• I have confirmed that this is not a duplicate issue by searching issues

• I have provided detailed steps to reproduce the issue