[Bug] Issue with org-projects and too many packages

[konsumer]

Godot Version & Plugin Version
Godot: 3.2.1
Plugin: 1.4.1

Describe the bug
I have a lot of github projects, under my name, and also under several orgs. The first time I ran your plugin, it crashed godot. I restarted and it was missing many of my projects in list (only 107, I have 430 under my name, and many more under other orgs.) I could not find the project I want (notnullgames/lol) in search or scrolling. It seems to have grabbed a few random org projects, but none in @notnullgames. Even if there was just an option to enter it manually and skip the godot-crashing bug while trying to load, I'd be all set (I know the names of my projects.)

Screenshots

[fenix-hub]

Hi @konsumer ,
First of all, please try updating the plugin to the latest version, which is 1.4.3, directly from main.
On the AssetLibrary the plugin has not been already updated (it is a manual process as far as I know).

GitHub APIs have by default an unchangable cap of 100 repositories per request (whatever the owner is, an org or a user).
So the plugin is basically fetching the first 100 repositories it can find, based on alphabetical order and owner settings (which you can change within the plugin, going to the Notification Bell > Settings).

Latest changes shouldn't make Godot crash, but anyway let me know how it is going updating the version.
I never had the opportunity to test it with a user with so many repositories, so eventually you could help me improving the behavior with this situation.
I could make a queue of requests to fetch all of your repositories (personally I don't like it since it may take a huge amount of time), or let the user request a specific repository to fetch, or let the user select an interval.
Waiting from you :)

[konsumer]

Yep, no crash on main, but also I did get it to work before by re-opening it. New version seems much faster at getting the same list, though, on a new computer, so no cache. Still limited to 107 repos, and you are right, they are alphabetical.

Messing around with settings did give me access to other repos, but still can't access the one I want. For me it would be ideal if it didn't download a list at all, and just allowed me to enter the repo-name right away.

Some recommendations:

• Use paging buttons & use much smaller pages. That's what those API params are made for, and it will make it feel snappier, and people can get to > 100 repos.
• Allow user to just enter repo-name. For me, I always know the name, or at least can look it up, and this is much faster than waiting for it to download a list, or paging through 50 pages. If I could manually add a list of repos I care about (only a few of my repos are godot repos) it would be easier to work with.
• You can also use an "infinite scroll" technique to update as the user scrolls, generally with many less items, per-page (so it's faster.) The basic idea is you get the total height (since you can know upfront how many records you will get, and the lines are fixed-height) and use the scroll offset to decide page to request, then offset it within the ScrollContainer when the user stops scrolling (or even slows down). I'm newish to godot, so I'm not clear on the details, but this is a common strategy on the web for long lists of data. One trick I often use is the just using the scroller as a view/controller for a data-structure I fill. so on stop-scroll, I check if I already have the records in view (in my cache-array) and if not, make a new request. I only request pages of data that are a little bigger than the visible area of the scroll-area. This can make scrolling back & forth very snappy feeling. Another bonus is that it feels intuitive, if the records are in alphabetical order. the user stops for a sec and sees they are on "A", and can estimate where to stop to get to the letter they want (maybe in a few tries.) It's like the best possible paging, in my opinion.

Hopefully that is helpful. I am super-busy right now, or I would jump in and offer to help, but maybe I can add some assistance, later.

[konsumer]

I'd need to dig in to how you are using it, but I quickly checked to see how one might get the data needed for this sort of scrolling. It seems like they don't allow random-access, at least in the graphql API, as far as I can tell, but you could still do a page-by-page scrolling system. You can go here and get repos (complete with total and first page) with this:

{
  viewer {
    repositories(first: 20) {
      nodes {
        id
        name
      }
      pageInfo {
        endCursor
      }
      totalCount
    }
  }
}

Then, use endCursor in repositories(after) to get the next 20, and so on, and so on.

{
  viewer {
    repositories(first: 20, after: "ENDCURSOR") {
      nodes {
        id
        name
      }
      pageInfo {
        endCursor
      }
    }
  }
}

Here is an example of the sort of page-page scrolling I mean (in react.) It's not quite as nice as "download data when user stops scrolling" but it's still nice.

[fenix-hub]

Hi @konsumer and thank you very much for this golden answer.

I'm happy you are not having crash issues with Godot anymore.
I'm aware of request paging and cursors, just never had the chance to actually use it with this amount of repositories, so never implemented it.

I think that a combination of infinite scrolling, paging and static loading would be the best option right now, since I too don't have much time these days, so I want to provide a working solution in the less time possible.
I could limit the loading of repositories to 50 repositories per "page", and the last item in the list is a button to press that lets you load other 50 repositories. Once a user presses a page button the next repositories will be loaded using cursors. Then you can already filter the repositories looking for names. Of course this solution will not let the user specifically scroll for a range of repositories automatically (which is the ultimate result I'd like to accomplish) but anyway would allow the user to reach the repository needed.

About implementing the repository name injection - I could implement that too for sure. It is the easiest and fastest solution to do honestly, since I just need to give you an input to insert the name and then the plugin is already able to do the rest.
The problem with this is that you will always have to add your repository manually once you open the editor, since the plugin doesn't use any cache about repositories currently, and everything is requested at runtime. I could for sure implement a cache, even only for this feature, so user-inserted repositories will be always requested AFTER the automatic fetching, but the problem is that at this point this could cause "inconsistency" (imagine that tha user, seeing only the first 50 repositories, has inserted other 50 repositories manually. The total loading time would be much and much slower the more repositories are user-iserted).
If it's okay for you to always insert the name of the repository you want to work on, it's fine.

I really like the infinite scroll idea you suggested anyway. It is not that hard to implement, just requires some time.
With the first implementation I talked about we are almost there, just always need the user to activate the loading, and the loading will always be snapped to a fixed range of repositories.
When I'll have time to implement the infinite loading I could also add a side-list of letters to press, so eventually the plugin could dynamically load based on the letter chosed.

[konsumer]

The Graphql API can also get gists & repos in 1 go for the first page:

{
  viewer {
    repositories(first: 20) {
      nodes {
        id
        name
      }
      pageInfo {
        endCursor
      }
      totalCount
    }
    gists(first:20) {
      nodes {
        id
        name
        description
      }
      pageInfo {
        endCursor
      }
      totalCount
    }
  }
}

The problem with this is that you will always have to add your repository manually once you open the editor, since the plugin doesn't use any cache about repositories currently, and everything is requested at runtime.

Ah, yeah, I can see the issue there. For me, a list of applicable repos would be much better (like a "favorites" or "following" or "bookmarked" list.) It doesn't need to be cached, but like I said, only a few of my repos are godot-related, so it's a lot of other things to wade through.

[konsumer]

Also, I guess I am a bit unclear, but I thought the purpose of this plugin was to allow the user to manage a single godot repo that's on github. It seems like a quick check of .git/config (if it exists) could even guess the repo and link it, right?

[konsumer]

func _ready():
  var f = File.new()
  var err = f.open("res://.git/config", File.READ)
  if err != OK:
    printerr("Could not open file, error code ", err)
  else:
    var text = f.get_as_text()
    f.close()
    var regex = RegEx.new()
    regex.compile("url = git@github\\.com:(?<repo>.+)\\.git")
    var result = regex.search(text)
    if result:
      print(result.get_string("repo"))
    else:
      print("not a github repo.")

[fenix-hub]

The purpose of this plugin is a more general managing tool for all of your repositories through Github (also including gist and collaboration, which Github itself doesn't provide as fast as this plugin is actually doing), without using Git.
Basically you don't specifically need a Git knowledge (if not the one required to use Github itself), and no git is used anyway.
Recognizing a git folder in your project is ok if you are already using git, but it is not the primary purpose of the plugin.
You could use the plugin to commit to whatever repository you want, and pull from whatever repository you like in the same project.

[konsumer]

Ah, ok! I thought it might help collaborate with a friend who is less used to git, but that makes perfect sense.

Thanks!

[fenix-hub]

You are welcome, indeed a lot of users who never even used Git and Github reach me out on Discord and use this plugin with their teams. Basically because I focused on working on a """simple and intuitive""" interface to provide basic Git and Github functions, also adding an easier collab management (in this plugin you only have to open your repository and click on the "Invite collaborator" button, then your collaborators don't even need to receive an email or be advertised, but they will receive an invitation directly in Github Integration itself and they will be allowed to see the repository).

Anyway, I like your idea actually. I could just resolve the issue with user-inserted repository exactly recognizing if the project you are working on has a .git folder. I could always load the project you are working on, if it is already a git repository.
What do you think about this?
I was thinking about "bookmars" for repositories too, but the problem for me is always the same: I could just end up with more bookmars than the amount of repositories the plugin is loading by default.

[konsumer]

Basically because I focused on working on a """simple and intuitive""" interface to provide basic Git and Github functions

I love it. I could see it being very useful as a standalone desktop client too, rather than a plugin, if I could get to all my repos :).

could always load the project you are working on, if it is already a git repository. What do you think about this?

Yes, this would be super-handy. That is sort of how I thought it worked anyway, and it would skip the whole problem of searching for the repo. like I could tell someone to clone the repo, then look at it in godot, and it'd be all setup, with your nice integrated views.

I was thinking about "bookmars" for repositories too, but the problem for me is always the same: I could just end up with more bookmars than the amount of repositories the plugin is loading by default.

Yep, I could see that. I have used a few git frontends over the years, as they can be helpful for tricky merges, and showing non-git-power-users how it all fits together. Github's desktop client, as well as sublimerge have a primary concept of "your repos" (bookmarks/clones) and then you "add a new repo" to add something else to the list. I guess I just meant that sort of thing. it seems like the purpose of your tool is a bit different, so maybe it doesn't apply.

[fenix-hub]

Yep, I could see that. I have used a few git frontends over the years, as they can be helpful for tricky merges, and showing non-git-power-users how it all fits together. Github's desktop client, as well as sublimerge have a primary concept of "your repos" (bookmarks/clones) and then you "add a new repo" to add something else to the list. I guess I just meant that sort of thing. it seems like the purpose of your tool is a bit different, so maybe it doesn't apply.

Yes, I just had a different idea in mind... but a different approach, within this plugin or even building up a different one could always be a choice for me. The only limit is the time.

Yes, this would be super-handy. That is sort of how I thought it worked anyway, and it would skip the whole problem of searching for the repo. like I could tell someone to clone the repo, then look at it in godot, and it'd be all setup, with your nice integrated views.

Ok nice, so currently my final decision would be to (1) implement this feature to resolve your specific issue (your snippet seems more than enough, so thank you for taking time to add it), then (2)add the pagination and cursor handiling, and later eventually the (3)infinite scrolling system could be implemented.
The plugin will just recognize if you are working on a git project, will fetch that from gihtub and will highlight it on the main panel in some sort of way.

I'll leave the issue opened anyway so we are always able to discuss here anyway.
Thanks!! Have a nice day.

[konsumer]

I'll leave the issue opened anyway so we are always able to discuss here anyway.

Sounds good! Let me know if you need any testing or anything else.

Thanks again, and you have a nice day, too!