Trait idea: cursors

[hoelzro]

Hi there! I was thinking about implementing a new trait, and I wanted to see if you had any feedback on its design and whether it would be appropriate for inclusion with Twitter::API or it should be its own standalone thing.

Basically, there are two things that I don't like about working with the Twitter API (not this module in particular - it's great! Just the API in general):

1. Different REST endpoints have different ways of paging through results; some use a next_cursor/cursor system, some use since_id and max_id.
2. I rather dislike this pattern:

my $tw = Twitter::API->new(...);
my $max_id;
while(1) {
  my $favorites = $tw->favorites({
    (defined($max_id) ? (max_id => $max_id - 1) : ()),
  });

  last unless @$favorites;
  # process $favorites here...
  $max_id = min(map { $_->{'id'} } @$favorites);
}

I would much rather write code like this, which is more concise and less error prone:

my $tw = Twitter::API->new(...);
my $favorites = $tw->favorites;
while(my $tweet = $favorites->next) {
  # process $tweet here...
}
# alternatively, you could call $favorites->all to return a list of all favorites

The $favorites value would be a cursor object that you could use above, but could still be an array reference so that it's backwards compatible with traditional usage. What do you think?

[semifor]

I like the idea, Rob. Sorry for the delay. Broke an arm last week which slowed me down considerably. Typing is a an exercise in frustration.

I wonder if it might be better to have a cursor or make_cursor (naming is hard) method that takes the underlying method and args as parameters:

my $tw = Twitter::API->new(...);
my $cursor = $tw->cursor(favorites => { tweet_mode => 'extended'});
while(my $tweet = $cursor->next) {
  # process $tweet here...
}

That way, it would work with any existing or new endpoint that uses cursors.

Hmm. It may need a parameter to specify the name of the results array. That varies: ids, statuses, etc. It may be discoverable though. In any case, it's a good idea.

You may want to mix code that uses the cursors with code that doesn't, so overriding favorites in a trait, for example, might not be best. So this can probably be added to core instead of a trait??

[hoelzro]

Broke an arm last week which slowed me down considerably.

Ouch, sorry to hear that! Get well soon!

Hmm. It may need a parameter to specify the name of the results array. That varies: ids, statuses, etc. It may be discoverable though.

Yeah, that's one thing that gaves me pause about a cursor method - the friends and user_timeline endpoints, for example, use two different cursor styles. The way I had in mind, I was thinking we could just annotate API methods on a case-by-case basis, but that means new endpoints wouldn't benefit from cursors, like you pointed out. On the other hand, we could inspect the result and see if it's a JSON array or JSON object with next_cursor property, etc.

If I implemented this as a trait, I would have something like user_timeline return a Twitter::API::Cursor object, which would be a blessed array reference, so you could call cursor methods on it, but you could also treat it as a regular array if you want to do current-style processing of the results. But maybe that's too magical; I don't know. I think I'll prototype a trait for use in my current project and see if I can shake out some conceptual bugs while your arm heals up!

[hoelzro]

I just thought I'd follow up on this; I haven't had a chance to try this idea out, because amusingly enough my wife went into labor a hours after my last message, so time has been a little short for me lately! I did, however, have a thought about my idea of blessing a regular array into a cursor class - one drawback to that is if one uses ref to perform different actions on a data structure. Granted, I think that usage might be rare, and there's always Scalar::Util::reftype, but I figured it was worth mentioning.

Another potential issue is ordering of results - let's use user_timeline as an example again. When fetching pages of results from that endpoint, you need to fetch pages in reverse chronological order, but I don't know if the Twitter API specifies if the entries in a page are sorted in reverse chronological order or not. If they're not, iterating over the cursor results could yield some interesting results depending on whether or not the code using the cursor assumes an ordering. I'll keep this in mind when I'm prototyping.

[semifor]

Congratulations!

Turns out user_timeline and the like don't use cursors. They use a different paging technique with count, max_id, and since_id parameters. See Working with Timelines.

I see Scalar::Util::RefType is deprecated and suggests Ref::Util instead. Twitter::API already has a dependency on Ref::Util, so we can readily use it.

I'm still in hunt-and-peck mode. Cast comes off one week from today so hopefully, I'll become more useful soon.

[hoelzro]

@semifor Right - I was thinking of offering a cursor-like abstraction that abstracts over details like whether something uses max_id/since_id like timelines, or an explicit cursor parameters like listing friends.